Repository: kubernetes-sigs/kwok
Pull request #911 (2024-01-11 14:27:37)

[KWOK] retry when applying Stages fails
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind feature

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes https://github.com/kubernetes-sigs/kwok/issues/816

#### Special notes for your reviewer:
Since we have a new design based on the weight delaying queue(https://github.com/kubernetes-sigs/kwok/pull/902), the old implementation https://github.com/kubernetes-sigs/kwok/pull/904  was rejected.   Will close the old one and start here. 


Currently, I **only modified the node controller** for a pre-review. Please feel free to give comments!

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
NONE
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kwok/pull/911#" title="Author self-approved">caozhuozi</a>*
**Once this PR has been reviewed and has the lgtm label**, please assign [yibozhuang](https://github.com/yibozhuang) for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkwok).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/kwok/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["yibozhuang"]} -->
netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *k8s-kwok* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 2db54d0647265db0b07a29f8652d7f1da30025e7 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/k8s-kwok/deploys/659ffadcb42dbe000881def1 |
caozhuozi: Hi, @wzshiming.
I added a `shoudRetry` function to determine whether an error needs to be retried.

Currently, we only check those network errors and bypass other error types.






wzshiming: I think like We can use `errors.Join` and `fmt.Errorf`
caozhuozi: Hi, @wzshiming. Do you mean to join the specific error with the newly defined error?
I have two concerns here:
1. I'm not sure if it is a **Go idiomatic** way
2. the other concern is that where should I put the `join` statement? There are 2 choices currently:
    * inside `computePatch` method, like:

      ```go
        func (c *NodeController) computePatch(node *corev1.Node, tpl string) ([]byte, error) {
        patch, err := c.renderer.ToJSON(tpl, node)
        if err != nil {
        return errors.Join(ComputePatchError, err)  // the original `err` will be lost
        }
        // omitted
        }
      ```
    *  at where `computePath` method is called, like

       ```go
       patch, err = c.computePatch(node, next.StatusTemplate)
       if err != nil {
           return errors.Join(ComputePatchError, err)
       }
       ```
       
   which one do you prefer?

wzshiming: I think you can finish this and verify the various corner cases locally
caozhuozi: I made several changes in this commit:
1. define a new error type `ErrComputePatch`
2. remove those resource `isNotFound` checks as they will be handled in `shouldRetry`, which is the **only entry** for determining errors.
3. move the `patch` operation out of `finalizersModify` and rename the method to `computeFinalizerPatch`. If any error occurs in `computeFinalizerPatch`, we should also mark the error as `ErrComputePatch`.
4. rename the `computePatch` to `computeStatusPatch`.
5. generalize `patchResource` method such that it can be used by patching status and patching finalizers.



caozhuozi: since we only need to check the network error, I just removed the defined `ErrComputePatch` in the latest commit.
wzshiming: /ok-to-test
k8s-ci-robot: @caozhuozi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | f5e69e3d2f0b6f08570ffe15a63545d8ba1d17d5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-verify-main/1747452378950406144) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=911). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Acaozhuozi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
caozhuozi: In this new commit,
* Get the  `finalizerModify` logic back
* check whether the key already exists in the map when adding a job
* Create a function in `utils` to obtain a default backoff setting shared by all kwok controllers for those retried jobs.
* rebase the code based on the recent changes.

If there are no problems with this version, I will start to change other controllers.


k8s-ci-robot: @caozhuozi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | 8284f62c7e8723bd331ae483c4e30eaf6c094735 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-verify-main/1747573708551098368) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=911). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Acaozhuozi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @caozhuozi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | 70c4c3b711b990ca8f24e2614d97a963106ca3a9 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-verify-main/1747577035661774848) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=911). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Acaozhuozi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @caozhuozi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | 70c4c3b711b990ca8f24e2614d97a963106ca3a9 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-verify-main/1747577035661774848) | true | `/test pull-kwok-verify-main`
pull-kwok-unit-test-main | b03ca8431d4960072677c75848f4cc539d138843 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-unit-test-main/1749038559026745344) | true | `/test pull-kwok-unit-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=911). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Acaozhuozi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @caozhuozi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | b03ca8431d4960072677c75848f4cc539d138843 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-unit-test-main/1749038559026745344) | true | `/test pull-kwok-unit-test-main`
pull-kwok-verify-main | b03ca8431d4960072677c75848f4cc539d138843 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-verify-main/1749038558921887744) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=911). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Acaozhuozi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @caozhuozi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | b03ca8431d4960072677c75848f4cc539d138843 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-unit-test-main/1749038559026745344) | true | `/test pull-kwok-unit-test-main`
pull-kwok-verify-main | b03ca8431d4960072677c75848f4cc539d138843 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-verify-main/1749038558921887744) | true | `/test pull-kwok-verify-main`
pull-kwok-e2e-test-main | b03ca8431d4960072677c75848f4cc539d138843 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-e2e-test-main/1749038559098048512) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=911). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Acaozhuozi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
wzshiming: Sorry for the delay, this PR may need to be rebase due to the conflict with #920, I will take time to test this PR this week.
caozhuozi: > Sorry for the delay, this PR may need to be rebase due to the conflict with #920, I will take time to test this PR this week.

Oh, @wzshiming! Understood! No worry! I will rebase it. 😊
k8s-ci-robot: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @caozhuozi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-e2e-test-main | b03ca8431d4960072677c75848f4cc539d138843 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-e2e-test-main/1749038559098048512) | true | `/test pull-kwok-e2e-test-main`
pull-kwok-build-main | ff2c2dd4c902c9b1934ee210426f681075d71370 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-build-main/1750159149859082240) | true | `/test pull-kwok-build-main`
pull-kwok-unit-test-main | ff2c2dd4c902c9b1934ee210426f681075d71370 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-unit-test-main/1750159149909413888) | true | `/test pull-kwok-unit-test-main`
pull-kwok-verify-main | ff2c2dd4c902c9b1934ee210426f681075d71370 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-verify-main/1750159149804556288) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=911). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Acaozhuozi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @caozhuozi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-build-main | ff2c2dd4c902c9b1934ee210426f681075d71370 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-build-main/1750159149859082240) | true | `/test pull-kwok-build-main`
pull-kwok-unit-test-main | ff2c2dd4c902c9b1934ee210426f681075d71370 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-unit-test-main/1750159149909413888) | true | `/test pull-kwok-unit-test-main`
pull-kwok-verify-main | ff2c2dd4c902c9b1934ee210426f681075d71370 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-verify-main/1750159149804556288) | true | `/test pull-kwok-verify-main`
pull-kwok-e2e-test-main | ff2c2dd4c902c9b1934ee210426f681075d71370 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/911/pull-kwok-e2e-test-main/1750159149984911360) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=911). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Acaozhuozi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/Makefile b/Makefile
index 354c5636..27810c66 100644
--- a/Makefile
+++ b/Makefile
@@ -76,7 +76,7 @@ IMAGE_PLATFORMS ?= linux/amd64 linux/arm64
 
 BINARY_PLATFORMS ?= linux/amd64 linux/arm64 darwin/amd64 darwin/arm64 windows/amd64 windows/arm64
 
-MANIFESTS ?= kwok kwokctl stage/fast
+MANIFESTS ?= kwok kwokctl stage/fast metrics/usage
 
 BUILDER ?= docker
 DOCKER_CLI_EXPERIMENTAL ?= enabled
diff --git a/kustomize/crd/bases/kwok.x-k8s.io_clusterresourceusages.yaml b/kustomize/crd/bases/kwok.x-k8s.io_clusterresourceusages.yaml
new file mode 100644
index 00000000..8f5ce845
--- /dev/null
+++ b/kustomize/crd/bases/kwok.x-k8s.io_clusterresourceusages.yaml
@@ -0,0 +1,148 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.13.0
+  name: clusterresourceusages.kwok.x-k8s.io
+spec:
+  group: kwok.x-k8s.io
+  names:
+    kind: ClusterResourceUsage
+    listKind: ClusterResourceUsageList
+    plural: clusterresourceusages
+    singular: clusterresourceusage
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: ClusterResourceUsage provides cluster-wide resource usage.
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Spec holds spec for cluster resource usage.
+            properties:
+              selector:
+                description: Selector is a selector to filter pods to configure.
+                properties:
+                  matchNames:
+                    description: MatchNames is a list of names to match. if not set,
+                      all names will be matched.
+                    items:
+                      type: string
+                    type: array
+                  matchNamespaces:
+                    description: MatchNamespaces is a list of namespaces to match.
+                      if not set, all namespaces will be matched.
+                    items:
+                      type: string
+                    type: array
+                type: object
+              usages:
+                description: Usages is a list of resource usage for the pod.
+                items:
+                  description: ResourceUsageContainer holds spec for resource usage
+                    container.
+                  properties:
+                    containers:
+                      description: Containers is list of container names.
+                      items:
+                        type: string
+                      type: array
+                    usage:
+                      additionalProperties:
+                        description: ResourceUsageValue holds value for resource usage.
+                        properties:
+                          expression:
+                            description: Expression is the expression for resource
+                              usage.
+                            type: string
+                          value:
+                            anyOf:
+                            - type: integer
+                            - type: string
+                            description: Value is the value for resource usage.
+                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                            x-kubernetes-int-or-string: true
+                        type: object
+                      description: Usage is a list of resource usage for the container.
+                      type: object
+                  type: object
+                type: array
+            type: object
+          status:
+            description: Status holds status for cluster resource usage
+            properties:
+              conditions:
+                description: Conditions holds conditions for cluster resource usage
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: LastTransitionTime is the last time the condition
+                        transitioned from one status to another. This should be when
+                        the underlying condition changed.  If that is not known, then
+                        using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: Message is a human readable message indicating
+                        details about the transition. This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    reason:
+                      description: Reason contains a programmatic identifier indicating
+                        the reason for the condition's last transition. Producers
+                        of specific condition types may define expected values and
+                        meanings for this field, and whether the values are considered
+                        a guaranteed API. The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: Status of the condition
+                      type: string
+                    type:
+                      description: Type of condition in CamelCase or in foo.example.com/CamelCase.
+                        Many .condition.type values are consistent across resources
+                        like Available, but because arbitrary conditions can be useful
+                        (see .node.status.conditions), the ability to deconflict is
+                        important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+                x-kubernetes-list-map-keys:
+                - type
+                x-kubernetes-list-type: map
+            type: object
+        required:
+        - metadata
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
diff --git a/kustomize/crd/bases/kwok.x-k8s.io_resourceusages.yaml b/kustomize/crd/bases/kwok.x-k8s.io_resourceusages.yaml
new file mode 100644
index 00000000..5e684b56
--- /dev/null
+++ b/kustomize/crd/bases/kwok.x-k8s.io_resourceusages.yaml
@@ -0,0 +1,132 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.13.0
+  name: resourceusages.kwok.x-k8s.io
+spec:
+  group: kwok.x-k8s.io
+  names:
+    kind: ResourceUsage
+    listKind: ResourceUsageList
+    plural: resourceusages
+    singular: resourceusage
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: ResourceUsage provides resource usage for a single pod.
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Spec holds spec for resource usage.
+            properties:
+              usages:
+                description: Usages is a list of resource usage for the pod.
+                items:
+                  description: ResourceUsageContainer holds spec for resource usage
+                    container.
+                  properties:
+                    containers:
+                      description: Containers is list of container names.
+                      items:
+                        type: string
+                      type: array
+                    usage:
+                      additionalProperties:
+                        description: ResourceUsageValue holds value for resource usage.
+                        properties:
+                          expression:
+                            description: Expression is the expression for resource
+                              usage.
+                            type: string
+                          value:
+                            anyOf:
+                            - type: integer
+                            - type: string
+                            description: Value is the value for resource usage.
+                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                            x-kubernetes-int-or-string: true
+                        type: object
+                      description: Usage is a list of resource usage for the container.
+                      type: object
+                  type: object
+                type: array
+            type: object
+          status:
+            description: Status holds status for resource usage
+            properties:
+              conditions:
+                description: Conditions holds conditions for resource usage
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: LastTransitionTime is the last time the condition
+                        transitioned from one status to another. This should be when
+                        the underlying condition changed.  If that is not known, then
+                        using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: Message is a human readable message indicating
+                        details about the transition. This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    reason:
+                      description: Reason contains a programmatic identifier indicating
+                        the reason for the condition's last transition. Producers
+                        of specific condition types may define expected values and
+                        meanings for this field, and whether the values are considered
+                        a guaranteed API. The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: Status of the condition
+                      type: string
+                    type:
+                      description: Type of condition in CamelCase or in foo.example.com/CamelCase.
+                        Many .condition.type values are consistent across resources
+                        like Available, but because arbitrary conditions can be useful
+                        (see .node.status.conditions), the ability to deconflict is
+                        important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+                x-kubernetes-list-map-keys:
+                - type
+                x-kubernetes-list-type: map
+            type: object
+        required:
+        - metadata
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
diff --git a/kustomize/crd/embed.go b/kustomize/crd/embed.go
index d82b613d..61dbcc13 100644
--- a/kustomize/crd/embed.go
+++ b/kustomize/crd/embed.go
@@ -58,6 +58,14 @@ var (
 	//go:embed bases/kwok.x-k8s.io_clusterportforwards.yaml
 	ClusterPortForward []byte
 
+	// ResourceUsage is the custom resource definition for resource usage.
+	//go:embed bases/kwok.x-k8s.io_resourceusages.yaml
+	ResourceUsage []byte
+
+	// ClusterResourceUsage is the custom resource definition for cluster resource usage.
+	//go:embed bases/kwok.x-k8s.io_clusterresourceusages.yaml
+	ClusterResourceUsage []byte
+
 	// Metric is the custom resource definition for metrics.
 	//go:embed bases/kwok.x-k8s.io_metrics.yaml
 	Metric []byte
diff --git a/kustomize/kwokctl/resource/pod.yaml b/kustomize/kwokctl/resource/pod.yaml
index e8eb8f96..993dd77a 100644
--- a/kustomize/kwokctl/resource/pod.yaml
+++ b/kustomize/kwokctl/resource/pod.yaml
@@ -20,11 +20,47 @@ template: |-
     {{ range $index, $container := .containers }}
     - name: {{ $container.name }}
       image: {{ $container.image }}
+      {{ if and $container $container.resources }}
+      {{ $resources := $container.resources }}
+      resources:
+        requests:
+        {{ range $key, $value := $resources.requests }}
+          {{ $key }}: {{ $value }}
+        {{ end }}
+        limits:
+        {{ if $resources.limits  }}
+        {{ range $key, $value := $resources.requests }}
+          {{ $key }}: {{ or ( index $resources.limits  $key ) $value }}
+        {{ end }}
+        {{ else }}
+        {{ range $key, $value := $resources.requests }}
+          {{ $key }}: {{ $value }}
+        {{ end }}
+        {{ end }}
+      {{ end }}
     {{ end }}
     initContainers:
     {{ range $index, $container := .initContainers }}
     - name: {{ $container.name }}
       image: {{ $container.image }}
+      {{ if and $container $container.resources }}
+      {{ $resources := $container.resources }}
+      resources:
+        requests:
+        {{ range $key, $value := $resources.requests }}
+          {{ $key }}: {{ $value }}
+        {{ end }}
+        limits:
+        {{ if $resources.limits  }}
+        {{ range $key, $value := $resources.requests }}
+          {{ $key }}: {{ or ( index $resources.limits  $key ) $value }}
+        {{ end }}
+        {{ else }}
+        {{ range $key, $value := $resources.requests }}
+          {{ $key }}: {{ $value }}
+        {{ end }}
+        {{ end }}
+      {{ end }}
     {{ end }}
     hostNetwork: {{ .hostNetwork }}
     nodeName: {{ .nodeName }}
diff --git a/kustomize/metrics/resource/kustomization.yaml b/kustomize/metrics/resource/kustomization.yaml
new file mode 100644
index 00000000..98d9e613
--- /dev/null
+++ b/kustomize/metrics/resource/kustomization.yaml
@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- metrics-resource.yaml
diff --git a/kustomize/metrics/resource/metrics-resource.yaml b/kustomize/metrics/resource/metrics-resource.yaml
new file mode 100644
index 00000000..f74bafaf
--- /dev/null
+++ b/kustomize/metrics/resource/metrics-resource.yaml
@@ -0,0 +1,92 @@
+kind: Metric
+apiVersion: kwok.x-k8s.io/v1alpha1
+metadata:
+  name: metrics-resource
+spec:
+  path: "/metrics/nodes/{nodeName}/metrics/resource"
+  metrics:
+  - name: scrape_error
+    dimension: node
+    help: |
+      [ALPHA] 1 if there was an error while getting metrics from the node, 0 otherwise
+    kind: gauge
+    value: '0'
+  - name: container_start_time_seconds
+    dimension: container
+    help: |
+      [ALPHA] Start time of the container since unix epoch in seconds
+    kind: gauge
+    labels:
+    - name: container
+      value: 'container.name'
+    - name: namespace
+      value: 'pod.metadata.namespace'
+    - name: pod
+      value: 'pod.metadata.name'
+    value: 'pod.SinceSecond()'
+  # CPU of the container
+  - name: container_cpu_usage_seconds_total
+    dimension: container
+    help: |
+      [ALPHA] Cumulative cpu time consumed by the container in core-seconds
+    kind: counter
+    labels:
+    - name: container
+      value: 'container.name'
+    - name: namespace
+      value: 'pod.metadata.namespace'
+    - name: pod
+      value: 'pod.metadata.name'
+    value: 'pod.CumulativeUsage("cpu", container.name)'
+  # Memory of the container
+  - name: container_memory_working_set_bytes
+    dimension: container
+    help: |
+      [ALPHA] Current working set of the container in bytes
+    kind: gauge
+    labels:
+      - name: container
+        value: 'container.name'
+      - name: namespace
+        value: 'pod.metadata.namespace'
+      - name: pod
+        value: 'pod.metadata.name'
+    value: 'pod.Usage("memory", container.name)'
+  # CPU of the pod
+  - name: pod_cpu_usage_seconds_total
+    dimension: pod
+    help: |
+      [ALPHA] Cumulative cpu time consumed by the pod in core-seconds
+    kind: counter
+    labels:
+      - name: namespace
+        value: 'pod.metadata.namespace'
+      - name: pod
+        value: 'pod.metadata.name'
+    value: 'pod.CumulativeUsage("cpu")'
+  # Memory of the pod
+  - name: pod_memory_working_set_bytes
+    dimension: pod
+    help: |
+      [ALPHA] Current working set of the pod in bytes
+    kind: gauge
+    labels:
+      - name: namespace
+        value: 'pod.metadata.namespace'
+      - name: pod
+        value: 'pod.metadata.name'
+    value: 'pod.Usage("memory")'
+  # CPU of the node
+  - name: node_cpu_usage_seconds_total
+    dimension: node
+    help: |
+      [ALPHA] Cumulative cpu time consumed by the node in core-seconds
+    kind: counter
+    value: 'node.CumulativeUsage("cpu")'
+  # Memory of the node
+  - name: node_memory_working_set_bytes
+    dimension: node
+    help: |
+      [ALPHA] Current working set of the node in bytes
+    kind: gauge
+    value: 'node.Usage("memory")'
diff --git a/kustomize/metrics/usage/kustomization.yaml b/kustomize/metrics/usage/kustomization.yaml
new file mode 100644
index 00000000..e85d244c
--- /dev/null
+++ b/kustomize/metrics/usage/kustomization.yaml
@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- usage-from-annotation.yaml
+- ../resource
diff --git a/kustomize/metrics/usage/usage-from-annotation.yaml b/kustomize/metrics/usage/usage-from-annotation.yaml
new file mode 100644
index 00000000..11876e75
--- /dev/null
+++ b/kustomize/metrics/usage/usage-from-annotation.yaml
@@ -0,0 +1,17 @@
+apiVersion: kwok.x-k8s.io/v1alpha1
+kind: ClusterResourceUsage
+metadata:
+  name: usage-from-annotation
+spec:
+  usages:
+  - usage:
+      cpu:
+        expression: |
+          "kwok.x-k8s.io/usage-cpu" in pod.metadata.annotations
+          ? Quantity(pod.metadata.annotations["kwok.x-k8s.io/usage-cpu"])
+          : Quantity("1m")
+      memory:
+        expression: |
+          "kwok.x-k8s.io/usage-memory" in pod.metadata.annotations
+          ? Quantity(pod.metadata.annotations["kwok.x-k8s.io/usage-memory"])
+          : Quantity("1Mi")
diff --git a/kustomize/rbac/role.yaml b/kustomize/rbac/role.yaml
index b2d99308..e2c28594 100644
--- a/kustomize/rbac/role.yaml
+++ b/kustomize/rbac/role.yaml
@@ -155,6 +155,25 @@ rules:
   verbs:
   - patch
   - update
+- apiGroups:
+  - kwok.x-k8s.io
+  resources:
+  - clusterresourceusages
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - kwok.x-k8s.io
+  resources:
+  - clusterresourceusages/status
+  verbs:
+  - patch
+  - update
 - apiGroups:
   - kwok.x-k8s.io
   resources:
@@ -231,6 +250,25 @@ rules:
   verbs:
   - patch
   - update
+- apiGroups:
+  - kwok.x-k8s.io
+  resources:
+  - resourceusages
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - kwok.x-k8s.io
+  resources:
+  - resourceusages/status
+  verbs:
+  - patch
+  - update
 - apiGroups:
   - kwok.x-k8s.io
   resources:
diff --git a/pkg/apis/internalversion/cluster_resource_usage_types.go b/pkg/apis/internalversion/cluster_resource_usage_types.go
new file mode 100644
index 00000000..4eb0890d
--- /dev/null
+++ b/pkg/apis/internalversion/cluster_resource_usage_types.go
@@ -0,0 +1,38 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package internalversion
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// ClusterResourceUsage provides cluster-wide resource usage.
+type ClusterResourceUsage struct {
+	// Standard list metadata.
+	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
+	metav1.ObjectMeta
+	// Spec holds spec for cluster resource usage.
+	Spec ClusterResourceUsageSpec
+}
+
+// ClusterResourceUsageSpec holds spec for cluster resource usage.
+type ClusterResourceUsageSpec struct {
+	// Selector is a selector to filter pods to configure.
+	Selector *ObjectSelector
+	// Usages is a list of resource usage for the pod.
+	Usages []ResourceUsageContainer
+}
diff --git a/pkg/apis/internalversion/conversion.go b/pkg/apis/internalversion/conversion.go
index c4b5db99..f507354c 100644
--- a/pkg/apis/internalversion/conversion.go
+++ b/pkg/apis/internalversion/conversion.go
@@ -322,6 +322,50 @@ func ConvertToInternalAttach(in *v1alpha1.Attach) (*Attach, error) {
 	return &out, nil
 }
 
+// ConvertToV1Alpha1ResourceUsage converts an internal version ResourceUsage to a v1alpha1.ResourceUsage.
+func ConvertToV1Alpha1ResourceUsage(in *ResourceUsage) (*v1alpha1.ResourceUsage, error) {
+	var out v1alpha1.ResourceUsage
+	out.APIVersion = v1alpha1.GroupVersion.String()
+	out.Kind = v1alpha1.ResourceUsageKind
+	err := Convert_internalversion_ResourceUsage_To_v1alpha1_ResourceUsage(in, &out, nil)
+	if err != nil {
+		return nil, err
+	}
+	return &out, nil
+}
+
+// ConvertToInternalResourceUsage converts a v1alpha1.ResourceUsage to an internal version.
+func ConvertToInternalResourceUsage(in *v1alpha1.ResourceUsage) (*ResourceUsage, error) {
+	var out ResourceUsage
+	err := Convert_v1alpha1_ResourceUsage_To_internalversion_ResourceUsage(in, &out, nil)
+	if err != nil {
+		return nil, err
+	}
+	return &out, nil
+}
+
+// ConvertToV1Alpha1ClusterResourceUsage converts an internal version ClusterResourceUsage to a v1alpha1.ClusterResourceUsage.
+func ConvertToV1Alpha1ClusterResourceUsage(in *ClusterResourceUsage) (*v1alpha1.ClusterResourceUsage, error) {
+	var out v1alpha1.ClusterResourceUsage
+	out.APIVersion = v1alpha1.GroupVersion.String()
+	out.Kind = v1alpha1.ClusterResourceUsageKind
+	err := Convert_internalversion_ClusterResourceUsage_To_v1alpha1_ClusterResourceUsage(in, &out, nil)
+	if err != nil {
+		return nil, err
+	}
+	return &out, nil
+}
+
+// ConvertToInternalClusterResourceUsage converts a v1alpha1.ClusterResourceUsage to an internal version.
+func ConvertToInternalClusterResourceUsage(in *v1alpha1.ClusterResourceUsage) (*ClusterResourceUsage, error) {
+	var out ClusterResourceUsage
+	err := Convert_v1alpha1_ClusterResourceUsage_To_internalversion_ClusterResourceUsage(in, &out, nil)
+	if err != nil {
+		return nil, err
+	}
+	return &out, nil
+}
+
 // ConvertToV1Alpha1Metric converts an internal version Metric to a v1alpha1.Metric.
 func ConvertToV1Alpha1Metric(in *Metric) (*v1alpha1.Metric, error) {
 	var out v1alpha1.Metric
diff --git a/pkg/apis/internalversion/resource_usage_types.go b/pkg/apis/internalversion/resource_usage_types.go
new file mode 100644
index 00000000..f07988b0
--- /dev/null
+++ b/pkg/apis/internalversion/resource_usage_types.go
@@ -0,0 +1,53 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package internalversion
+
+import (
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// ResourceUsage provides resource usage for a single pod.
+type ResourceUsage struct {
+	// Standard list metadata.
+	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
+	metav1.ObjectMeta
+	// Spec holds spec for resource usage.
+	Spec ResourceUsageSpec
+}
+
+// ResourceUsageSpec holds spec for resource usage.
+type ResourceUsageSpec struct {
+	// Usages is a list of resource usage for the pod.
+	Usages []ResourceUsageContainer
+}
+
+// ResourceUsageContainer holds spec for resource usage container.
+type ResourceUsageContainer struct {
+	// Containers is list of container names.
+	Containers []string
+	// Usage is a list of resource usage for the container.
+	Usage map[string]ResourceUsageValue
+}
+
+// ResourceUsageValue holds value for resource usage.
+type ResourceUsageValue struct {
+	// Value is the value for resource usage.
+	Value *resource.Quantity
+	// Expression is the expression for resource usage.
+	Expression *string
+}
diff --git a/pkg/apis/internalversion/zz_generated.conversion.go b/pkg/apis/internalversion/zz_generated.conversion.go
index a23b0425..2ce5a057 100644
--- a/pkg/apis/internalversion/zz_generated.conversion.go
+++ b/pkg/apis/internalversion/zz_generated.conversion.go
@@ -25,6 +25,7 @@ import (
 	json "encoding/json"
 	unsafe "unsafe"
 
+	resource "k8s.io/apimachinery/pkg/api/resource"
 	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	conversion "k8s.io/apimachinery/pkg/conversion"
 	runtime "k8s.io/apimachinery/pkg/runtime"
@@ -149,6 +150,26 @@ func RegisterConversions(s *runtime.Scheme) error {
 	}); err != nil {
 		return err
 	}
+	if err := s.AddGeneratedConversionFunc((*ClusterResourceUsage)(nil), (*v1alpha1.ClusterResourceUsage)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_ClusterResourceUsage_To_v1alpha1_ClusterResourceUsage(a.(*ClusterResourceUsage), b.(*v1alpha1.ClusterResourceUsage), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.ClusterResourceUsage)(nil), (*ClusterResourceUsage)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_ClusterResourceUsage_To_internalversion_ClusterResourceUsage(a.(*v1alpha1.ClusterResourceUsage), b.(*ClusterResourceUsage), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*ClusterResourceUsageSpec)(nil), (*v1alpha1.ClusterResourceUsageSpec)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_ClusterResourceUsageSpec_To_v1alpha1_ClusterResourceUsageSpec(a.(*ClusterResourceUsageSpec), b.(*v1alpha1.ClusterResourceUsageSpec), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.ClusterResourceUsageSpec)(nil), (*ClusterResourceUsageSpec)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_ClusterResourceUsageSpec_To_internalversion_ClusterResourceUsageSpec(a.(*v1alpha1.ClusterResourceUsageSpec), b.(*ClusterResourceUsageSpec), scope)
+	}); err != nil {
+		return err
+	}
 	if err := s.AddGeneratedConversionFunc((*Component)(nil), (*configv1alpha1.Component)(nil), func(a, b interface{}, scope conversion.Scope) error {
 		return Convert_internalversion_Component_To_v1alpha1_Component(a.(*Component), b.(*configv1alpha1.Component), scope)
 	}); err != nil {
@@ -479,6 +500,46 @@ func RegisterConversions(s *runtime.Scheme) error {
 	}); err != nil {
 		return err
 	}
+	if err := s.AddGeneratedConversionFunc((*ResourceUsage)(nil), (*v1alpha1.ResourceUsage)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_ResourceUsage_To_v1alpha1_ResourceUsage(a.(*ResourceUsage), b.(*v1alpha1.ResourceUsage), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.ResourceUsage)(nil), (*ResourceUsage)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_ResourceUsage_To_internalversion_ResourceUsage(a.(*v1alpha1.ResourceUsage), b.(*ResourceUsage), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*ResourceUsageContainer)(nil), (*v1alpha1.ResourceUsageContainer)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_ResourceUsageContainer_To_v1alpha1_ResourceUsageContainer(a.(*ResourceUsageContainer), b.(*v1alpha1.ResourceUsageContainer), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.ResourceUsageContainer)(nil), (*ResourceUsageContainer)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_ResourceUsageContainer_To_internalversion_ResourceUsageContainer(a.(*v1alpha1.ResourceUsageContainer), b.(*ResourceUsageContainer), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*ResourceUsageSpec)(nil), (*v1alpha1.ResourceUsageSpec)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_ResourceUsageSpec_To_v1alpha1_ResourceUsageSpec(a.(*ResourceUsageSpec), b.(*v1alpha1.ResourceUsageSpec), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.ResourceUsageSpec)(nil), (*ResourceUsageSpec)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_ResourceUsageSpec_To_internalversion_ResourceUsageSpec(a.(*v1alpha1.ResourceUsageSpec), b.(*ResourceUsageSpec), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*ResourceUsageValue)(nil), (*v1alpha1.ResourceUsageValue)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_ResourceUsageValue_To_v1alpha1_ResourceUsageValue(a.(*ResourceUsageValue), b.(*v1alpha1.ResourceUsageValue), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.ResourceUsageValue)(nil), (*ResourceUsageValue)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_ResourceUsageValue_To_internalversion_ResourceUsageValue(a.(*v1alpha1.ResourceUsageValue), b.(*ResourceUsageValue), scope)
+	}); err != nil {
+		return err
+	}
 	if err := s.AddGeneratedConversionFunc((*SecurityContext)(nil), (*v1alpha1.SecurityContext)(nil), func(a, b interface{}, scope conversion.Scope) error {
 		return Convert_internalversion_SecurityContext_To_v1alpha1_SecurityContext(a.(*SecurityContext), b.(*v1alpha1.SecurityContext), scope)
 	}); err != nil {
@@ -926,6 +987,56 @@ func Convert_v1alpha1_ClusterPortForwardSpec_To_internalversion_ClusterPortForwa
 	return autoConvert_v1alpha1_ClusterPortForwardSpec_To_internalversion_ClusterPortForwardSpec(in, out, s)
 }
 
+func autoConvert_internalversion_ClusterResourceUsage_To_v1alpha1_ClusterResourceUsage(in *ClusterResourceUsage, out *v1alpha1.ClusterResourceUsage, s conversion.Scope) error {
+	out.ObjectMeta = in.ObjectMeta
+	if err := Convert_internalversion_ClusterResourceUsageSpec_To_v1alpha1_ClusterResourceUsageSpec(&in.Spec, &out.Spec, s); err != nil {
+		return err
+	}
+	return nil
+}
+
+// Convert_internalversion_ClusterResourceUsage_To_v1alpha1_ClusterResourceUsage is an autogenerated conversion function.
+func Convert_internalversion_ClusterResourceUsage_To_v1alpha1_ClusterResourceUsage(in *ClusterResourceUsage, out *v1alpha1.ClusterResourceUsage, s conversion.Scope) error {
+	return autoConvert_internalversion_ClusterResourceUsage_To_v1alpha1_ClusterResourceUsage(in, out, s)
+}
+
+func autoConvert_v1alpha1_ClusterResourceUsage_To_internalversion_ClusterResourceUsage(in *v1alpha1.ClusterResourceUsage, out *ClusterResourceUsage, s conversion.Scope) error {
+	// INFO: in.TypeMeta opted out of conversion generation
+	out.ObjectMeta = in.ObjectMeta
+	if err := Convert_v1alpha1_ClusterResourceUsageSpec_To_internalversion_ClusterResourceUsageSpec(&in.Spec, &out.Spec, s); err != nil {
+		return err
+	}
+	// INFO: in.Status opted out of conversion generation
+	return nil
+}
+
+// Convert_v1alpha1_ClusterResourceUsage_To_internalversion_ClusterResourceUsage is an autogenerated conversion function.
+func Convert_v1alpha1_ClusterResourceUsage_To_internalversion_ClusterResourceUsage(in *v1alpha1.ClusterResourceUsage, out *ClusterResourceUsage, s conversion.Scope) error {
+	return autoConvert_v1alpha1_ClusterResourceUsage_To_internalversion_ClusterResourceUsage(in, out, s)
+}
+
+func autoConvert_internalversion_ClusterResourceUsageSpec_To_v1alpha1_ClusterResourceUsageSpec(in *ClusterResourceUsageSpec, out *v1alpha1.ClusterResourceUsageSpec, s conversion.Scope) error {
+	out.Selector = (*v1alpha1.ObjectSelector)(unsafe.Pointer(in.Selector))
+	out.Usages = *(*[]v1alpha1.ResourceUsageContainer)(unsafe.Pointer(&in.Usages))
+	return nil
+}
+
+// Convert_internalversion_ClusterResourceUsageSpec_To_v1alpha1_ClusterResourceUsageSpec is an autogenerated conversion function.
+func Convert_internalversion_ClusterResourceUsageSpec_To_v1alpha1_ClusterResourceUsageSpec(in *ClusterResourceUsageSpec, out *v1alpha1.ClusterResourceUsageSpec, s conversion.Scope) error {
+	return autoConvert_internalversion_ClusterResourceUsageSpec_To_v1alpha1_ClusterResourceUsageSpec(in, out, s)
+}
+
+func autoConvert_v1alpha1_ClusterResourceUsageSpec_To_internalversion_ClusterResourceUsageSpec(in *v1alpha1.ClusterResourceUsageSpec, out *ClusterResourceUsageSpec, s conversion.Scope) error {
+	out.Selector = (*ObjectSelector)(unsafe.Pointer(in.Selector))
+	out.Usages = *(*[]ResourceUsageContainer)(unsafe.Pointer(&in.Usages))
+	return nil
+}
+
+// Convert_v1alpha1_ClusterResourceUsageSpec_To_internalversion_ClusterResourceUsageSpec is an autogenerated conversion function.
+func Convert_v1alpha1_ClusterResourceUsageSpec_To_internalversion_ClusterResourceUsageSpec(in *v1alpha1.ClusterResourceUsageSpec, out *ClusterResourceUsageSpec, s conversion.Scope) error {
+	return autoConvert_v1alpha1_ClusterResourceUsageSpec_To_internalversion_ClusterResourceUsageSpec(in, out, s)
+}
+
 func autoConvert_internalversion_Component_To_v1alpha1_Component(in *Component, out *configv1alpha1.Component, s conversion.Scope) error {
 	out.Name = in.Name
 	out.Links = *(*[]string)(unsafe.Pointer(&in.Links))
@@ -2092,6 +2203,98 @@ func Convert_v1alpha1_PortForwardSpec_To_internalversion_PortForwardSpec(in *v1a
 	return autoConvert_v1alpha1_PortForwardSpec_To_internalversion_PortForwardSpec(in, out, s)
 }
 
+func autoConvert_internalversion_ResourceUsage_To_v1alpha1_ResourceUsage(in *ResourceUsage, out *v1alpha1.ResourceUsage, s conversion.Scope) error {
+	out.ObjectMeta = in.ObjectMeta
+	if err := Convert_internalversion_ResourceUsageSpec_To_v1alpha1_ResourceUsageSpec(&in.Spec, &out.Spec, s); err != nil {
+		return err
+	}
+	return nil
+}
+
+// Convert_internalversion_ResourceUsage_To_v1alpha1_ResourceUsage is an autogenerated conversion function.
+func Convert_internalversion_ResourceUsage_To_v1alpha1_ResourceUsage(in *ResourceUsage, out *v1alpha1.ResourceUsage, s conversion.Scope) error {
+	return autoConvert_internalversion_ResourceUsage_To_v1alpha1_ResourceUsage(in, out, s)
+}
+
+func autoConvert_v1alpha1_ResourceUsage_To_internalversion_ResourceUsage(in *v1alpha1.ResourceUsage, out *ResourceUsage, s conversion.Scope) error {
+	// INFO: in.TypeMeta opted out of conversion generation
+	out.ObjectMeta = in.ObjectMeta
+	if err := Convert_v1alpha1_ResourceUsageSpec_To_internalversion_ResourceUsageSpec(&in.Spec, &out.Spec, s); err != nil {
+		return err
+	}
+	// INFO: in.Status opted out of conversion generation
+	return nil
+}
+
+// Convert_v1alpha1_ResourceUsage_To_internalversion_ResourceUsage is an autogenerated conversion function.
+func Convert_v1alpha1_ResourceUsage_To_internalversion_ResourceUsage(in *v1alpha1.ResourceUsage, out *ResourceUsage, s conversion.Scope) error {
+	return autoConvert_v1alpha1_ResourceUsage_To_internalversion_ResourceUsage(in, out, s)
+}
+
+func autoConvert_internalversion_ResourceUsageContainer_To_v1alpha1_ResourceUsageContainer(in *ResourceUsageContainer, out *v1alpha1.ResourceUsageContainer, s conversion.Scope) error {
+	out.Containers = *(*[]string)(unsafe.Pointer(&in.Containers))
+	out.Usage = *(*map[string]v1alpha1.ResourceUsageValue)(unsafe.Pointer(&in.Usage))
+	return nil
+}
+
+// Convert_internalversion_ResourceUsageContainer_To_v1alpha1_ResourceUsageContainer is an autogenerated conversion function.
+func Convert_internalversion_ResourceUsageContainer_To_v1alpha1_ResourceUsageContainer(in *ResourceUsageContainer, out *v1alpha1.ResourceUsageContainer, s conversion.Scope) error {
+	return autoConvert_internalversion_ResourceUsageContainer_To_v1alpha1_ResourceUsageContainer(in, out, s)
+}
+
+func autoConvert_v1alpha1_ResourceUsageContainer_To_internalversion_ResourceUsageContainer(in *v1alpha1.ResourceUsageContainer, out *ResourceUsageContainer, s conversion.Scope) error {
+	out.Containers = *(*[]string)(unsafe.Pointer(&in.Containers))
+	out.Usage = *(*map[string]ResourceUsageValue)(unsafe.Pointer(&in.Usage))
+	return nil
+}
+
+// Convert_v1alpha1_ResourceUsageContainer_To_internalversion_ResourceUsageContainer is an autogenerated conversion function.
+func Convert_v1alpha1_ResourceUsageContainer_To_internalversion_ResourceUsageContainer(in *v1alpha1.ResourceUsageContainer, out *ResourceUsageContainer, s conversion.Scope) error {
+	return autoConvert_v1alpha1_ResourceUsageContainer_To_internalversion_ResourceUsageContainer(in, out, s)
+}
+
+func autoConvert_internalversion_ResourceUsageSpec_To_v1alpha1_ResourceUsageSpec(in *ResourceUsageSpec, out *v1alpha1.ResourceUsageSpec, s conversion.Scope) error {
+	out.Usages = *(*[]v1alpha1.ResourceUsageContainer)(unsafe.Pointer(&in.Usages))
+	return nil
+}
+
+// Convert_internalversion_ResourceUsageSpec_To_v1alpha1_ResourceUsageSpec is an autogenerated conversion function.
+func Convert_internalversion_ResourceUsageSpec_To_v1alpha1_ResourceUsageSpec(in *ResourceUsageSpec, out *v1alpha1.ResourceUsageSpec, s conversion.Scope) error {
+	return autoConvert_internalversion_ResourceUsageSpec_To_v1alpha1_ResourceUsageSpec(in, out, s)
+}
+
+func autoConvert_v1alpha1_ResourceUsageSpec_To_internalversion_ResourceUsageSpec(in *v1alpha1.ResourceUsageSpec, out *ResourceUsageSpec, s conversion.Scope) error {
+	out.Usages = *(*[]ResourceUsageContainer)(unsafe.Pointer(&in.Usages))
+	return nil
+}
+
+// Convert_v1alpha1_ResourceUsageSpec_To_internalversion_ResourceUsageSpec is an autogenerated conversion function.
+func Convert_v1alpha1_ResourceUsageSpec_To_internalversion_ResourceUsageSpec(in *v1alpha1.ResourceUsageSpec, out *ResourceUsageSpec, s conversion.Scope) error {
+	return autoConvert_v1alpha1_ResourceUsageSpec_To_internalversion_ResourceUsageSpec(in, out, s)
+}
+
+func autoConvert_internalversion_ResourceUsageValue_To_v1alpha1_ResourceUsageValue(in *ResourceUsageValue, out *v1alpha1.ResourceUsageValue, s conversion.Scope) error {
+	out.Value = (*resource.Quantity)(unsafe.Pointer(in.Value))
+	out.Expression = (*string)(unsafe.Pointer(in.Expression))
+	return nil
+}
+
+// Convert_internalversion_ResourceUsageValue_To_v1alpha1_ResourceUsageValue is an autogenerated conversion function.
+func Convert_internalversion_ResourceUsageValue_To_v1alpha1_ResourceUsageValue(in *ResourceUsageValue, out *v1alpha1.ResourceUsageValue, s conversion.Scope) error {
+	return autoConvert_internalversion_ResourceUsageValue_To_v1alpha1_ResourceUsageValue(in, out, s)
+}
+
+func autoConvert_v1alpha1_ResourceUsageValue_To_internalversion_ResourceUsageValue(in *v1alpha1.ResourceUsageValue, out *ResourceUsageValue, s conversion.Scope) error {
+	out.Value = (*resource.Quantity)(unsafe.Pointer(in.Value))
+	out.Expression = (*string)(unsafe.Pointer(in.Expression))
+	return nil
+}
+
+// Convert_v1alpha1_ResourceUsageValue_To_internalversion_ResourceUsageValue is an autogenerated conversion function.
+func Convert_v1alpha1_ResourceUsageValue_To_internalversion_ResourceUsageValue(in *v1alpha1.ResourceUsageValue, out *ResourceUsageValue, s conversion.Scope) error {
+	return autoConvert_v1alpha1_ResourceUsageValue_To_internalversion_ResourceUsageValue(in, out, s)
+}
+
 func autoConvert_internalversion_SecurityContext_To_v1alpha1_SecurityContext(in *SecurityContext, out *v1alpha1.SecurityContext, s conversion.Scope) error {
 	out.RunAsUser = (*int64)(unsafe.Pointer(in.RunAsUser))
 	out.RunAsGroup = (*int64)(unsafe.Pointer(in.RunAsGroup))
diff --git a/pkg/apis/internalversion/zz_generated.deepcopy.go b/pkg/apis/internalversion/zz_generated.deepcopy.go
index a176e046..b3d4420a 100644
--- a/pkg/apis/internalversion/zz_generated.deepcopy.go
+++ b/pkg/apis/internalversion/zz_generated.deepcopy.go
@@ -271,6 +271,52 @@ func (in *ClusterPortForwardSpec) DeepCopy() *ClusterPortForwardSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ClusterResourceUsage) DeepCopyInto(out *ClusterResourceUsage) {
+	*out = *in
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterResourceUsage.
+func (in *ClusterResourceUsage) DeepCopy() *ClusterResourceUsage {
+	if in == nil {
+		return nil
+	}
+	out := new(ClusterResourceUsage)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ClusterResourceUsageSpec) DeepCopyInto(out *ClusterResourceUsageSpec) {
+	*out = *in
+	if in.Selector != nil {
+		in, out := &in.Selector, &out.Selector
+		*out = new(ObjectSelector)
+		(*in).DeepCopyInto(*out)
+	}
+	if in.Usages != nil {
+		in, out := &in.Usages, &out.Usages
+		*out = make([]ResourceUsageContainer, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterResourceUsageSpec.
+func (in *ClusterResourceUsageSpec) DeepCopy() *ClusterResourceUsageSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(ClusterResourceUsageSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Component) DeepCopyInto(out *Component) {
 	*out = *in
@@ -995,6 +1041,101 @@ func (in *PortForwardSpec) DeepCopy() *PortForwardSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsage) DeepCopyInto(out *ResourceUsage) {
+	*out = *in
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsage.
+func (in *ResourceUsage) DeepCopy() *ResourceUsage {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsage)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsageContainer) DeepCopyInto(out *ResourceUsageContainer) {
+	*out = *in
+	if in.Containers != nil {
+		in, out := &in.Containers, &out.Containers
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.Usage != nil {
+		in, out := &in.Usage, &out.Usage
+		*out = make(map[string]ResourceUsageValue, len(*in))
+		for key, val := range *in {
+			(*out)[key] = *val.DeepCopy()
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsageContainer.
+func (in *ResourceUsageContainer) DeepCopy() *ResourceUsageContainer {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsageContainer)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsageSpec) DeepCopyInto(out *ResourceUsageSpec) {
+	*out = *in
+	if in.Usages != nil {
+		in, out := &in.Usages, &out.Usages
+		*out = make([]ResourceUsageContainer, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsageSpec.
+func (in *ResourceUsageSpec) DeepCopy() *ResourceUsageSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsageSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsageValue) DeepCopyInto(out *ResourceUsageValue) {
+	*out = *in
+	if in.Value != nil {
+		in, out := &in.Value, &out.Value
+		x := (*in).DeepCopy()
+		*out = &x
+	}
+	if in.Expression != nil {
+		in, out := &in.Expression, &out.Expression
+		*out = new(string)
+		**out = **in
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsageValue.
+func (in *ResourceUsageValue) DeepCopy() *ResourceUsageValue {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsageValue)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *SecurityContext) DeepCopyInto(out *SecurityContext) {
 	*out = *in
diff --git a/pkg/apis/v1alpha1/cluster_resource_usage_types.go b/pkg/apis/v1alpha1/cluster_resource_usage_types.go
new file mode 100644
index 00000000..44bbbf03
--- /dev/null
+++ b/pkg/apis/v1alpha1/cluster_resource_usage_types.go
@@ -0,0 +1,80 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const (
+	// ClusterResourceUsageKind is the kind for ClusterResourceUsage.
+	ClusterResourceUsageKind = "ClusterResourceUsage"
+)
+
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+// +genclient
+// +genclient:nonNamespaced
+// +kubebuilder:subresource:status
+// +kubebuilder:resource:scope=Cluster
+// +kubebuilder:rbac:groups=kwok.x-k8s.io,resources=clusterresourceusages,verbs=create;delete;get;list;patch;update;watch
+// +kubebuilder:rbac:groups=kwok.x-k8s.io,resources=clusterresourceusages/status,verbs=update;patch
+
+// ClusterResourceUsage provides cluster-wide resource usage.
+type ClusterResourceUsage struct {
+	//+k8s:conversion-gen=false
+	metav1.TypeMeta `json:",inline"`
+	// Standard list metadata.
+	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
+	metav1.ObjectMeta `json:"metadata"`
+	// Spec holds spec for cluster resource usage.
+	Spec ClusterResourceUsageSpec `json:"spec"`
+	// Status holds status for cluster resource usage
+	//+k8s:conversion-gen=false
+	Status ClusterResourceUsageStatus `json:"status,omitempty"`
+}
+
+// ClusterResourceUsageStatus holds status for cluster resource usage
+type ClusterResourceUsageStatus struct {
+	// Conditions holds conditions for cluster resource usage
+	// +patchMergeKey=type
+	// +patchStrategy=merge
+	// +listType=map
+	// +listMapKey=type
+	Conditions []Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
+}
+
+// ClusterResourceUsageSpec holds spec for cluster resource usage.
+type ClusterResourceUsageSpec struct {
+	// Selector is a selector to filter pods to configure.
+	Selector *ObjectSelector `json:"selector,omitempty"`
+	// Usages is a list of resource usage for the pod.
+	Usages []ResourceUsageContainer `json:"usages,omitempty"`
+}
+
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+// +kubebuilder:object:root=true
+
+// ClusterResourceUsageList is a list of ClusterResourceUsage.
+type ClusterResourceUsageList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata"`
+	Items           []ClusterResourceUsage `json:"items"`
+}
+
+func init() {
+	SchemeBuilder.Register(&ClusterResourceUsage{}, &ClusterResourceUsageList{})
+}
diff --git a/pkg/apis/v1alpha1/resource_usage_types.go b/pkg/apis/v1alpha1/resource_usage_types.go
new file mode 100644
index 00000000..8915a31c
--- /dev/null
+++ b/pkg/apis/v1alpha1/resource_usage_types.go
@@ -0,0 +1,93 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const (
+	// ResourceUsageKind is the kind for resource usage.
+	ResourceUsageKind = "ResourceUsage"
+)
+
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+// +genclient
+// +kubebuilder:subresource:status
+// +kubebuilder:rbac:groups=kwok.x-k8s.io,resources=resourceusages,verbs=create;delete;get;list;patch;update;watch
+// +kubebuilder:rbac:groups=kwok.x-k8s.io,resources=resourceusages/status,verbs=update;patch
+
+// ResourceUsage provides resource usage for a single pod.
+type ResourceUsage struct {
+	//+k8s:conversion-gen=false
+	metav1.TypeMeta `json:",inline"`
+	// Standard list metadata.
+	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
+	metav1.ObjectMeta `json:"metadata"`
+	// Spec holds spec for resource usage.
+	Spec ResourceUsageSpec `json:"spec"`
+	// Status holds status for resource usage
+	//+k8s:conversion-gen=false
+	Status ResourceUsageStatus `json:"status,omitempty"`
+}
+
+// ResourceUsageStatus holds status for resource usage
+type ResourceUsageStatus struct {
+	// Conditions holds conditions for resource usage
+	// +patchMergeKey=type
+	// +patchStrategy=merge
+	// +listType=map
+	// +listMapKey=type
+	Conditions []Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
+}
+
+// ResourceUsageSpec holds spec for resource usage.
+type ResourceUsageSpec struct {
+	// Usages is a list of resource usage for the pod.
+	Usages []ResourceUsageContainer `json:"usages,omitempty"`
+}
+
+// ResourceUsageContainer holds spec for resource usage container.
+type ResourceUsageContainer struct {
+	// Containers is list of container names.
+	Containers []string `json:"containers,omitempty"`
+	// Usage is a list of resource usage for the container.
+	Usage map[string]ResourceUsageValue `json:"usage,omitempty"`
+}
+
+// ResourceUsageValue holds value for resource usage.
+type ResourceUsageValue struct {
+	// Value is the value for resource usage.
+	Value *resource.Quantity `json:"value,omitempty"`
+	// Expression is the expression for resource usage.
+	Expression *string `json:"expression,omitempty"`
+}
+
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+// +kubebuilder:object:root=true
+
+// ResourceUsageList is a list of ResourceUsage.
+type ResourceUsageList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata"`
+	Items           []ResourceUsage `json:"items"`
+}
+
+func init() {
+	SchemeBuilder.Register(&ResourceUsage{}, &ResourceUsageList{})
+}
diff --git a/pkg/apis/v1alpha1/zz_generated.deepcopy.go b/pkg/apis/v1alpha1/zz_generated.deepcopy.go
index 2f00a3c7..2d726b62 100644
--- a/pkg/apis/v1alpha1/zz_generated.deepcopy.go
+++ b/pkg/apis/v1alpha1/zz_generated.deepcopy.go
@@ -606,6 +606,118 @@ func (in *ClusterPortForwardStatus) DeepCopy() *ClusterPortForwardStatus {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ClusterResourceUsage) DeepCopyInto(out *ClusterResourceUsage) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	in.Status.DeepCopyInto(&out.Status)
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterResourceUsage.
+func (in *ClusterResourceUsage) DeepCopy() *ClusterResourceUsage {
+	if in == nil {
+		return nil
+	}
+	out := new(ClusterResourceUsage)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ClusterResourceUsage) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ClusterResourceUsageList) DeepCopyInto(out *ClusterResourceUsageList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]ClusterResourceUsage, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterResourceUsageList.
+func (in *ClusterResourceUsageList) DeepCopy() *ClusterResourceUsageList {
+	if in == nil {
+		return nil
+	}
+	out := new(ClusterResourceUsageList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ClusterResourceUsageList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ClusterResourceUsageSpec) DeepCopyInto(out *ClusterResourceUsageSpec) {
+	*out = *in
+	if in.Selector != nil {
+		in, out := &in.Selector, &out.Selector
+		*out = new(ObjectSelector)
+		(*in).DeepCopyInto(*out)
+	}
+	if in.Usages != nil {
+		in, out := &in.Usages, &out.Usages
+		*out = make([]ResourceUsageContainer, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterResourceUsageSpec.
+func (in *ClusterResourceUsageSpec) DeepCopy() *ClusterResourceUsageSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(ClusterResourceUsageSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ClusterResourceUsageStatus) DeepCopyInto(out *ClusterResourceUsageStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterResourceUsageStatus.
+func (in *ClusterResourceUsageStatus) DeepCopy() *ClusterResourceUsageStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(ClusterResourceUsageStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Condition) DeepCopyInto(out *Condition) {
 	*out = *in
@@ -1329,6 +1441,167 @@ func (in *PortForwardStatus) DeepCopy() *PortForwardStatus {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsage) DeepCopyInto(out *ResourceUsage) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	in.Status.DeepCopyInto(&out.Status)
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsage.
+func (in *ResourceUsage) DeepCopy() *ResourceUsage {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsage)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ResourceUsage) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsageContainer) DeepCopyInto(out *ResourceUsageContainer) {
+	*out = *in
+	if in.Containers != nil {
+		in, out := &in.Containers, &out.Containers
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.Usage != nil {
+		in, out := &in.Usage, &out.Usage
+		*out = make(map[string]ResourceUsageValue, len(*in))
+		for key, val := range *in {
+			(*out)[key] = *val.DeepCopy()
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsageContainer.
+func (in *ResourceUsageContainer) DeepCopy() *ResourceUsageContainer {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsageContainer)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsageList) DeepCopyInto(out *ResourceUsageList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]ResourceUsage, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsageList.
+func (in *ResourceUsageList) DeepCopy() *ResourceUsageList {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsageList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ResourceUsageList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsageSpec) DeepCopyInto(out *ResourceUsageSpec) {
+	*out = *in
+	if in.Usages != nil {
+		in, out := &in.Usages, &out.Usages
+		*out = make([]ResourceUsageContainer, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsageSpec.
+func (in *ResourceUsageSpec) DeepCopy() *ResourceUsageSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsageSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsageStatus) DeepCopyInto(out *ResourceUsageStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsageStatus.
+func (in *ResourceUsageStatus) DeepCopy() *ResourceUsageStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsageStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceUsageValue) DeepCopyInto(out *ResourceUsageValue) {
+	*out = *in
+	if in.Value != nil {
+		in, out := &in.Value, &out.Value
+		x := (*in).DeepCopy()
+		*out = &x
+	}
+	if in.Expression != nil {
+		in, out := &in.Expression, &out.Expression
+		*out = new(string)
+		**out = **in
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceUsageValue.
+func (in *ResourceUsageValue) DeepCopy() *ResourceUsageValue {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceUsageValue)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *SecurityContext) DeepCopyInto(out *SecurityContext) {
 	*out = *in
diff --git a/pkg/client/clientset/versioned/typed/apis/v1alpha1/apis_client.go b/pkg/client/clientset/versioned/typed/apis/v1alpha1/apis_client.go
index 55276196..0d388c01 100644
--- a/pkg/client/clientset/versioned/typed/apis/v1alpha1/apis_client.go
+++ b/pkg/client/clientset/versioned/typed/apis/v1alpha1/apis_client.go
@@ -33,10 +33,12 @@ type KwokV1alpha1Interface interface {
 	ClusterExecsGetter
 	ClusterLogsGetter
 	ClusterPortForwardsGetter
+	ClusterResourceUsagesGetter
 	ExecsGetter
 	LogsGetter
 	MetricsGetter
 	PortForwardsGetter
+	ResourceUsagesGetter
 	StagesGetter
 }
 
@@ -65,6 +67,10 @@ func (c *KwokV1alpha1Client) ClusterPortForwards() ClusterPortForwardInterface {
 	return newClusterPortForwards(c)
 }
 
+func (c *KwokV1alpha1Client) ClusterResourceUsages() ClusterResourceUsageInterface {
+	return newClusterResourceUsages(c)
+}
+
 func (c *KwokV1alpha1Client) Execs(namespace string) ExecInterface {
 	return newExecs(c, namespace)
 }
@@ -81,6 +87,10 @@ func (c *KwokV1alpha1Client) PortForwards(namespace string) PortForwardInterface
 	return newPortForwards(c, namespace)
 }
 
+func (c *KwokV1alpha1Client) ResourceUsages(namespace string) ResourceUsageInterface {
+	return newResourceUsages(c, namespace)
+}
+
 func (c *KwokV1alpha1Client) Stages() StageInterface {
 	return newStages(c)
 }
diff --git a/pkg/client/clientset/versioned/typed/apis/v1alpha1/clusterresourceusage.go b/pkg/client/clientset/versioned/typed/apis/v1alpha1/clusterresourceusage.go
new file mode 100644
index 00000000..9a408a56
--- /dev/null
+++ b/pkg/client/clientset/versioned/typed/apis/v1alpha1/clusterresourceusage.go
@@ -0,0 +1,184 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	"context"
+	"time"
+
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	rest "k8s.io/client-go/rest"
+	v1alpha1 "sigs.k8s.io/kwok/pkg/apis/v1alpha1"
+	scheme "sigs.k8s.io/kwok/pkg/client/clientset/versioned/scheme"
+)
+
+// ClusterResourceUsagesGetter has a method to return a ClusterResourceUsageInterface.
+// A group's client should implement this interface.
+type ClusterResourceUsagesGetter interface {
+	ClusterResourceUsages() ClusterResourceUsageInterface
+}
+
+// ClusterResourceUsageInterface has methods to work with ClusterResourceUsage resources.
+type ClusterResourceUsageInterface interface {
+	Create(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.CreateOptions) (*v1alpha1.ClusterResourceUsage, error)
+	Update(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.UpdateOptions) (*v1alpha1.ClusterResourceUsage, error)
+	UpdateStatus(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.UpdateOptions) (*v1alpha1.ClusterResourceUsage, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*v1alpha1.ClusterResourceUsage, error)
+	List(ctx context.Context, opts v1.ListOptions) (*v1alpha1.ClusterResourceUsageList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.ClusterResourceUsage, err error)
+	ClusterResourceUsageExpansion
+}
+
+// clusterResourceUsages implements ClusterResourceUsageInterface
+type clusterResourceUsages struct {
+	client rest.Interface
+}
+
+// newClusterResourceUsages returns a ClusterResourceUsages
+func newClusterResourceUsages(c *KwokV1alpha1Client) *clusterResourceUsages {
+	return &clusterResourceUsages{
+		client: c.RESTClient(),
+	}
+}
+
+// Get takes name of the clusterResourceUsage, and returns the corresponding clusterResourceUsage object, and an error if there is any.
+func (c *clusterResourceUsages) Get(ctx context.Context, name string, options v1.GetOptions) (result *v1alpha1.ClusterResourceUsage, err error) {
+	result = &v1alpha1.ClusterResourceUsage{}
+	err = c.client.Get().
+		Resource("clusterresourceusages").
+		Name(name).
+		VersionedParams(&options, scheme.ParameterCodec).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// List takes label and field selectors, and returns the list of ClusterResourceUsages that match those selectors.
+func (c *clusterResourceUsages) List(ctx context.Context, opts v1.ListOptions) (result *v1alpha1.ClusterResourceUsageList, err error) {
+	var timeout time.Duration
+	if opts.TimeoutSeconds != nil {
+		timeout = time.Duration(*opts.TimeoutSeconds) * time.Second
+	}
+	result = &v1alpha1.ClusterResourceUsageList{}
+	err = c.client.Get().
+		Resource("clusterresourceusages").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Watch returns a watch.Interface that watches the requested clusterResourceUsages.
+func (c *clusterResourceUsages) Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) {
+	var timeout time.Duration
+	if opts.TimeoutSeconds != nil {
+		timeout = time.Duration(*opts.TimeoutSeconds) * time.Second
+	}
+	opts.Watch = true
+	return c.client.Get().
+		Resource("clusterresourceusages").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Watch(ctx)
+}
+
+// Create takes the representation of a clusterResourceUsage and creates it.  Returns the server's representation of the clusterResourceUsage, and an error, if there is any.
+func (c *clusterResourceUsages) Create(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.CreateOptions) (result *v1alpha1.ClusterResourceUsage, err error) {
+	result = &v1alpha1.ClusterResourceUsage{}
+	err = c.client.Post().
+		Resource("clusterresourceusages").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(clusterResourceUsage).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Update takes the representation of a clusterResourceUsage and updates it. Returns the server's representation of the clusterResourceUsage, and an error, if there is any.
+func (c *clusterResourceUsages) Update(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.UpdateOptions) (result *v1alpha1.ClusterResourceUsage, err error) {
+	result = &v1alpha1.ClusterResourceUsage{}
+	err = c.client.Put().
+		Resource("clusterresourceusages").
+		Name(clusterResourceUsage.Name).
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(clusterResourceUsage).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// UpdateStatus was generated because the type contains a Status member.
+// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+func (c *clusterResourceUsages) UpdateStatus(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.UpdateOptions) (result *v1alpha1.ClusterResourceUsage, err error) {
+	result = &v1alpha1.ClusterResourceUsage{}
+	err = c.client.Put().
+		Resource("clusterresourceusages").
+		Name(clusterResourceUsage.Name).
+		SubResource("status").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(clusterResourceUsage).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Delete takes name of the clusterResourceUsage and deletes it. Returns an error if one occurs.
+func (c *clusterResourceUsages) Delete(ctx context.Context, name string, opts v1.DeleteOptions) error {
+	return c.client.Delete().
+		Resource("clusterresourceusages").
+		Name(name).
+		Body(&opts).
+		Do(ctx).
+		Error()
+}
+
+// DeleteCollection deletes a collection of objects.
+func (c *clusterResourceUsages) DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error {
+	var timeout time.Duration
+	if listOpts.TimeoutSeconds != nil {
+		timeout = time.Duration(*listOpts.TimeoutSeconds) * time.Second
+	}
+	return c.client.Delete().
+		Resource("clusterresourceusages").
+		VersionedParams(&listOpts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Body(&opts).
+		Do(ctx).
+		Error()
+}
+
+// Patch applies the patch and returns the patched clusterResourceUsage.
+func (c *clusterResourceUsages) Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.ClusterResourceUsage, err error) {
+	result = &v1alpha1.ClusterResourceUsage{}
+	err = c.client.Patch(pt).
+		Resource("clusterresourceusages").
+		Name(name).
+		SubResource(subresources...).
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(data).
+		Do(ctx).
+		Into(result)
+	return
+}
diff --git a/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_apis_client.go b/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_apis_client.go
index 4af83899..e5ef1f63 100644
--- a/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_apis_client.go
+++ b/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_apis_client.go
@@ -48,6 +48,10 @@ func (c *FakeKwokV1alpha1) ClusterPortForwards() v1alpha1.ClusterPortForwardInte
 	return &FakeClusterPortForwards{c}
 }
 
+func (c *FakeKwokV1alpha1) ClusterResourceUsages() v1alpha1.ClusterResourceUsageInterface {
+	return &FakeClusterResourceUsages{c}
+}
+
 func (c *FakeKwokV1alpha1) Execs(namespace string) v1alpha1.ExecInterface {
 	return &FakeExecs{c, namespace}
 }
@@ -64,6 +68,10 @@ func (c *FakeKwokV1alpha1) PortForwards(namespace string) v1alpha1.PortForwardIn
 	return &FakePortForwards{c, namespace}
 }
 
+func (c *FakeKwokV1alpha1) ResourceUsages(namespace string) v1alpha1.ResourceUsageInterface {
+	return &FakeResourceUsages{c, namespace}
+}
+
 func (c *FakeKwokV1alpha1) Stages() v1alpha1.StageInterface {
 	return &FakeStages{c}
 }
diff --git a/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_clusterresourceusage.go b/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_clusterresourceusage.go
new file mode 100644
index 00000000..a12340ca
--- /dev/null
+++ b/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_clusterresourceusage.go
@@ -0,0 +1,132 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	"context"
+
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	testing "k8s.io/client-go/testing"
+	v1alpha1 "sigs.k8s.io/kwok/pkg/apis/v1alpha1"
+)
+
+// FakeClusterResourceUsages implements ClusterResourceUsageInterface
+type FakeClusterResourceUsages struct {
+	Fake *FakeKwokV1alpha1
+}
+
+var clusterresourceusagesResource = v1alpha1.SchemeGroupVersion.WithResource("clusterresourceusages")
+
+var clusterresourceusagesKind = v1alpha1.SchemeGroupVersion.WithKind("ClusterResourceUsage")
+
+// Get takes name of the clusterResourceUsage, and returns the corresponding clusterResourceUsage object, and an error if there is any.
+func (c *FakeClusterResourceUsages) Get(ctx context.Context, name string, options v1.GetOptions) (result *v1alpha1.ClusterResourceUsage, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewRootGetAction(clusterresourceusagesResource, name), &v1alpha1.ClusterResourceUsage{})
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ClusterResourceUsage), err
+}
+
+// List takes label and field selectors, and returns the list of ClusterResourceUsages that match those selectors.
+func (c *FakeClusterResourceUsages) List(ctx context.Context, opts v1.ListOptions) (result *v1alpha1.ClusterResourceUsageList, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewRootListAction(clusterresourceusagesResource, clusterresourceusagesKind, opts), &v1alpha1.ClusterResourceUsageList{})
+	if obj == nil {
+		return nil, err
+	}
+
+	label, _, _ := testing.ExtractFromListOptions(opts)
+	if label == nil {
+		label = labels.Everything()
+	}
+	list := &v1alpha1.ClusterResourceUsageList{ListMeta: obj.(*v1alpha1.ClusterResourceUsageList).ListMeta}
+	for _, item := range obj.(*v1alpha1.ClusterResourceUsageList).Items {
+		if label.Matches(labels.Set(item.Labels)) {
+			list.Items = append(list.Items, item)
+		}
+	}
+	return list, err
+}
+
+// Watch returns a watch.Interface that watches the requested clusterResourceUsages.
+func (c *FakeClusterResourceUsages) Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) {
+	return c.Fake.
+		InvokesWatch(testing.NewRootWatchAction(clusterresourceusagesResource, opts))
+}
+
+// Create takes the representation of a clusterResourceUsage and creates it.  Returns the server's representation of the clusterResourceUsage, and an error, if there is any.
+func (c *FakeClusterResourceUsages) Create(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.CreateOptions) (result *v1alpha1.ClusterResourceUsage, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewRootCreateAction(clusterresourceusagesResource, clusterResourceUsage), &v1alpha1.ClusterResourceUsage{})
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ClusterResourceUsage), err
+}
+
+// Update takes the representation of a clusterResourceUsage and updates it. Returns the server's representation of the clusterResourceUsage, and an error, if there is any.
+func (c *FakeClusterResourceUsages) Update(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.UpdateOptions) (result *v1alpha1.ClusterResourceUsage, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewRootUpdateAction(clusterresourceusagesResource, clusterResourceUsage), &v1alpha1.ClusterResourceUsage{})
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ClusterResourceUsage), err
+}
+
+// UpdateStatus was generated because the type contains a Status member.
+// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+func (c *FakeClusterResourceUsages) UpdateStatus(ctx context.Context, clusterResourceUsage *v1alpha1.ClusterResourceUsage, opts v1.UpdateOptions) (*v1alpha1.ClusterResourceUsage, error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewRootUpdateSubresourceAction(clusterresourceusagesResource, "status", clusterResourceUsage), &v1alpha1.ClusterResourceUsage{})
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ClusterResourceUsage), err
+}
+
+// Delete takes name of the clusterResourceUsage and deletes it. Returns an error if one occurs.
+func (c *FakeClusterResourceUsages) Delete(ctx context.Context, name string, opts v1.DeleteOptions) error {
+	_, err := c.Fake.
+		Invokes(testing.NewRootDeleteActionWithOptions(clusterresourceusagesResource, name, opts), &v1alpha1.ClusterResourceUsage{})
+	return err
+}
+
+// DeleteCollection deletes a collection of objects.
+func (c *FakeClusterResourceUsages) DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error {
+	action := testing.NewRootDeleteCollectionAction(clusterresourceusagesResource, listOpts)
+
+	_, err := c.Fake.Invokes(action, &v1alpha1.ClusterResourceUsageList{})
+	return err
+}
+
+// Patch applies the patch and returns the patched clusterResourceUsage.
+func (c *FakeClusterResourceUsages) Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.ClusterResourceUsage, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewRootPatchSubresourceAction(clusterresourceusagesResource, name, pt, data, subresources...), &v1alpha1.ClusterResourceUsage{})
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ClusterResourceUsage), err
+}
diff --git a/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_resourceusage.go b/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_resourceusage.go
new file mode 100644
index 00000000..9611a710
--- /dev/null
+++ b/pkg/client/clientset/versioned/typed/apis/v1alpha1/fake/fake_resourceusage.go
@@ -0,0 +1,141 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	"context"
+
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	testing "k8s.io/client-go/testing"
+	v1alpha1 "sigs.k8s.io/kwok/pkg/apis/v1alpha1"
+)
+
+// FakeResourceUsages implements ResourceUsageInterface
+type FakeResourceUsages struct {
+	Fake *FakeKwokV1alpha1
+	ns   string
+}
+
+var resourceusagesResource = v1alpha1.SchemeGroupVersion.WithResource("resourceusages")
+
+var resourceusagesKind = v1alpha1.SchemeGroupVersion.WithKind("ResourceUsage")
+
+// Get takes name of the resourceUsage, and returns the corresponding resourceUsage object, and an error if there is any.
+func (c *FakeResourceUsages) Get(ctx context.Context, name string, options v1.GetOptions) (result *v1alpha1.ResourceUsage, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewGetAction(resourceusagesResource, c.ns, name), &v1alpha1.ResourceUsage{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ResourceUsage), err
+}
+
+// List takes label and field selectors, and returns the list of ResourceUsages that match those selectors.
+func (c *FakeResourceUsages) List(ctx context.Context, opts v1.ListOptions) (result *v1alpha1.ResourceUsageList, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewListAction(resourceusagesResource, resourceusagesKind, c.ns, opts), &v1alpha1.ResourceUsageList{})
+
+	if obj == nil {
+		return nil, err
+	}
+
+	label, _, _ := testing.ExtractFromListOptions(opts)
+	if label == nil {
+		label = labels.Everything()
+	}
+	list := &v1alpha1.ResourceUsageList{ListMeta: obj.(*v1alpha1.ResourceUsageList).ListMeta}
+	for _, item := range obj.(*v1alpha1.ResourceUsageList).Items {
+		if label.Matches(labels.Set(item.Labels)) {
+			list.Items = append(list.Items, item)
+		}
+	}
+	return list, err
+}
+
+// Watch returns a watch.Interface that watches the requested resourceUsages.
+func (c *FakeResourceUsages) Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) {
+	return c.Fake.
+		InvokesWatch(testing.NewWatchAction(resourceusagesResource, c.ns, opts))
+
+}
+
+// Create takes the representation of a resourceUsage and creates it.  Returns the server's representation of the resourceUsage, and an error, if there is any.
+func (c *FakeResourceUsages) Create(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.CreateOptions) (result *v1alpha1.ResourceUsage, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewCreateAction(resourceusagesResource, c.ns, resourceUsage), &v1alpha1.ResourceUsage{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ResourceUsage), err
+}
+
+// Update takes the representation of a resourceUsage and updates it. Returns the server's representation of the resourceUsage, and an error, if there is any.
+func (c *FakeResourceUsages) Update(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.UpdateOptions) (result *v1alpha1.ResourceUsage, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewUpdateAction(resourceusagesResource, c.ns, resourceUsage), &v1alpha1.ResourceUsage{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ResourceUsage), err
+}
+
+// UpdateStatus was generated because the type contains a Status member.
+// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+func (c *FakeResourceUsages) UpdateStatus(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.UpdateOptions) (*v1alpha1.ResourceUsage, error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewUpdateSubresourceAction(resourceusagesResource, "status", c.ns, resourceUsage), &v1alpha1.ResourceUsage{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ResourceUsage), err
+}
+
+// Delete takes name of the resourceUsage and deletes it. Returns an error if one occurs.
+func (c *FakeResourceUsages) Delete(ctx context.Context, name string, opts v1.DeleteOptions) error {
+	_, err := c.Fake.
+		Invokes(testing.NewDeleteActionWithOptions(resourceusagesResource, c.ns, name, opts), &v1alpha1.ResourceUsage{})
+
+	return err
+}
+
+// DeleteCollection deletes a collection of objects.
+func (c *FakeResourceUsages) DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error {
+	action := testing.NewDeleteCollectionAction(resourceusagesResource, c.ns, listOpts)
+
+	_, err := c.Fake.Invokes(action, &v1alpha1.ResourceUsageList{})
+	return err
+}
+
+// Patch applies the patch and returns the patched resourceUsage.
+func (c *FakeResourceUsages) Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.ResourceUsage, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewPatchSubresourceAction(resourceusagesResource, c.ns, name, pt, data, subresources...), &v1alpha1.ResourceUsage{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.ResourceUsage), err
+}
diff --git a/pkg/client/clientset/versioned/typed/apis/v1alpha1/generated_expansion.go b/pkg/client/clientset/versioned/typed/apis/v1alpha1/generated_expansion.go
index 5925af6a..aba6b743 100644
--- a/pkg/client/clientset/versioned/typed/apis/v1alpha1/generated_expansion.go
+++ b/pkg/client/clientset/versioned/typed/apis/v1alpha1/generated_expansion.go
@@ -28,6 +28,8 @@ type ClusterLogsExpansion interface{}
 
 type ClusterPortForwardExpansion interface{}
 
+type ClusterResourceUsageExpansion interface{}
+
 type ExecExpansion interface{}
 
 type LogsExpansion interface{}
@@ -36,4 +38,6 @@ type MetricExpansion interface{}
 
 type PortForwardExpansion interface{}
 
+type ResourceUsageExpansion interface{}
+
 type StageExpansion interface{}
diff --git a/pkg/client/clientset/versioned/typed/apis/v1alpha1/resourceusage.go b/pkg/client/clientset/versioned/typed/apis/v1alpha1/resourceusage.go
new file mode 100644
index 00000000..e3c7abab
--- /dev/null
+++ b/pkg/client/clientset/versioned/typed/apis/v1alpha1/resourceusage.go
@@ -0,0 +1,195 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	"context"
+	"time"
+
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	rest "k8s.io/client-go/rest"
+	v1alpha1 "sigs.k8s.io/kwok/pkg/apis/v1alpha1"
+	scheme "sigs.k8s.io/kwok/pkg/client/clientset/versioned/scheme"
+)
+
+// ResourceUsagesGetter has a method to return a ResourceUsageInterface.
+// A group's client should implement this interface.
+type ResourceUsagesGetter interface {
+	ResourceUsages(namespace string) ResourceUsageInterface
+}
+
+// ResourceUsageInterface has methods to work with ResourceUsage resources.
+type ResourceUsageInterface interface {
+	Create(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.CreateOptions) (*v1alpha1.ResourceUsage, error)
+	Update(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.UpdateOptions) (*v1alpha1.ResourceUsage, error)
+	UpdateStatus(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.UpdateOptions) (*v1alpha1.ResourceUsage, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*v1alpha1.ResourceUsage, error)
+	List(ctx context.Context, opts v1.ListOptions) (*v1alpha1.ResourceUsageList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.ResourceUsage, err error)
+	ResourceUsageExpansion
+}
+
+// resourceUsages implements ResourceUsageInterface
+type resourceUsages struct {
+	client rest.Interface
+	ns     string
+}
+
+// newResourceUsages returns a ResourceUsages
+func newResourceUsages(c *KwokV1alpha1Client, namespace string) *resourceUsages {
+	return &resourceUsages{
+		client: c.RESTClient(),
+		ns:     namespace,
+	}
+}
+
+// Get takes name of the resourceUsage, and returns the corresponding resourceUsage object, and an error if there is any.
+func (c *resourceUsages) Get(ctx context.Context, name string, options v1.GetOptions) (result *v1alpha1.ResourceUsage, err error) {
+	result = &v1alpha1.ResourceUsage{}
+	err = c.client.Get().
+		Namespace(c.ns).
+		Resource("resourceusages").
+		Name(name).
+		VersionedParams(&options, scheme.ParameterCodec).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// List takes label and field selectors, and returns the list of ResourceUsages that match those selectors.
+func (c *resourceUsages) List(ctx context.Context, opts v1.ListOptions) (result *v1alpha1.ResourceUsageList, err error) {
+	var timeout time.Duration
+	if opts.TimeoutSeconds != nil {
+		timeout = time.Duration(*opts.TimeoutSeconds) * time.Second
+	}
+	result = &v1alpha1.ResourceUsageList{}
+	err = c.client.Get().
+		Namespace(c.ns).
+		Resource("resourceusages").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Watch returns a watch.Interface that watches the requested resourceUsages.
+func (c *resourceUsages) Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) {
+	var timeout time.Duration
+	if opts.TimeoutSeconds != nil {
+		timeout = time.Duration(*opts.TimeoutSeconds) * time.Second
+	}
+	opts.Watch = true
+	return c.client.Get().
+		Namespace(c.ns).
+		Resource("resourceusages").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Watch(ctx)
+}
+
+// Create takes the representation of a resourceUsage and creates it.  Returns the server's representation of the resourceUsage, and an error, if there is any.
+func (c *resourceUsages) Create(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.CreateOptions) (result *v1alpha1.ResourceUsage, err error) {
+	result = &v1alpha1.ResourceUsage{}
+	err = c.client.Post().
+		Namespace(c.ns).
+		Resource("resourceusages").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(resourceUsage).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Update takes the representation of a resourceUsage and updates it. Returns the server's representation of the resourceUsage, and an error, if there is any.
+func (c *resourceUsages) Update(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.UpdateOptions) (result *v1alpha1.ResourceUsage, err error) {
+	result = &v1alpha1.ResourceUsage{}
+	err = c.client.Put().
+		Namespace(c.ns).
+		Resource("resourceusages").
+		Name(resourceUsage.Name).
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(resourceUsage).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// UpdateStatus was generated because the type contains a Status member.
+// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+func (c *resourceUsages) UpdateStatus(ctx context.Context, resourceUsage *v1alpha1.ResourceUsage, opts v1.UpdateOptions) (result *v1alpha1.ResourceUsage, err error) {
+	result = &v1alpha1.ResourceUsage{}
+	err = c.client.Put().
+		Namespace(c.ns).
+		Resource("resourceusages").
+		Name(resourceUsage.Name).
+		SubResource("status").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(resourceUsage).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Delete takes name of the resourceUsage and deletes it. Returns an error if one occurs.
+func (c *resourceUsages) Delete(ctx context.Context, name string, opts v1.DeleteOptions) error {
+	return c.client.Delete().
+		Namespace(c.ns).
+		Resource("resourceusages").
+		Name(name).
+		Body(&opts).
+		Do(ctx).
+		Error()
+}
+
+// DeleteCollection deletes a collection of objects.
+func (c *resourceUsages) DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error {
+	var timeout time.Duration
+	if listOpts.TimeoutSeconds != nil {
+		timeout = time.Duration(*listOpts.TimeoutSeconds) * time.Second
+	}
+	return c.client.Delete().
+		Namespace(c.ns).
+		Resource("resourceusages").
+		VersionedParams(&listOpts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Body(&opts).
+		Do(ctx).
+		Error()
+}
+
+// Patch applies the patch and returns the patched resourceUsage.
+func (c *resourceUsages) Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.ResourceUsage, err error) {
+	result = &v1alpha1.ResourceUsage{}
+	err = c.client.Patch(pt).
+		Namespace(c.ns).
+		Resource("resourceusages").
+		Name(name).
+		SubResource(subresources...).
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(data).
+		Do(ctx).
+		Into(result)
+	return
+}
diff --git a/pkg/config/config.go b/pkg/config/config.go
index 835a177b..df4ac0c5 100644
--- a/pkg/config/config.go
+++ b/pkg/config/config.go
@@ -161,6 +161,18 @@ var configHandlers = map[string]configHandler{
 		MutateToInternal: mutateToInternalConfig(internalversion.ConvertToInternalClusterAttach),
 		MutateToVersiond: mutateToVersiondConfig(internalversion.ConvertToV1Alpha1ClusterAttach),
 	},
+	v1alpha1.ResourceUsageKind: {
+		Unmarshal:        unmarshalConfig[*v1alpha1.ResourceUsage],
+		Marshal:          marshalConfig,
+		MutateToInternal: mutateToInternalConfig(internalversion.ConvertToInternalResourceUsage),
+		MutateToVersiond: mutateToVersiondConfig(internalversion.ConvertToV1Alpha1ResourceUsage),
+	},
+	v1alpha1.ClusterResourceUsageKind: {
+		Unmarshal:        unmarshalConfig[*v1alpha1.ClusterResourceUsage],
+		Marshal:          marshalConfig,
+		MutateToInternal: mutateToInternalConfig(internalversion.ConvertToInternalClusterResourceUsage),
+		MutateToVersiond: mutateToVersiondConfig(internalversion.ConvertToV1Alpha1ClusterResourceUsage),
+	},
 	v1alpha1.MetricKind: {
 		Unmarshal:        unmarshalConfig[*v1alpha1.Metric],
 		Marshal:          marshalConfig,
diff --git a/pkg/kwok/cmd/root.go b/pkg/kwok/cmd/root.go
index 15160519..5a2cd09c 100644
--- a/pkg/kwok/cmd/root.go
+++ b/pkg/kwok/cmd/root.go
@@ -102,16 +102,18 @@ func NewCommand(ctx context.Context) *cobra.Command {
 }
 
 var crdDefines = map[string]struct{}{
-	v1alpha1.StageKind:              {},
-	v1alpha1.AttachKind:             {},
-	v1alpha1.ClusterAttachKind:      {},
-	v1alpha1.ExecKind:               {},
-	v1alpha1.ClusterExecKind:        {},
-	v1alpha1.PortForwardKind:        {},
-	v1alpha1.ClusterPortForwardKind: {},
-	v1alpha1.LogsKind:               {},
-	v1alpha1.ClusterLogsKind:        {},
-	v1alpha1.MetricKind:             {},
+	v1alpha1.StageKind:                {},
+	v1alpha1.AttachKind:               {},
+	v1alpha1.ClusterAttachKind:        {},
+	v1alpha1.ExecKind:                 {},
+	v1alpha1.ClusterExecKind:          {},
+	v1alpha1.PortForwardKind:          {},
+	v1alpha1.ClusterPortForwardKind:   {},
+	v1alpha1.LogsKind:                 {},
+	v1alpha1.ClusterLogsKind:          {},
+	v1alpha1.ResourceUsageKind:        {},
+	v1alpha1.ClusterResourceUsageKind: {},
+	v1alpha1.MetricKind:               {},
 }
 
 func runE(ctx context.Context, flags *flagpole) error {
@@ -338,6 +340,18 @@ func startServer(ctx context.Context, flags *flagpole, ctr *controllers.Controll
 			return err
 		}
 
+		clusterResourceUsages := config.FilterWithTypeFromContext[*internalversion.ClusterResourceUsage](ctx)
+		err = checkConfigOrCRD(flags.Options.EnableCRDs, v1alpha1.ClusterResourceUsageKind, clusterResourceUsages)
+		if err != nil {
+			return err
+		}
+
+		resourceUsages := config.FilterWithTypeFromContext[*internalversion.ResourceUsage](ctx)
+		err = checkConfigOrCRD(flags.Options.EnableCRDs, v1alpha1.ResourceUsageKind, resourceUsages)
+		if err != nil {
+			return err
+		}
+
 		metrics := config.FilterWithTypeFromContext[*internalversion.Metric](ctx)
 		err = checkConfigOrCRD(flags.Options.EnableCRDs, v1alpha1.MetricKind, metrics)
 		if err != nil {
@@ -345,20 +359,22 @@ func startServer(ctx context.Context, flags *flagpole, ctr *controllers.Controll
 		}
 
 		conf := server.Config{
-			TypedKwokClient:     typedKwokClient,
-			EnableCRDs:          flags.Options.EnableCRDs,
-			ClusterPortForwards: clusterPortForwards,
-			PortForwards:        portForwards,
-			ClusterExecs:        clusterExecs,
-			Execs:               execs,
-			ClusterLogs:         clusterLogs,
-			Logs:                logs,
-			ClusterAttaches:     clusterAttaches,
-			Attaches:            attaches,
-			Metrics:             metrics,
-			DataSource:          ctr,
-			NodeCacheGetter:     ctr.GetNodeCache(),
-			PodCacheGetter:      ctr.GetPodCache(),
+			TypedKwokClient:       typedKwokClient,
+			EnableCRDs:            flags.Options.EnableCRDs,
+			ClusterPortForwards:   clusterPortForwards,
+			PortForwards:          portForwards,
+			ClusterExecs:          clusterExecs,
+			Execs:                 execs,
+			ClusterLogs:           clusterLogs,
+			Logs:                  logs,
+			ClusterAttaches:       clusterAttaches,
+			Attaches:              attaches,
+			ClusterResourceUsages: clusterResourceUsages,
+			ResourceUsages:        resourceUsages,
+			Metrics:               metrics,
+			DataSource:            ctr,
+			NodeCacheGetter:       ctr.GetNodeCache(),
+			PodCacheGetter:        ctr.GetPodCache(),
 		}
 		svc, err := server.NewServer(conf)
 		if err != nil {
diff --git a/pkg/kwok/controllers/node_controller.go b/pkg/kwok/controllers/node_controller.go
index a3920b8b..8faf31f2 100644
--- a/pkg/kwok/controllers/node_controller.go
+++ b/pkg/kwok/controllers/node_controller.go
@@ -23,9 +23,9 @@ import (
 	"fmt"
 	"net"
 	"sync/atomic"
+	"time"
 
 	corev1 "k8s.io/api/core/v1"
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/types"
@@ -43,6 +43,7 @@ import (
 	"sigs.k8s.io/kwok/pkg/utils/informer"
 	"sigs.k8s.io/kwok/pkg/utils/maps"
 	"sigs.k8s.io/kwok/pkg/utils/queue"
+	"sigs.k8s.io/kwok/pkg/utils/wait"
 )
 
 var (
@@ -98,8 +99,9 @@ type NodeController struct {
 	preprocessChan                        chan *corev1.Node
 	playStageParallelism                  uint
 	lifecycle                             resources.Getter[Lifecycle]
-	delayQueue                            queue.DelayingQueue[resourceStageJob[*corev1.Node]]
+	delayQueue                            queue.WeightDelayingQueue[resourceStageJob[*corev1.Node]]
 	delayQueueMapping                     maps.SyncMap[string, resourceStageJob[*corev1.Node]]
+	backoff                               wait.Backoff
 	recorder                              record.EventRecorder
 	readOnlyFunc                          func(nodeName string) bool
 	enableMetrics                         bool
@@ -159,7 +161,8 @@ func NewNodeController(conf NodeControllerConfig) (*NodeController, error) {
 		nodeIP:                                conf.NodeIP,
 		nodeName:                              conf.NodeName,
 		nodePort:                              conf.NodePort,
-		delayQueue:                            queue.NewDelayingQueue[resourceStageJob[*corev1.Node]](conf.Clock),
+		delayQueue:                            queue.NewWeightDelayingQueue[resourceStageJob[*corev1.Node]](conf.Clock),
+		backoff:                               defaultBackoff(),
 		lifecycle:                             conf.Lifecycle,
 		playStageParallelism:                  conf.PlayStageParallelism,
 		preprocessChan:                        make(chan *corev1.Node),
@@ -264,7 +267,7 @@ loop:
 	logger.Info("Stop watch nodes")
 }
 
-// finalizersModify modify finalizers of node
+// finalizersModify modifies the finalizers of a node
 func (c *NodeController) finalizersModify(ctx context.Context, node *corev1.Node, finalizers *internalversion.StageFinalizers) (*corev1.Node, error) {
 	ops := finalizersModify(node.Finalizers, finalizers)
 	if len(ops) == 0 {
@@ -282,12 +285,6 @@ func (c *NodeController) finalizersModify(ctx context.Context, node *corev1.Node
 
 	result, err := c.typedClient.CoreV1().Nodes().Patch(ctx, node.Name, types.JSONPatchType, data, metav1.PatchOptions{})
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Patch node finalizers",
-				"err", err,
-			)
-			return nil, nil
-		}
 		return nil, err
 	}
 	logger.Info("Patch node finalizers")
@@ -303,12 +300,6 @@ func (c *NodeController) deleteResource(ctx context.Context, node *corev1.Node)
 
 	err := c.typedClient.CoreV1().Nodes().Delete(ctx, node.Name, deleteOpt)
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Delete node",
-				"err", err,
-			)
-			return nil
-		}
 		return err
 	}
 
@@ -390,31 +381,52 @@ func (c *NodeController) preprocess(ctx context.Context, node *corev1.Node) erro
 	}
 
 	item := resourceStageJob[*corev1.Node]{
-		Resource: node,
-		Stage:    stage,
-		Key:      key,
-	}
-
-	c.delayQueue.AddAfter(item, delay)
-	c.delayQueueMapping.Store(key, item)
-
+		Resource:   node,
+		Stage:      stage,
+		Key:        key,
+		RetryCount: new(uint64),
+	}
+	// we add a normal(fresh) stage job with weight 0,
+	// resulting in that it will always be processed with high priority compared to those retry ones
+	c.addStageJob(item, delay, 0)
 	return nil
 }
 
 // playStageWorker receives the resource from the playStageChan and play the stage
 func (c *NodeController) playStageWorker(ctx context.Context) {
+	logger := log.FromContext(ctx)
+
 	for ctx.Err() == nil {
 		node, ok := c.delayQueue.GetOrWaitWithDone(ctx.Done())
 		if !ok {
 			return
 		}
 		c.delayQueueMapping.Delete(node.Key)
-		c.playStage(ctx, node.Resource, node.Stage)
+		needRetry, err := c.playStage(ctx, node.Resource, node.Stage)
+		if err != nil {
+			logger.Error("failed to apply stage", err,
+				"node", node.Key,
+				"stage", node.Stage.Name(),
+			)
+		}
+		if needRetry {
+			retryCount := atomic.AddUint64(node.RetryCount, 1) - 1
+			logger.Info("retrying for failed job",
+				"node", node.Key,
+				"stage", node.Stage.Name(),
+				"retry", retryCount,
+			)
+			// for failed jobs, we re-push them into the queue with a lower weight
+			// and a backoff period to avoid blocking normal tasks
+			retryDelay := backoffDelayByStep(retryCount, c.backoff)
+			c.addStageJob(node, retryDelay, 1)
+		}
 	}
 }
 
-// playStage plays the stage
-func (c *NodeController) playStage(ctx context.Context, node *corev1.Node, stage *LifecycleStage) {
+// playStage plays the stage.
+// The returned boolean indicates whether the applying action needs to be retried.
+func (c *NodeController) playStage(ctx context.Context, node *corev1.Node, stage *LifecycleStage) (bool, error) {
 	next := stage.Next()
 	logger := log.FromContext(ctx)
 	logger = logger.With(
@@ -440,22 +452,19 @@ func (c *NodeController) playStage(ctx context.Context, node *corev1.Node, stage
 	if next.Finalizers != nil {
 		result, err = c.finalizersModify(ctx, node, next.Finalizers)
 		if err != nil {
-			logger.Error("Failed to finalizers of node", err)
-			return
+			return shouldRetry(err), fmt.Errorf("failed to patch the finalizer of node %s: %w", node.Name, err)
 		}
 	}
 
 	if next.Delete {
 		err = c.deleteResource(ctx, node)
 		if err != nil {
-			logger.Error("Failed to delete node", err)
-			return
+			return shouldRetry(err), fmt.Errorf("failed to delete node %s: %w", node.Name, err)
 		}
 	} else if next.StatusTemplate != "" {
-		patch, err = c.computePatch(node, next.StatusTemplate)
+		patch, err = c.computeStatusPatch(node, next.StatusTemplate)
 		if err != nil {
-			logger.Error("Failed to configure node", err)
-			return
+			return shouldRetry(err), fmt.Errorf("failed to compute the status patch of node %s: %w", node.Name, err)
 		}
 		if patch == nil {
 			logger.Debug("Skip node",
@@ -464,8 +473,7 @@ func (c *NodeController) playStage(ctx context.Context, node *corev1.Node, stage
 		} else {
 			result, err = c.patchResource(ctx, node, patch)
 			if err != nil {
-				logger.Error("Failed to patch node", err)
-				return
+				return shouldRetry(err), fmt.Errorf("failed to patch node %s: %w", node.Name, err)
 			}
 		}
 	}
@@ -475,6 +483,7 @@ func (c *NodeController) playStage(ctx context.Context, node *corev1.Node, stage
 			"reason", "immediateNextStage is true")
 		c.preprocessChan <- result
 	}
+	return false, nil
 }
 
 func (c *NodeController) readOnly(nodeName string) bool {
@@ -493,19 +502,13 @@ func (c *NodeController) patchResource(ctx context.Context, node *corev1.Node, p
 
 	result, err := c.typedClient.CoreV1().Nodes().Patch(ctx, node.Name, types.StrategicMergePatchType, patch, metav1.PatchOptions{}, "status")
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Patch node",
-				"err", err,
-			)
-			return nil, nil
-		}
 		return nil, err
 	}
 	logger.Info("Patch node")
 	return result, nil
 }
 
-func (c *NodeController) computePatch(node *corev1.Node, tpl string) ([]byte, error) {
+func (c *NodeController) computeStatusPatch(node *corev1.Node, tpl string) ([]byte, error) {
 	patch, err := c.renderer.ToJSON(tpl, node)
 	if err != nil {
 		return nil, err
@@ -618,3 +621,12 @@ func (c *NodeController) funcNodeName() string {
 func (c *NodeController) funcNodePort() int {
 	return c.nodePort
 }
+
+// addStageJob adds a stage to be applied into the underlying weight delay queue and the associated helper map
+func (c *NodeController) addStageJob(job resourceStageJob[*corev1.Node], delay time.Duration, weight int) {
+	old, loaded := c.delayQueueMapping.Swap(job.Key, job)
+	if loaded {
+		c.delayQueue.Cancel(old)
+	}
+	c.delayQueue.AddWeightAfter(job, weight, delay)
+}
diff --git a/pkg/kwok/controllers/pod_controller.go b/pkg/kwok/controllers/pod_controller.go
index f98d0cf5..8d99bd92 100644
--- a/pkg/kwok/controllers/pod_controller.go
+++ b/pkg/kwok/controllers/pod_controller.go
@@ -21,9 +21,10 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"sync/atomic"
+	"time"
 
 	corev1 "k8s.io/api/core/v1"
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/types"
@@ -41,6 +42,7 @@ import (
 	"sigs.k8s.io/kwok/pkg/utils/informer"
 	"sigs.k8s.io/kwok/pkg/utils/maps"
 	"sigs.k8s.io/kwok/pkg/utils/queue"
+	"sigs.k8s.io/kwok/pkg/utils/wait"
 )
 
 var (
@@ -65,7 +67,8 @@ type PodController struct {
 	preprocessChan                        chan *corev1.Pod
 	playStageParallelism                  uint
 	lifecycle                             resources.Getter[Lifecycle]
-	delayQueue                            queue.DelayingQueue[resourceStageJob[*corev1.Pod]]
+	delayQueue                            queue.WeightDelayingQueue[resourceStageJob[*corev1.Pod]]
+	backoff                               wait.Backoff
 	delayQueueMapping                     maps.SyncMap[string, resourceStageJob[*corev1.Pod]]
 	recorder                              record.EventRecorder
 	readOnlyFunc                          func(nodeName string) bool
@@ -126,7 +129,8 @@ func NewPodController(conf PodControllerConfig) (*PodController, error) {
 		nodeIP:                                conf.NodeIP,
 		defaultCIDR:                           conf.CIDR,
 		nodeGetFunc:                           conf.NodeGetFunc,
-		delayQueue:                            queue.NewDelayingQueue[resourceStageJob[*corev1.Pod]](conf.Clock),
+		delayQueue:                            queue.NewWeightDelayingQueue[resourceStageJob[*corev1.Pod]](conf.Clock),
+		backoff:                               defaultBackoff(),
 		lifecycle:                             conf.Lifecycle,
 		playStageParallelism:                  conf.PlayStageParallelism,
 		preprocessChan:                        make(chan *corev1.Pod),
@@ -155,7 +159,7 @@ func (c *PodController) Start(ctx context.Context, events <-chan informer.Event[
 	return nil
 }
 
-// finalizersModify modify the finalizers of the pod
+// finalizersModify modifies the finalizers of a pod
 func (c *PodController) finalizersModify(ctx context.Context, pod *corev1.Pod, finalizers *internalversion.StageFinalizers) (*corev1.Pod, error) {
 	ops := finalizersModify(pod.Finalizers, finalizers)
 	if len(ops) == 0 {
@@ -174,12 +178,6 @@ func (c *PodController) finalizersModify(ctx context.Context, pod *corev1.Pod, f
 
 	result, err := c.typedClient.CoreV1().Pods(pod.Namespace).Patch(ctx, pod.Name, types.JSONPatchType, data, metav1.PatchOptions{})
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Patch pod finalizers",
-				"err", err,
-			)
-			return nil, nil
-		}
 		return nil, err
 	}
 	logger.Info("Patch pod finalizers")
@@ -196,12 +194,6 @@ func (c *PodController) deleteResource(ctx context.Context, pod *corev1.Pod) err
 
 	err := c.typedClient.CoreV1().Pods(pod.Namespace).Delete(ctx, pod.Name, deleteOpt)
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Delete pod",
-				"err", err,
-			)
-			return nil
-		}
 		return err
 	}
 
@@ -285,31 +277,52 @@ func (c *PodController) preprocess(ctx context.Context, pod *corev1.Pod) error {
 	}
 
 	item := resourceStageJob[*corev1.Pod]{
-		Resource: pod,
-		Stage:    stage,
-		Key:      key,
-	}
-
-	c.delayQueue.AddAfter(item, delay)
-	c.delayQueueMapping.Store(key, item)
-
+		Resource:   pod,
+		Stage:      stage,
+		Key:        key,
+		RetryCount: new(uint64),
+	}
+	// we add a normal(fresh) stage job with weight 0,
+	// resulting in that it will always be processed with high priority compared to those retry ones
+	c.addStageJob(item, delay, 0)
 	return nil
 }
 
 // playStageWorker receives the resource from the playStageChan and play the stage
 func (c *PodController) playStageWorker(ctx context.Context) {
+	logger := log.FromContext(ctx)
+
 	for ctx.Err() == nil {
 		pod, ok := c.delayQueue.GetOrWaitWithDone(ctx.Done())
 		if !ok {
 			return
 		}
 		c.delayQueueMapping.Delete(pod.Key)
-		c.playStage(ctx, pod.Resource, pod.Stage)
+		needRetry, err := c.playStage(ctx, pod.Resource, pod.Stage)
+		if err != nil {
+			logger.Error("failed to apply stage", err,
+				"pod", pod.Key,
+				"stage", pod.Stage.Name(),
+			)
+		}
+		if needRetry {
+			retryCount := atomic.AddUint64(pod.RetryCount, 1) - 1
+			logger.Info("retrying for failed job",
+				"pod", pod.Key,
+				"stage", pod.Stage.Name(),
+				"retry", retryCount,
+			)
+			// for failed jobs, we re-push them into the queue with a lower weight
+			// and a backoff period to avoid blocking normal tasks
+			retryDelay := backoffDelayByStep(retryCount, c.backoff)
+			c.addStageJob(pod, retryDelay, 1)
+		}
 	}
 }
 
-// playStage plays the stage
-func (c *PodController) playStage(ctx context.Context, pod *corev1.Pod, stage *LifecycleStage) {
+// playStage plays the stage.
+// The returned boolean indicates whether the applying action needs to be retried.
+func (c *PodController) playStage(ctx context.Context, pod *corev1.Pod, stage *LifecycleStage) (bool, error) {
 	next := stage.Next()
 	logger := log.FromContext(ctx)
 	logger = logger.With(
@@ -336,22 +349,21 @@ func (c *PodController) playStage(ctx context.Context, pod *corev1.Pod, stage *L
 	if next.Finalizers != nil {
 		result, err = c.finalizersModify(ctx, pod, next.Finalizers)
 		if err != nil {
-			logger.Error("Failed to finalizers", err)
-			return
+			return shouldRetry(err), fmt.Errorf("failed to patch the finalizer of pod %s: %w", pod.Name, err)
 		}
 	}
 
 	if next.Delete {
 		err = c.deleteResource(ctx, pod)
 		if err != nil {
-			logger.Error("Failed to delete pod", err)
-			return
+			if err != nil {
+				return shouldRetry(err), fmt.Errorf("failed to delete pod %s: %w", pod.Name, err)
+			}
 		}
 	} else if next.StatusTemplate != "" {
-		patch, err = c.configureResource(pod, next.StatusTemplate)
+		patch, err = c.computeStatusPatch(pod, next.StatusTemplate)
 		if err != nil {
-			logger.Error("Failed to configure pod", err)
-			return
+			return shouldRetry(err), fmt.Errorf("failed to compute the status patch of pod %s: %w", pod.Name, err)
 		}
 		if patch == nil {
 			logger.Debug("Skip pod",
@@ -360,8 +372,7 @@ func (c *PodController) playStage(ctx context.Context, pod *corev1.Pod, stage *L
 		} else {
 			result, err = c.patchResource(ctx, pod, patch)
 			if err != nil {
-				logger.Error("Failed to patch pod", err)
-				return
+				return shouldRetry(err), fmt.Errorf("failed to patch pod %s: %w", pod.Name, err)
 			}
 		}
 	}
@@ -371,6 +382,7 @@ func (c *PodController) playStage(ctx context.Context, pod *corev1.Pod, stage *L
 			"reason", "immediateNextStage is true")
 		c.preprocessChan <- result
 	}
+	return false, nil
 }
 
 func (c *PodController) readOnly(nodeName string) bool {
@@ -390,12 +402,6 @@ func (c *PodController) patchResource(ctx context.Context, pod *corev1.Pod, patc
 
 	result, err := c.typedClient.CoreV1().Pods(pod.Namespace).Patch(ctx, pod.Name, types.StrategicMergePatchType, patch, metav1.PatchOptions{}, "status")
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Patch pod",
-				"err", err,
-			)
-			return nil, nil
-		}
 		return nil, err
 	}
 	logger.Info("Patch pod")
@@ -547,7 +553,7 @@ func (c *PodController) recyclingPodIP(ctx context.Context, pod *corev1.Pod) {
 	}
 }
 
-func (c *PodController) configureResource(pod *corev1.Pod, template string) ([]byte, error) {
+func (c *PodController) computeStatusPatch(pod *corev1.Pod, template string) ([]byte, error) {
 	if !c.enableCNI {
 		// Mark the pod IP that existed before the kubelet was started
 		if _, has := c.nodeGetFunc(pod.Spec.NodeName); has {
@@ -713,3 +719,12 @@ func (c *PodController) List(nodeName string) ([]log.ObjectRef, bool) {
 	}
 	return m.Keys(), true
 }
+
+// addStageJob adds a stage to be applied into the underlying weight delay queue and the associated helper map
+func (c *PodController) addStageJob(job resourceStageJob[*corev1.Pod], delay time.Duration, weight int) {
+	old, loaded := c.delayQueueMapping.Swap(job.Key, job)
+	if loaded {
+		c.delayQueue.Cancel(old)
+	}
+	c.delayQueue.AddWeightAfter(job, weight, delay)
+}
diff --git a/pkg/kwok/controllers/stage_controller.go b/pkg/kwok/controllers/stage_controller.go
index 78c96c14..43d8fd91 100644
--- a/pkg/kwok/controllers/stage_controller.go
+++ b/pkg/kwok/controllers/stage_controller.go
@@ -21,9 +21,10 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"sync/atomic"
+	"time"
 
 	corev1 "k8s.io/api/core/v1"
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/labels"
@@ -44,6 +45,7 @@ import (
 	"sigs.k8s.io/kwok/pkg/utils/informer"
 	"sigs.k8s.io/kwok/pkg/utils/maps"
 	"sigs.k8s.io/kwok/pkg/utils/queue"
+	"sigs.k8s.io/kwok/pkg/utils/wait"
 )
 
 // StageController is a fake resources implementation that can be used to test
@@ -59,7 +61,8 @@ type StageController struct {
 	preprocessChan                        chan *unstructured.Unstructured
 	playStageParallelism                  uint
 	lifecycle                             resources.Getter[Lifecycle]
-	delayQueue                            queue.DelayingQueue[resourceStageJob[*unstructured.Unstructured]]
+	delayQueue                            queue.WeightDelayingQueue[resourceStageJob[*unstructured.Unstructured]]
+	backoff                               wait.Backoff
 	delayQueueMapping                     maps.SyncMap[string, resourceStageJob[*unstructured.Unstructured]]
 	recorder                              record.EventRecorder
 }
@@ -107,7 +110,8 @@ func NewStageController(conf StageControllerConfig) (*StageController, error) {
 		gvr:                                   conf.GVR,
 		disregardStatusWithAnnotationSelector: disregardStatusWithAnnotationSelector,
 		disregardStatusWithLabelSelector:      disregardStatusWithLabelSelector,
-		delayQueue:                            queue.NewDelayingQueue[resourceStageJob[*unstructured.Unstructured]](conf.Clock),
+		delayQueue:                            queue.NewWeightDelayingQueue[resourceStageJob[*unstructured.Unstructured]](conf.Clock),
+		backoff:                               defaultBackoff(),
 		lifecycle:                             conf.Lifecycle,
 		playStageParallelism:                  conf.PlayStageParallelism,
 		preprocessChan:                        make(chan *unstructured.Unstructured),
@@ -129,7 +133,7 @@ func (c *StageController) Start(ctx context.Context, events <-chan informer.Even
 	return nil
 }
 
-// finalizersModify modify the finalizers of the resource
+// finalizersModify modifies the finalizers of a resource
 func (c *StageController) finalizersModify(ctx context.Context, resource *unstructured.Unstructured, finalizers *internalversion.StageFinalizers) (*unstructured.Unstructured, error) {
 	ops := finalizersModify(resource.GetFinalizers(), finalizers)
 	if len(ops) == 0 {
@@ -152,12 +156,6 @@ func (c *StageController) finalizersModify(ctx context.Context, resource *unstru
 	}
 	result, err := cli.Patch(ctx, resource.GetName(), types.JSONPatchType, data, metav1.PatchOptions{})
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Patch resource finalizers",
-				"err", err,
-			)
-			return nil, nil
-		}
 		return nil, err
 	}
 	logger.Info("Patch resource finalizers")
@@ -178,12 +176,6 @@ func (c *StageController) deleteResource(ctx context.Context, resource *unstruct
 	}
 	err := cli.Delete(ctx, resource.GetName(), deleteOpt)
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Delete resource",
-				"err", err,
-			)
-			return nil
-		}
 		return err
 	}
 
@@ -202,7 +194,7 @@ func (c *StageController) preprocessWorker(ctx context.Context) {
 		case resource := <-c.preprocessChan:
 			err := c.preprocess(ctx, resource)
 			if err != nil {
-				logger.Error("Failed to preprocess node", err,
+				logger.Error("Failed to preprocess resource", err,
 					"resource", log.KObj(resource),
 				)
 			}
@@ -265,31 +257,53 @@ func (c *StageController) preprocess(ctx context.Context, resource *unstructured
 	}
 
 	item := resourceStageJob[*unstructured.Unstructured]{
-		Resource: resource,
-		Stage:    stage,
-		Key:      key,
+		Resource:   resource,
+		Stage:      stage,
+		Key:        key,
+		RetryCount: new(uint64),
 	}
 
-	c.delayQueue.AddAfter(item, delay)
-	c.delayQueueMapping.Store(key, item)
-
+	// we add a normal(fresh) stage job with weight 0,
+	// resulting in that it will always be processed with high priority compared to those retry ones
+	c.addStageJob(item, delay, 0)
 	return nil
 }
 
 // playStageWorker receives the resource from the playStageChan and play the stage
 func (c *StageController) playStageWorker(ctx context.Context) {
+	logger := log.FromContext(ctx)
+
 	for ctx.Err() == nil {
 		resource, ok := c.delayQueue.GetOrWaitWithDone(ctx.Done())
 		if !ok {
 			return
 		}
 		c.delayQueueMapping.Delete(resource.Key)
-		c.playStage(ctx, resource.Resource, resource.Stage)
+		needRetry, err := c.playStage(ctx, resource.Resource, resource.Stage)
+		if err != nil {
+			logger.Error("failed to apply stage", err,
+				"resource", resource.Key,
+				"stage", resource.Stage.Name(),
+			)
+		}
+		if needRetry {
+			retryCount := atomic.AddUint64(resource.RetryCount, 1) - 1
+			logger.Info("retrying for failed job",
+				"resource", resource.Key,
+				"stage", resource.Stage.Name(),
+				"retry", retryCount,
+			)
+			// for failed jobs, we re-push them into the queue with a lower weight
+			// and a backoff period to avoid blocking normal tasks
+			retryDelay := backoffDelayByStep(retryCount, c.backoff)
+			c.addStageJob(resource, retryDelay, 1)
+		}
 	}
 }
 
-// playStage plays the stage
-func (c *StageController) playStage(ctx context.Context, resource *unstructured.Unstructured, stage *LifecycleStage) {
+// playStage plays the stage.
+// The returned boolean indicates whether the applying action needs to be retried.
+func (c *StageController) playStage(ctx context.Context, resource *unstructured.Unstructured, stage *LifecycleStage) (bool, error) {
 	next := stage.Next()
 	logger := log.FromContext(ctx)
 	logger = logger.With(
@@ -315,32 +329,28 @@ func (c *StageController) playStage(ctx context.Context, resource *unstructured.
 	if next.Finalizers != nil {
 		result, err = c.finalizersModify(ctx, resource, next.Finalizers)
 		if err != nil {
-			logger.Error("Failed to finalizers", err)
-			return
+			return shouldRetry(err), fmt.Errorf("failed to patch the finalizer of resource %s: %w", resource.GetName(), err)
 		}
 	}
 
 	if next.Delete {
 		err = c.deleteResource(ctx, resource)
 		if err != nil {
-			logger.Error("Failed to delete resource", err)
-			return
+			return shouldRetry(err), fmt.Errorf("failed to delete resource %s: %w", resource.GetName(), err)
 		}
 	} else if next.StatusTemplate != "" {
-		patch, err = c.configureResource(resource, next.StatusTemplate)
+		patch, err = c.computeStatusPatch(resource, next.StatusTemplate)
 		if err != nil {
-			logger.Error("Failed to configure resource", err)
-			return
+			return shouldRetry(err), fmt.Errorf("failed to compute the status patch of resource %s: %w", resource.GetName(), err)
 		}
 		if patch == nil {
 			logger.Debug("Skip resource",
 				"reason", "do not need to modify",
 			)
 		} else {
-			result, err = c.patchResource(ctx, resource, patch, stage.next)
+			result, err = c.patchResource(ctx, resource, patch, next)
 			if err != nil {
-				logger.Error("Failed to patch resource", err)
-				return
+				return shouldRetry(err), fmt.Errorf("failed to patch resource %s: %w", resource.GetName(), err)
 			}
 		}
 	}
@@ -350,6 +360,7 @@ func (c *StageController) playStage(ctx context.Context, resource *unstructured.
 			"reason", "immediateNextStage is true")
 		c.preprocessChan <- result
 	}
+	return false, nil
 }
 
 // patchResource patches the resource
@@ -385,12 +396,6 @@ func (c *StageController) patchResource(ctx context.Context, resource *unstructu
 	}
 	result, err := cli.Patch(ctx, resource.GetName(), types.MergePatchType, patch, metav1.PatchOptions{}, subresource...)
 	if err != nil {
-		if apierrors.IsNotFound(err) {
-			logger.Warn("Patch resource",
-				"err", err,
-			)
-			return nil, nil
-		}
 		return nil, err
 	}
 	logger.Info("Patch resource")
@@ -454,7 +459,7 @@ loop:
 	logger.Info("Stop watch resources")
 }
 
-func (c *StageController) configureResource(resource *unstructured.Unstructured, template string) ([]byte, error) {
+func (c *StageController) computeStatusPatch(resource *unstructured.Unstructured, template string) ([]byte, error) {
 	patch, err := c.computePatch(resource, template)
 	if err != nil {
 		return nil, err
@@ -500,3 +505,12 @@ func (c *StageController) computePatch(resource *unstructured.Unstructured, tpl
 	}
 	return patchData, nil
 }
+
+// addStageJob adds a stage to be applied into the underlying weight delay queue and the associated helper map
+func (c *StageController) addStageJob(job resourceStageJob[*unstructured.Unstructured], delay time.Duration, weight int) {
+	old, loaded := c.delayQueueMapping.Swap(job.Key, job)
+	if loaded {
+		c.delayQueue.Cancel(old)
+	}
+	c.delayQueue.AddWeightAfter(job, weight, delay)
+}
diff --git a/pkg/kwok/controllers/utils.go b/pkg/kwok/controllers/utils.go
index ce3c797f..a198710e 100644
--- a/pkg/kwok/controllers/utils.go
+++ b/pkg/kwok/controllers/utils.go
@@ -17,12 +17,17 @@ limitations under the License.
 package controllers
 
 import (
+	"math"
 	"net"
 	"sync"
+	"time"
 
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	"k8s.io/apimachinery/pkg/labels"
+	utilnet "k8s.io/apimachinery/pkg/util/net"
 
 	utilsnet "sigs.k8s.io/kwok/pkg/utils/net"
+	"sigs.k8s.io/kwok/pkg/utils/wait"
 )
 
 func parseCIDR(s string) (*net.IPNet, error) {
@@ -112,4 +117,37 @@ type resourceStageJob[T any] struct {
 	Resource T
 	Stage    *LifecycleStage
 	Key      string
+	// RetryCount is used for tracking the retry times of a job.
+	// Must be initialized to 0.
+	RetryCount *uint64
+}
+
+// defaultBackoff provides a backoff setting for kwok controllers to apply failed jobs
+func defaultBackoff() wait.Backoff {
+	return wait.Backoff{Duration: 1 * time.Second, Factor: 2.0, Jitter: 0.2, Cap: 32 * time.Minute}
+}
+
+// backoffDelayByStep calculates the backoff delay period based on steps
+func backoffDelayByStep(steps uint64, c wait.Backoff) time.Duration {
+	delay := math.Min(
+		float64(c.Duration)*math.Pow(c.Factor, float64(steps)),
+		float64(c.Cap))
+	return wait.Jitter(time.Duration(delay), c.Jitter)
+}
+
+// shouldRetry determines if a certain error needs to be retried
+func shouldRetry(err error) bool {
+	// if apiserver is not reachable
+	if utilnet.IsConnectionRefused(err) {
+		return true
+	}
+	// if it is a network issue reported by apiserver side
+	if apierrors.IsServerTimeout(err) ||
+		apierrors.IsTimeout(err) ||
+		apierrors.IsServiceUnavailable(err) ||
+		apierrors.IsTooManyRequests(err) {
+		return true
+	}
+	// ignore all other cases
+	return false
 }
diff --git a/pkg/kwok/metrics/cel/evaluate.go b/pkg/kwok/metrics/cel/evaluate.go
index 63178386..5d026b3b 100644
--- a/pkg/kwok/metrics/cel/evaluate.go
+++ b/pkg/kwok/metrics/cel/evaluate.go
@@ -28,6 +28,7 @@ import (
 	"github.com/google/cel-go/common/types/ref"
 	"github.com/wzshiming/easycel"
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
@@ -38,6 +39,14 @@ type NodeEvaluatorConfig struct {
 
 	Now                    func() time.Time
 	StartedContainersTotal func(nodeName string) int64
+
+	ContainerResourceUsage func(resourceName, podNamespace, podName, containerName string) float64
+	PodResourceUsage       func(resourceName, podNamespace, podName string) float64
+	NodeResourceUsage      func(resourceName, nodeName string) float64
+
+	ContainerResourceCumulativeUsage func(resourceName, podNamespace, podName, containerName string) float64
+	PodResourceCumulativeUsage       func(resourceName, podNamespace, podName string) float64
+	NodeResourceCumulativeUsage      func(resourceName, nodeName string) float64
 }
 
 // NewEnvironment returns a MetricEvaluator that is able to evaluate node metrics
@@ -103,16 +112,24 @@ func (e *Environment) init() error {
 			}
 			return types.Duration{Duration: t.Duration}
 		},
+		func(t resource.Quantity) Quantity {
+			return NewQuantity(&t)
+		},
+		NewResourceList,
 	}
+
 	types := []any{
 		corev1.Node{},
 		corev1.NodeSpec{},
 		corev1.NodeStatus{},
 		corev1.Pod{},
 		corev1.PodSpec{},
+		corev1.ResourceRequirements{},
 		corev1.PodStatus{},
 		corev1.Container{},
 		metav1.ObjectMeta{},
+		Quantity{},
+		ResourceList{},
 	}
 
 	vars := map[string]any{
@@ -134,6 +151,11 @@ func (e *Environment) init() error {
 		mathRandName               = "Rand"
 		sinceSecondName            = "SinceSecond"
 		unixSecondName             = "UnixSecond"
+
+		quantityName = "Quantity"
+
+		usageName           = "Usage"
+		cumulativeUsageName = "CumulativeUsage"
 	)
 	if e.conf.Now != nil {
 		funcs[nowOldName] = append(funcs[nowOldName], e.conf.Now)
@@ -151,6 +173,44 @@ func (e *Environment) init() error {
 	methods[unixSecondName] = append(methods[unixSecondName], unixSecond)
 	funcs[unixSecondName] = append(funcs[unixSecondName], unixSecond)
 
+	funcs[quantityName] = append(funcs[quantityName], NewQuantityFromString)
+
+	if e.conf.ContainerResourceUsage != nil {
+		methods[usageName] = append(methods[usageName], func(pod corev1.Pod, resourceName string, containerName string) float64 {
+			return e.conf.ContainerResourceUsage(resourceName, pod.Namespace, pod.Name, containerName)
+		})
+	}
+
+	if e.conf.PodResourceUsage != nil {
+		methods[usageName] = append(methods[usageName], func(pod corev1.Pod, resourceName string) float64 {
+			return e.conf.PodResourceUsage(resourceName, pod.Namespace, pod.Name)
+		})
+	}
+
+	if e.conf.NodeResourceUsage != nil {
+		methods[usageName] = append(methods[usageName], func(node corev1.Node, resourceName string) float64 {
+			return e.conf.NodeResourceUsage(resourceName, node.Name)
+		})
+	}
+
+	if e.conf.ContainerResourceCumulativeUsage != nil {
+		methods[cumulativeUsageName] = append(methods[cumulativeUsageName], func(pod corev1.Pod, resourceName string, containerName string) float64 {
+			return e.conf.ContainerResourceCumulativeUsage(resourceName, pod.Namespace, pod.Name, containerName)
+		})
+	}
+
+	if e.conf.PodResourceCumulativeUsage != nil {
+		methods[cumulativeUsageName] = append(methods[cumulativeUsageName], func(pod corev1.Pod, resourceName string) float64 {
+			return e.conf.PodResourceCumulativeUsage(resourceName, pod.Namespace, pod.Name)
+		})
+	}
+
+	if e.conf.NodeResourceCumulativeUsage != nil {
+		methods[cumulativeUsageName] = append(methods[cumulativeUsageName], func(node corev1.Node, resourceName string) float64 {
+			return e.conf.NodeResourceCumulativeUsage(resourceName, node.Name)
+		})
+	}
+
 	if e.conf.StartedContainersTotal != nil {
 		startedContainersTotal := e.conf.StartedContainersTotal
 		startedContainersTotalByNode := func(node corev1.Node) float64 {
@@ -309,6 +369,8 @@ func (e *Evaluator) EvaluateFloat64(data Data) (float64, error) {
 			return 1, nil
 		}
 		return 0, nil
+	case Quantity:
+		return v.Quantity.AsApproximateFloat64(), nil
 	default:
 		return 0, fmt.Errorf("unsupported metric value type: %T", v)
 	}
diff --git a/pkg/kwok/metrics/cel/quantity.go b/pkg/kwok/metrics/cel/quantity.go
new file mode 100644
index 00000000..9b7f82ec
--- /dev/null
+++ b/pkg/kwok/metrics/cel/quantity.go
@@ -0,0 +1,201 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package cel
+
+import (
+	"fmt"
+	"reflect"
+
+	"github.com/google/cel-go/cel"
+	"github.com/google/cel-go/common/types"
+	"github.com/google/cel-go/common/types/ref"
+	"github.com/google/cel-go/common/types/traits"
+	"k8s.io/apimachinery/pkg/api/resource"
+)
+
+var (
+	// QuantityType singleton.
+	QuantityType = cel.ObjectType("kubernetes.Quantity",
+		traits.AdderType,
+		traits.ComparerType,
+		traits.DividerType,
+		traits.MultiplierType,
+		traits.NegatorType,
+		traits.SubtractorType,
+	)
+)
+
+// Quantity is a wrapper around k8s.io/apimachinery/pkg/api/resource.Quantity
+type Quantity struct {
+	Quantity *resource.Quantity
+}
+
+// NewQuantity creates a new Quantity
+func NewQuantity(q *resource.Quantity) Quantity {
+	if q == nil {
+		q = resource.NewScaledQuantity(0, resource.Nano)
+	}
+	return Quantity{Quantity: q}
+}
+
+// NewQuantityFromString creates a new Quantity from a string
+func NewQuantityFromString(s string) (Quantity, error) {
+	r, err := resource.ParseQuantity(s)
+	if err != nil {
+		return Quantity{}, err
+	}
+	return NewQuantity(&r), nil
+}
+
+func newQuantityFromNanoInt64(v int64) Quantity {
+	r := resource.NewScaledQuantity(v, resource.Nano)
+	return NewQuantity(r)
+}
+
+func newQuantityFromFloat64(v float64) Quantity {
+	r := resource.NewScaledQuantity(int64(v*10e9), resource.Nano)
+	return NewQuantity(r)
+}
+
+func (q Quantity) nano() int64 {
+	return q.Quantity.ScaledValue(resource.Nano)
+}
+
+func (q Quantity) float() float64 {
+	return q.Quantity.AsApproximateFloat64()
+}
+
+// ConvertToNative implements the ref.Val interface.
+func (q Quantity) ConvertToNative(typeDesc reflect.Type) (any, error) {
+	switch typeDesc.Kind() {
+	case reflect.Float32, reflect.Float64:
+		v := q.Quantity.AsApproximateFloat64()
+		return reflect.ValueOf(v).Convert(typeDesc).Interface(), nil
+	}
+	return nil, fmt.Errorf("type conversion error from Quantity to '%v'", typeDesc)
+}
+
+// ConvertToType implements the ref.Val interface.
+func (q Quantity) ConvertToType(typeVal ref.Type) ref.Val {
+	switch typeVal {
+	case types.DoubleType:
+		v := q.Quantity.AsApproximateFloat64()
+		return types.Double(v)
+	case types.StringType:
+		v := q.Quantity.String()
+		return types.String(v)
+	case types.TypeType:
+		return QuantityType
+	}
+	return types.NewErr("type conversion error from '%s' to '%s'", QuantityType, typeVal)
+}
+
+// Equal implements the ref.Val interface.
+func (q Quantity) Equal(other ref.Val) ref.Val {
+	otherQuantity, ok := other.(Quantity)
+	if !ok {
+		return types.MaybeNoSuchOverloadErr(other)
+	}
+	return types.Bool(q.Quantity.Equal(*otherQuantity.Quantity))
+}
+
+// Type implements the ref.Val interface.
+func (q Quantity) Type() ref.Type {
+	return QuantityType
+}
+
+// Value implements the ref.Val interface.
+func (q Quantity) Value() any {
+	return q.Quantity
+}
+
+// Add implements the traits.Adder interface.
+func (q Quantity) Add(other ref.Val) ref.Val {
+	otherQuantity, ok := other.(Quantity)
+	if !ok {
+		return types.MaybeNoSuchOverloadErr(other)
+	}
+	r := q.Quantity.DeepCopy()
+	r.Add(*otherQuantity.Quantity)
+	return NewQuantity(&r)
+}
+
+// Subtract implements the traits.Subtractor interface.
+func (q Quantity) Subtract(subtrahend ref.Val) ref.Val {
+	otherQuantity, ok := subtrahend.(Quantity)
+	if !ok {
+		return types.MaybeNoSuchOverloadErr(subtrahend)
+	}
+	r := q.Quantity.DeepCopy()
+	r.Sub(*otherQuantity.Quantity)
+	return NewQuantity(&r)
+}
+
+// Negate implements the traits.Negater interface.
+func (q Quantity) Negate() ref.Val {
+	r := q.Quantity.DeepCopy()
+	r.Neg()
+	return NewQuantity(&r)
+}
+
+// Divide implements the traits.Divider interface.
+func (q Quantity) Divide(other ref.Val) ref.Val {
+	switch other.Type() {
+	case types.IntType:
+		otherInt := other.(types.Int)
+		return newQuantityFromNanoInt64(q.nano() / int64(otherInt))
+	case types.UintType:
+		otherUint := other.(types.Uint)
+		return newQuantityFromNanoInt64(q.nano() / int64(otherUint))
+	case types.DoubleType:
+		otherDouble := other.(types.Double)
+		return newQuantityFromFloat64(q.float() / float64(otherDouble))
+	}
+
+	return types.MaybeNoSuchOverloadErr(other)
+}
+
+// Multiply implements the traits.Multiplier interface.
+func (q Quantity) Multiply(other ref.Val) ref.Val {
+	switch other.Type() {
+	case types.IntType:
+		otherInt := other.(types.Int)
+		return newQuantityFromNanoInt64(q.nano() * int64(otherInt))
+	case types.UintType:
+		otherUint := other.(types.Uint)
+		return newQuantityFromNanoInt64(q.nano() * int64(otherUint))
+	case types.DoubleType:
+		otherDouble := other.(types.Double)
+		return newQuantityFromFloat64(q.float() * float64(otherDouble))
+	}
+
+	return types.MaybeNoSuchOverloadErr(other)
+}
+
+// Compare implements the traits.Comparer interface.
+func (q Quantity) Compare(other ref.Val) ref.Val {
+	otherQuantity, ok := other.(Quantity)
+	if !ok {
+		return types.MaybeNoSuchOverloadErr(other)
+	}
+	return types.Int(q.Quantity.Cmp(*otherQuantity.Quantity))
+}
+
+// IsZeroValue implements the traits.Zeroer interface.
+func (q Quantity) IsZeroValue() bool {
+	return q.Quantity.IsZero()
+}
diff --git a/pkg/kwok/metrics/cel/resource_list.go b/pkg/kwok/metrics/cel/resource_list.go
new file mode 100644
index 00000000..358bff67
--- /dev/null
+++ b/pkg/kwok/metrics/cel/resource_list.go
@@ -0,0 +1,122 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package cel
+
+import (
+	"fmt"
+	"reflect"
+
+	"github.com/google/cel-go/cel"
+	"github.com/google/cel-go/common/types"
+	"github.com/google/cel-go/common/types/ref"
+	"github.com/google/cel-go/common/types/traits"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	// ResourceListType singleton.
+	ResourceListType = cel.ObjectType("kubernetes.ResourceList",
+		traits.ContainerType,
+		traits.IndexerType,
+		traits.SizerType,
+	)
+)
+
+// ResourceList is a wrapper around k8s.io/api/core/v1.ResourceList
+type ResourceList struct {
+	List corev1.ResourceList
+}
+
+// NewResourceList creates a new ResourceList
+func NewResourceList(list corev1.ResourceList) ResourceList {
+	return ResourceList{List: list}
+}
+
+// ConvertToNative implements the ref.Val interface.
+func (r ResourceList) ConvertToNative(typeDesc reflect.Type) (any, error) {
+	return nil, fmt.Errorf("unsupported conversion from '%s' to '%v'", ResourceListType, typeDesc)
+}
+
+// ConvertToType implements the ref.Val interface.
+func (r ResourceList) ConvertToType(typeValue ref.Type) ref.Val {
+	return types.NewErr("type conversion error from '%s' to '%s'", ResourceListType, typeValue)
+}
+
+// Equal implements the ref.Val interface.
+func (r ResourceList) Equal(other ref.Val) ref.Val {
+	otherResourceList, ok := other.(ResourceList)
+	if !ok {
+		return types.MaybeNoSuchOverloadErr(other)
+	}
+	if len(r.List) != len(otherResourceList.List) {
+		return types.False
+	}
+	for k, v := range r.List {
+		otherVal, ok := otherResourceList.List[k]
+		if !ok {
+			return types.False
+		}
+		if !v.Equal(otherVal) {
+			return types.False
+		}
+	}
+	return types.True
+}
+
+// Type implements the ref.Val interface.
+func (r ResourceList) Type() ref.Type {
+	return ResourceListType
+}
+
+// Value implements the ref.Val interface.
+func (r ResourceList) Value() any {
+	return r.List
+}
+
+// Contains implements the traits.Container interface.
+func (r ResourceList) Contains(index ref.Val) ref.Val {
+	key, ok := index.(types.String)
+	if !ok {
+		return types.MaybeNoSuchOverloadErr(index)
+	}
+	_, found := r.List[corev1.ResourceName(key)]
+	return types.Bool(found)
+}
+
+// Get implements the traits.Indexer interface.
+func (r ResourceList) Get(index ref.Val) ref.Val {
+	key, ok := index.(types.String)
+	if !ok {
+		return types.MaybeNoSuchOverloadErr(index)
+	}
+
+	val, found := r.List[corev1.ResourceName(key)]
+	if !found {
+		return NewQuantity(nil)
+	}
+	return NewQuantity(&val)
+}
+
+// IsZeroValue implements the traits.Zeroer interface.
+func (r ResourceList) IsZeroValue() bool {
+	return len(r.List) == 0
+}
+
+// Size implements the traits.Sizer interface.
+func (r ResourceList) Size() ref.Val {
+	return types.Int(len(r.List))
+}
diff --git a/pkg/kwok/server/metrics.go b/pkg/kwok/server/metrics.go
index 069c83c1..1c3d7db9 100644
--- a/pkg/kwok/server/metrics.go
+++ b/pkg/kwok/server/metrics.go
@@ -32,22 +32,43 @@ import (
 	"sigs.k8s.io/kwok/pkg/log"
 )
 
-// InstallMetrics registers the metrics handler on the given mux.
-func (s *Server) InstallMetrics(ctx context.Context) error {
-	promHandler := promhttp.Handler()
-
-	selfMetric := func(req *restful.Request, resp *restful.Response) {
-		promHandler.ServeHTTP(resp.ResponseWriter, req.Request)
+func (s *Server) initCEL() error {
+	if s.env != nil {
+		return fmt.Errorf("CEL environment already initialized")
 	}
 
 	env, err := cel.NewEnvironment(cel.NodeEvaluatorConfig{
 		EnableEvaluatorCache:   true,
 		EnableResultCache:      true,
 		StartedContainersTotal: s.dataSource.StartedContainersTotal,
+
+		ContainerResourceUsage: s.containerResourceUsage,
+		PodResourceUsage:       s.podResourceUsage,
+		NodeResourceUsage:      s.nodeResourceUsage,
+
+		ContainerResourceCumulativeUsage: s.containerResourceCumulativeUsage,
+		PodResourceCumulativeUsage:       s.podResourceCumulativeUsage,
+		NodeResourceCumulativeUsage:      s.nodeResourceCumulativeUsage,
 	})
 	if err != nil {
 		return fmt.Errorf("failed to create CEL environment: %w", err)
 	}
+	s.env = env
+	return nil
+}
+
+// InstallMetrics registers the metrics handler on the given mux.
+func (s *Server) InstallMetrics(ctx context.Context) error {
+	err := s.initCEL()
+	if err != nil {
+		return err
+	}
+
+	promHandler := promhttp.Handler()
+
+	selfMetric := func(req *restful.Request, resp *restful.Response) {
+		promHandler.ServeHTTP(resp.ResponseWriter, req.Request)
+	}
 
 	const rootPath = "/metrics"
 	ws := new(restful.WebService)
@@ -55,56 +76,58 @@ func (s *Server) InstallMetrics(ctx context.Context) error {
 	ws.Route(ws.GET("/").To(selfMetric))
 	s.restfulCont.Add(ws)
 
-	hasPaths := map[string]struct{}{}
-	logger := log.FromContext(ctx)
 	syncd, ok := s.metrics.(resources.Synced)
 	if ok {
-		go func() {
-			for range syncd.Sync() {
-				newHasPaths := map[string]struct{}{}
-				for _, m := range s.metrics.Get() {
-					if !strings.HasPrefix(m.Spec.Path, rootPath) {
-						logger.Warn("metric path does not start with "+rootPath, "path", m.Spec.Path)
-						continue
-					}
-
-					path := strings.TrimPrefix(m.Spec.Path, rootPath)
-					newHasPaths[path] = struct{}{}
-					if _, ok := hasPaths[path]; ok {
-						err := ws.RemoveRoute(http.MethodGet, path)
-						if err != nil {
-							logger.Error("Failed to remove route", err, "path", path)
-						}
-					}
-					ws.Route(ws.GET(path).
-						To(s.getMetrics(m, env)))
-				}
-
-				for path := range hasPaths {
-					if _, ok := newHasPaths[path]; !ok {
-						err := ws.RemoveRoute(http.MethodGet, path)
-						if err != nil {
-							logger.Error("Failed to remove route", err, "path", path)
-						}
-					}
-				}
-
-				hasPaths = newHasPaths
-			}
-		}()
+		go s.dynamicMetricsPath(ctx, ws, syncd, rootPath)
 	} else {
 		for _, m := range s.metrics.Get() {
 			if !strings.HasPrefix(m.Spec.Path, rootPath) {
 				return fmt.Errorf("metric path %q does not start with %q", m.Spec.Path, rootPath)
 			}
 			ws.Route(ws.GET(strings.TrimPrefix(m.Spec.Path, rootPath)).
-				To(s.getMetrics(m, env)))
+				To(s.getMetrics(m, s.env)))
 		}
 	}
 
 	return nil
 }
 
+func (s *Server) dynamicMetricsPath(ctx context.Context, ws *restful.WebService, syncd resources.Synced, rootPath string) {
+	logger := log.FromContext(ctx)
+	hasPaths := map[string]struct{}{}
+	for range syncd.Sync() {
+		newHasPaths := map[string]struct{}{}
+		for _, m := range s.metrics.Get() {
+			if !strings.HasPrefix(m.Spec.Path, rootPath) {
+				logger.Warn("metric path does not start with "+rootPath, "path", m.Spec.Path)
+				continue
+			}
+
+			path := strings.TrimPrefix(m.Spec.Path, rootPath)
+			newHasPaths[path] = struct{}{}
+			if _, ok := hasPaths[path]; ok {
+				err := ws.RemoveRoute(http.MethodGet, path)
+				if err != nil {
+					logger.Error("Failed to remove route", err, "path", path)
+				}
+			}
+			ws.Route(ws.GET(path).
+				To(s.getMetrics(m, s.env)))
+		}
+
+		for path := range hasPaths {
+			if _, ok := newHasPaths[path]; !ok {
+				err := ws.RemoveRoute(http.MethodGet, path)
+				if err != nil {
+					logger.Error("Failed to remove route", err, "path", path)
+				}
+			}
+		}
+
+		hasPaths = newHasPaths
+	}
+}
+
 func (s *Server) getMetrics(metric *internalversion.Metric, env *cel.Environment) func(req *restful.Request, resp *restful.Response) {
 	return func(req *restful.Request, resp *restful.Response) {
 		nodeName := req.PathParameter("nodeName")
diff --git a/pkg/kwok/server/metrics_resource_usage.go b/pkg/kwok/server/metrics_resource_usage.go
new file mode 100644
index 00000000..7fac3242
--- /dev/null
+++ b/pkg/kwok/server/metrics_resource_usage.go
@@ -0,0 +1,264 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package server
+
+import (
+	"fmt"
+	"time"
+
+	corev1 "k8s.io/api/core/v1"
+
+	"sigs.k8s.io/kwok/pkg/apis/internalversion"
+	"sigs.k8s.io/kwok/pkg/kwok/metrics/cel"
+	"sigs.k8s.io/kwok/pkg/log"
+	"sigs.k8s.io/kwok/pkg/utils/slices"
+)
+
+type cumulative struct {
+	time  time.Time
+	value float64
+}
+
+func (s *Server) containerResourceCumulativeUsage(resourceName, podNamespace, podName, containerName string) float64 {
+	key := fmt.Sprintf("%s/%s/%s/%s", resourceName, podNamespace, podName, containerName)
+	v := s.containerResourceUsage(resourceName, podNamespace, podName, containerName)
+
+	now := time.Now()
+	s.cumulativesMut.Lock()
+	defer s.cumulativesMut.Unlock()
+	c, ok := s.cumulatives[key]
+	if ok {
+		c.value += now.Sub(c.time).Seconds() * v
+	}
+
+	c.time = now
+	s.cumulatives[key] = c
+
+	return c.value
+}
+
+func (s *Server) podResourceCumulativeUsage(resourceName, podNamespace, podName string) float64 {
+	pod, ok := s.podCacheGetter.GetWithNamespace(podName, podNamespace)
+	if !ok {
+		return 0
+	}
+
+	sum := 0.0
+	for _, c := range pod.Spec.Containers {
+		sum += s.containerResourceCumulativeUsage(resourceName, podNamespace, podName, c.Name)
+	}
+	return sum
+}
+
+func (s *Server) nodeResourceCumulativeUsage(resourceName, nodeName string) float64 {
+	node, ok := s.nodeCacheGetter.Get(nodeName)
+	if !ok {
+		return 0
+	}
+
+	pods, ok := s.dataSource.ListPods(nodeName)
+	if !ok {
+		return 0
+	}
+
+	data := cel.Data{
+		Node: node,
+	}
+
+	sum := 0.0
+	for _, pi := range pods {
+		pod, ok := s.podCacheGetter.GetWithNamespace(pi.Name, pi.Namespace)
+		if !ok {
+			continue
+		}
+		data.Pod = pod
+		for _, c := range pod.Spec.Containers {
+			c := c
+			data.Container = &c
+			sum += s.evaluateContainerResourceUsage(resourceName, data)
+		}
+	}
+
+	now := time.Now()
+	key := nodeName
+	s.cumulativesMut.Lock()
+	defer s.cumulativesMut.Unlock()
+	c, ok := s.cumulatives[key]
+	if ok {
+		c.value += now.Sub(c.time).Seconds() * sum
+	}
+
+	c.time = now
+	s.cumulatives[key] = c
+
+	return c.value
+}
+
+func (s *Server) containerResourceUsage(resourceName, podNamespace, podName, containerName string) float64 {
+	pod, ok := s.podCacheGetter.GetWithNamespace(podName, podNamespace)
+	if !ok {
+		return 0
+	}
+
+	node, ok := s.nodeCacheGetter.Get(pod.Spec.NodeName)
+	if !ok {
+		return 0
+	}
+
+	c, ok := slices.Find(pod.Spec.Containers, func(c corev1.Container) bool {
+		return c.Name == containerName
+	})
+	if !ok {
+		return 0
+	}
+	data := cel.Data{
+		Node:      node,
+		Pod:       pod,
+		Container: &c,
+	}
+	return s.evaluateContainerResourceUsage(resourceName, data)
+}
+
+func (s *Server) evaluateContainerResourceUsage(resourceName string, data cel.Data) float64 {
+	u, err := s.getResourceUsage(data.Pod.Name, data.Pod.Namespace, data.Container.Name)
+	if err != nil {
+		logger := log.FromContext(s.ctx)
+		logger.Error("failed to get resource usage", err, "pod", log.KRef(data.Pod.Namespace, data.Pod.Name), "container", data.Container.Name)
+		return 0
+	}
+	if u.Usage == nil {
+		return 0
+	}
+	r := u.Usage[resourceName]
+	if r.Value != nil {
+		return r.Value.AsApproximateFloat64()
+	}
+
+	if r.Expression != nil {
+		eval, err := s.env.Compile(*r.Expression)
+		if err != nil {
+			logger := log.FromContext(s.ctx)
+			logger.Error("failed to compile expression", err, "expression", r)
+			return 0
+		}
+
+		out, err := eval.EvaluateFloat64(data)
+		if err != nil {
+			logger := log.FromContext(s.ctx)
+			logger.Error("failed to evaluate expression", err, "expression", r)
+			return 0
+		}
+		return out
+	}
+	return 0
+}
+
+func (s *Server) podResourceUsage(resourceName, podNamespace, podName string) float64 {
+	pod, ok := s.podCacheGetter.GetWithNamespace(podName, podNamespace)
+	if !ok {
+		return 0
+	}
+
+	node, ok := s.nodeCacheGetter.Get(pod.Spec.NodeName)
+	if !ok {
+		return 0
+	}
+
+	data := cel.Data{
+		Node: node,
+		Pod:  pod,
+	}
+
+	sum := 0.0
+	for _, c := range pod.Spec.Containers {
+		c := c
+		data.Container = &c
+		sum += s.evaluateContainerResourceUsage(resourceName, data)
+	}
+	return sum
+}
+
+func (s *Server) nodeResourceUsage(resourceName, nodeName string) float64 {
+	node, ok := s.nodeCacheGetter.Get(nodeName)
+	if !ok {
+		return 0
+	}
+
+	pods, ok := s.dataSource.ListPods(nodeName)
+	if !ok {
+		return 0
+	}
+
+	data := cel.Data{
+		Node: node,
+	}
+
+	sum := 0.0
+	for _, pi := range pods {
+		pod, ok := s.podCacheGetter.GetWithNamespace(pi.Name, pi.Namespace)
+		if !ok {
+			continue
+		}
+		data.Pod = pod
+		for _, c := range pod.Spec.Containers {
+			c := c
+			data.Container = &c
+			sum += s.evaluateContainerResourceUsage(resourceName, data)
+		}
+	}
+	return sum
+}
+
+func (s *Server) getResourceUsage(podName, podNamespace, containerName string) (*internalversion.ResourceUsageContainer, error) {
+	u, has := slices.Find(s.resourceUsages.Get(), func(a *internalversion.ResourceUsage) bool {
+		return a.Name == podName && a.Namespace == podNamespace
+	})
+	if has {
+		u, found := findUsageInUsages(containerName, u.Spec.Usages)
+		if found {
+			return u, nil
+		}
+		return nil, fmt.Errorf("no resource usage found for container %q in pod %q", containerName, log.KRef(podNamespace, podName))
+	}
+
+	for _, cl := range s.clusterResourceUsages.Get() {
+		if !cl.Spec.Selector.Match(podName, podNamespace) {
+			continue
+		}
+
+		u, found := findUsageInUsages(containerName, cl.Spec.Usages)
+		if found {
+			return u, nil
+		}
+	}
+
+	return nil, fmt.Errorf("no resource usage found for container %q in pod %q", containerName, log.KRef(podNamespace, podName))
+}
+
+func findUsageInUsages(containerName string, usages []internalversion.ResourceUsageContainer) (*internalversion.ResourceUsageContainer, bool) {
+	var defaultUsage *internalversion.ResourceUsageContainer
+	for i, u := range usages {
+		if len(u.Containers) == 0 && defaultUsage == nil {
+			defaultUsage = &usages[i]
+			continue
+		}
+		if slices.Contains(u.Containers, containerName) {
+			return &u, true
+		}
+	}
+	return defaultUsage, defaultUsage != nil
+}
diff --git a/pkg/kwok/server/server.go b/pkg/kwok/server/server.go
index 366fe8ee..af084c4d 100644
--- a/pkg/kwok/server/server.go
+++ b/pkg/kwok/server/server.go
@@ -22,6 +22,7 @@ import (
 	"net"
 	"net/http"
 	"net/http/httptest"
+	"sync"
 	"time"
 
 	"github.com/emicklei/go-restful/v3"
@@ -35,6 +36,7 @@ import (
 	"sigs.k8s.io/kwok/pkg/client/clientset/versioned"
 	"sigs.k8s.io/kwok/pkg/config/resources"
 	"sigs.k8s.io/kwok/pkg/kwok/metrics"
+	"sigs.k8s.io/kwok/pkg/kwok/metrics/cel"
 	"sigs.k8s.io/kwok/pkg/log"
 	"sigs.k8s.io/kwok/pkg/utils/informer"
 	"sigs.k8s.io/kwok/pkg/utils/maps"
@@ -48,6 +50,8 @@ const (
 
 // Server is a server that can serve HTTP/HTTPS requests.
 type Server struct {
+	ctx context.Context
+
 	typedKwokClient versioned.Interface
 
 	enableCRDs []string
@@ -58,18 +62,25 @@ type Server struct {
 	streamCreationTimeout time.Duration
 	bufPool               *pools.Pool[[]byte]
 
-	clusterPortForwards resources.Getter[[]*internalversion.ClusterPortForward]
-	portForwards        resources.Getter[[]*internalversion.PortForward]
-	clusterExecs        resources.Getter[[]*internalversion.ClusterExec]
-	execs               resources.Getter[[]*internalversion.Exec]
-	clusterLogs         resources.Getter[[]*internalversion.ClusterLogs]
-	logs                resources.Getter[[]*internalversion.Logs]
-	clusterAttaches     resources.Getter[[]*internalversion.ClusterAttach]
-	attaches            resources.Getter[[]*internalversion.Attach]
-	metrics             resources.Getter[[]*internalversion.Metric]
+	clusterPortForwards   resources.Getter[[]*internalversion.ClusterPortForward]
+	portForwards          resources.Getter[[]*internalversion.PortForward]
+	clusterExecs          resources.Getter[[]*internalversion.ClusterExec]
+	execs                 resources.Getter[[]*internalversion.Exec]
+	clusterLogs           resources.Getter[[]*internalversion.ClusterLogs]
+	logs                  resources.Getter[[]*internalversion.Logs]
+	clusterAttaches       resources.Getter[[]*internalversion.ClusterAttach]
+	attaches              resources.Getter[[]*internalversion.Attach]
+	clusterResourceUsages resources.Getter[[]*internalversion.ClusterResourceUsage]
+	resourceUsages        resources.Getter[[]*internalversion.ResourceUsage]
+	metrics               resources.Getter[[]*internalversion.Metric]
 
 	metricsUpdateHandler maps.SyncMap[string, *metrics.UpdateHandler]
 
+	cumulatives    map[string]cumulative
+	cumulativesMut sync.Mutex
+
+	env *cel.Environment
+
 	dataSource      DataSource
 	nodeCacheGetter informer.Getter[*corev1.Node]
 	podCacheGetter  informer.Getter[*corev1.Pod]
@@ -87,15 +98,17 @@ type Config struct {
 	TypedKwokClient versioned.Interface
 	EnableCRDs      []string
 
-	ClusterPortForwards []*internalversion.ClusterPortForward
-	PortForwards        []*internalversion.PortForward
-	ClusterExecs        []*internalversion.ClusterExec
-	Execs               []*internalversion.Exec
-	ClusterLogs         []*internalversion.ClusterLogs
-	Logs                []*internalversion.Logs
-	ClusterAttaches     []*internalversion.ClusterAttach
-	Attaches            []*internalversion.Attach
-	Metrics             []*internalversion.Metric
+	ClusterPortForwards   []*internalversion.ClusterPortForward
+	PortForwards          []*internalversion.PortForward
+	ClusterExecs          []*internalversion.ClusterExec
+	Execs                 []*internalversion.Exec
+	ClusterLogs           []*internalversion.ClusterLogs
+	Logs                  []*internalversion.Logs
+	ClusterAttaches       []*internalversion.ClusterAttach
+	Attaches              []*internalversion.Attach
+	ClusterResourceUsages []*internalversion.ClusterResourceUsage
+	ResourceUsages        []*internalversion.ResourceUsage
+	Metrics               []*internalversion.Metric
 
 	DataSource      DataSource
 	NodeCacheGetter informer.Getter[*corev1.Node]
@@ -113,15 +126,19 @@ func NewServer(conf Config) (*Server, error) {
 		idleTimeout:           1 * time.Hour,
 		streamCreationTimeout: remotecommandconsts.DefaultStreamCreationTimeout,
 
-		clusterPortForwards: resources.NewStaticGetter(conf.ClusterPortForwards),
-		portForwards:        resources.NewStaticGetter(conf.PortForwards),
-		clusterExecs:        resources.NewStaticGetter(conf.ClusterExecs),
-		execs:               resources.NewStaticGetter(conf.Execs),
-		clusterLogs:         resources.NewStaticGetter(conf.ClusterLogs),
-		logs:                resources.NewStaticGetter(conf.Logs),
-		clusterAttaches:     resources.NewStaticGetter(conf.ClusterAttaches),
-		attaches:            resources.NewStaticGetter(conf.Attaches),
-		metrics:             resources.NewStaticGetter(conf.Metrics),
+		clusterPortForwards:   resources.NewStaticGetter(conf.ClusterPortForwards),
+		portForwards:          resources.NewStaticGetter(conf.PortForwards),
+		clusterExecs:          resources.NewStaticGetter(conf.ClusterExecs),
+		execs:                 resources.NewStaticGetter(conf.Execs),
+		clusterLogs:           resources.NewStaticGetter(conf.ClusterLogs),
+		logs:                  resources.NewStaticGetter(conf.Logs),
+		clusterAttaches:       resources.NewStaticGetter(conf.ClusterAttaches),
+		attaches:              resources.NewStaticGetter(conf.Attaches),
+		clusterResourceUsages: resources.NewStaticGetter(conf.ClusterResourceUsages),
+		resourceUsages:        resources.NewStaticGetter(conf.ResourceUsages),
+		metrics:               resources.NewStaticGetter(conf.Metrics),
+
+		cumulatives: map[string]cumulative{},
 
 		dataSource:      conf.DataSource,
 		podCacheGetter:  conf.PodCacheGetter,
@@ -328,6 +345,52 @@ func (s *Server) initWatchCRD(ctx context.Context) ([]resources.Starter, error)
 			)
 			starters = append(starters, attaches)
 			s.attaches = attaches
+		case v1alpha1.ClusterResourceUsageKind:
+			if len(s.clusterResourceUsages.Get()) != 0 {
+				return nil, fmt.Errorf("cluster resource usage already exists, cannot watch CRD")
+			}
+			clusterResourceUsages := resources.NewDynamicGetter[
+				[]*internalversion.ClusterResourceUsage,
+				*v1alpha1.ClusterResourceUsage,
+				*v1alpha1.ClusterResourceUsageList,
+			](
+				cli.KwokV1alpha1().ClusterResourceUsages(),
+				func(objs []*v1alpha1.ClusterResourceUsage) []*internalversion.ClusterResourceUsage {
+					return slices.FilterAndMap(objs, func(obj *v1alpha1.ClusterResourceUsage) (*internalversion.ClusterResourceUsage, bool) {
+						r, err := internalversion.ConvertToInternalClusterResourceUsage(obj)
+						if err != nil {
+							logger.Error("failed to convert to internal cluster resource usage", err, "obj", obj)
+							return nil, false
+						}
+						return r, true
+					})
+				},
+			)
+			starters = append(starters, clusterResourceUsages)
+			s.clusterResourceUsages = clusterResourceUsages
+		case v1alpha1.ResourceUsageKind:
+			if len(s.resourceUsages.Get()) != 0 {
+				return nil, fmt.Errorf("resource usage already exists, cannot watch CRD")
+			}
+			resourceUsages := resources.NewDynamicGetter[
+				[]*internalversion.ResourceUsage,
+				*v1alpha1.ResourceUsage,
+				*v1alpha1.ResourceUsageList,
+			](
+				cli.KwokV1alpha1().ResourceUsages(""),
+				func(objs []*v1alpha1.ResourceUsage) []*internalversion.ResourceUsage {
+					return slices.FilterAndMap(objs, func(obj *v1alpha1.ResourceUsage) (*internalversion.ResourceUsage, bool) {
+						r, err := internalversion.ConvertToInternalResourceUsage(obj)
+						if err != nil {
+							logger.Error("failed to convert to internal resource usage", err, "obj", obj)
+							return nil, false
+						}
+						return r, true
+					})
+				},
+			)
+			starters = append(starters, resourceUsages)
+			s.resourceUsages = resourceUsages
 		case v1alpha1.MetricKind:
 			if len(s.metrics.Get()) != 0 {
 				return nil, fmt.Errorf("metrics already exists, cannot watch CRD")
@@ -401,6 +464,8 @@ func (s *Server) Run(ctx context.Context, address string, certFile, privateKeyFi
 	ctx, cancel := context.WithCancel(ctx)
 	defer cancel()
 
+	s.ctx = ctx
+
 	errCh := make(chan error, 1)
 
 	if certFile != "" && privateKeyFile != "" {
diff --git a/pkg/kwokctl/components/metrics_server.go b/pkg/kwokctl/components/metrics_server.go
index e0c7f622..8a72a5a8 100644
--- a/pkg/kwokctl/components/metrics_server.go
+++ b/pkg/kwokctl/components/metrics_server.go
@@ -47,6 +47,7 @@ func BuildMetricsServerComponent(conf BuildMetricsServerComponentConfig) (compon
 	metricsServerArgs := []string{
 		"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
 		"--kubelet-use-node-status-port",
+		"--kubelet-insecure-tls", // TODO: remove this flag
 		"--metric-resolution=15s",
 	}
 
diff --git a/pkg/kwokctl/components/metrics_server_apiservice.yaml.tpl b/pkg/kwokctl/components/metrics_server_apiservice.yaml.tpl
index 06af3a4a..25499e7d 100644
--- a/pkg/kwokctl/components/metrics_server_apiservice.yaml.tpl
+++ b/pkg/kwokctl/components/metrics_server_apiservice.yaml.tpl
@@ -1,8 +1,6 @@
 apiVersion: apiregistration.k8s.io/v1
 kind: APIService
 metadata:
-  labels:
-    k8s-app: metrics-server
   name: v1beta1.metrics.k8s.io
 spec:
   group: metrics.k8s.io
@@ -11,29 +9,15 @@ spec:
   service:
     name: metrics-server
     namespace: kube-system
-    {{ if .Port }}
     port: {{ .Port }}
-    {{ end }}
   version: v1beta1
   versionPriority: 100
 ---
 apiVersion: v1
 kind: Service
 metadata:
-  labels:
-    k8s-app: metrics-server
   name: metrics-server
   namespace: kube-system
 spec:
-{{ if .ExternalName }}
   externalName: {{ .ExternalName }}
   type: ExternalName
-{{ else }}
-  ports:
-  - name: https
-    port: 443
-    protocol: TCP
-    targetPort: https
-  selector:
-    k8s-app: metrics-server
-{{ end }}
diff --git a/pkg/kwokctl/runtime/cluster.go b/pkg/kwokctl/runtime/cluster.go
index d327b6bf..0ab20383 100644
--- a/pkg/kwokctl/runtime/cluster.go
+++ b/pkg/kwokctl/runtime/cluster.go
@@ -211,6 +211,16 @@ func (c *Cluster) Save(ctx context.Context) error {
 		objs = appendIntoInternalObjects(objs, stages...)
 	}
 
+	if !slices.Contains(conf.Options.EnableCRDs, v1alpha1.ResourceUsageKind) {
+		stages := config.FilterWithTypeFromContext[*internalversion.ResourceUsage](ctx)
+		objs = appendIntoInternalObjects(objs, stages...)
+	}
+
+	if !slices.Contains(conf.Options.EnableCRDs, v1alpha1.ClusterResourceUsageKind) {
+		stages := config.FilterWithTypeFromContext[*internalversion.ClusterResourceUsage](ctx)
+		objs = appendIntoInternalObjects(objs, stages...)
+	}
+
 	if !slices.Contains(conf.Options.EnableCRDs, v1alpha1.AttachKind) {
 		stages := config.FilterWithTypeFromContext[*internalversion.Attach](ctx)
 		objs = appendIntoInternalObjects(objs, stages...)
@@ -567,14 +577,16 @@ func (c *Cluster) InitCRDs(ctx context.Context) error {
 }
 
 var crdDefines = map[string][]byte{
-	v1alpha1.StageKind:              crd.Stage,
-	v1alpha1.AttachKind:             crd.Attach,
-	v1alpha1.ClusterAttachKind:      crd.ClusterAttach,
-	v1alpha1.ExecKind:               crd.Exec,
-	v1alpha1.ClusterExecKind:        crd.ClusterExec,
-	v1alpha1.PortForwardKind:        crd.PortForward,
-	v1alpha1.ClusterPortForwardKind: crd.ClusterPortForward,
-	v1alpha1.LogsKind:               crd.Logs,
-	v1alpha1.ClusterLogsKind:        crd.ClusterLogs,
-	v1alpha1.MetricKind:             crd.Metric,
+	v1alpha1.StageKind:                crd.Stage,
+	v1alpha1.AttachKind:               crd.Attach,
+	v1alpha1.ClusterAttachKind:        crd.ClusterAttach,
+	v1alpha1.ExecKind:                 crd.Exec,
+	v1alpha1.ClusterExecKind:          crd.ClusterExec,
+	v1alpha1.PortForwardKind:          crd.PortForward,
+	v1alpha1.ClusterPortForwardKind:   crd.ClusterPortForward,
+	v1alpha1.LogsKind:                 crd.Logs,
+	v1alpha1.ClusterLogsKind:          crd.ClusterLogs,
+	v1alpha1.ResourceUsageKind:        crd.ResourceUsage,
+	v1alpha1.ClusterResourceUsageKind: crd.ClusterResourceUsage,
+	v1alpha1.MetricKind:               crd.Metric,
 }
diff --git a/pkg/kwokctl/runtime/kind/cluster.go b/pkg/kwokctl/runtime/kind/cluster.go
index 32a70753..61de3da4 100644
--- a/pkg/kwokctl/runtime/kind/cluster.go
+++ b/pkg/kwokctl/runtime/kind/cluster.go
@@ -1429,7 +1429,10 @@ func (c *Cluster) InitCRs(ctx context.Context) error {
 
 	buf := bytes.NewBuffer(nil)
 	if conf.EnableMetricsServer {
-		apiservice, err := components.BuildMetricsServerAPIService(components.BuildMetricsServerAPIServiceConfig{})
+		apiservice, err := components.BuildMetricsServerAPIService(components.BuildMetricsServerAPIServiceConfig{
+			Port:         4443,
+			ExternalName: "localhost",
+		})
 		if err != nil {
 			return err
 		}
diff --git a/site/content/en/docs/generated/apis.md b/site/content/en/docs/generated/apis.md
index c8d4dba9..2f176f54 100644
--- a/site/content/en/docs/generated/apis.md
+++ b/site/content/en/docs/generated/apis.md
@@ -295,6 +295,9 @@ Resource Types:
 <a href="#kwok.x-k8s.io/v1alpha1.ClusterPortForward">ClusterPortForward</a>
 </li>
 <li>
+<a href="#kwok.x-k8s.io/v1alpha1.ClusterResourceUsage">ClusterResourceUsage</a>
+</li>
+<li>
 <a href="#kwok.x-k8s.io/v1alpha1.Exec">Exec</a>
 </li>
 <li>
@@ -307,6 +310,9 @@ Resource Types:
 <a href="#kwok.x-k8s.io/v1alpha1.PortForward">PortForward</a>
 </li>
 <li>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsage">ResourceUsage</a>
+</li>
+<li>
 <a href="#kwok.x-k8s.io/v1alpha1.Stage">Stage</a>
 </li></ul>
 <h3 id="kwok.x-k8s.io/v1alpha1.Attach">
@@ -821,6 +827,111 @@ ClusterPortForwardStatus
 </tr>
 </tbody>
 </table>
+<h3 id="kwok.x-k8s.io/v1alpha1.ClusterResourceUsage">
+ClusterResourceUsage
+<a href="#kwok.x-k8s.io%2fv1alpha1.ClusterResourceUsage"> #</a>
+</h3>
+<p>
+<p>ClusterResourceUsage provides cluster-wide resource usage.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>apiVersion</code>
+string
+</td>
+<td>
+<code>
+kwok.x-k8s.io/v1alpha1
+</code>
+</td>
+</tr>
+<tr>
+<td>
+<code>kind</code>
+string
+</td>
+<td><code>ClusterResourceUsage</code></td>
+</tr>
+<tr>
+<td>
+<code>metadata</code>
+<em>
+<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta">
+Kubernetes meta/v1.ObjectMeta
+</a>
+</em>
+</td>
+<td>
+<p>Standard list metadata.
+More info: <a href="https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata">https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata</a></p>
+Refer to the Kubernetes API documentation for the fields of the
+<code>metadata</code> field.
+</td>
+</tr>
+<tr>
+<td>
+<code>spec</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ClusterResourceUsageSpec">
+ClusterResourceUsageSpec
+</a>
+</em>
+</td>
+<td>
+<p>Spec holds spec for cluster resource usage.</p>
+<table>
+<tr>
+<td>
+<code>selector</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ObjectSelector">
+ObjectSelector
+</a>
+</em>
+</td>
+<td>
+<p>Selector is a selector to filter pods to configure.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>usages</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageContainer">
+[]ResourceUsageContainer
+</a>
+</em>
+</td>
+<td>
+<p>Usages is a list of resource usage for the pod.</p>
+</td>
+</tr>
+</table>
+</td>
+</tr>
+<tr>
+<td>
+<code>status</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ClusterResourceUsageStatus">
+ClusterResourceUsageStatus
+</a>
+</em>
+</td>
+<td>
+<p>Status holds status for cluster resource usage</p>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="kwok.x-k8s.io/v1alpha1.Exec">
 Exec
 <a href="#kwok.x-k8s.io%2fv1alpha1.Exec"> #</a>
@@ -1200,6 +1311,98 @@ PortForwardStatus
 </tr>
 </tbody>
 </table>
+<h3 id="kwok.x-k8s.io/v1alpha1.ResourceUsage">
+ResourceUsage
+<a href="#kwok.x-k8s.io%2fv1alpha1.ResourceUsage"> #</a>
+</h3>
+<p>
+<p>ResourceUsage provides resource usage for a single pod.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>apiVersion</code>
+string
+</td>
+<td>
+<code>
+kwok.x-k8s.io/v1alpha1
+</code>
+</td>
+</tr>
+<tr>
+<td>
+<code>kind</code>
+string
+</td>
+<td><code>ResourceUsage</code></td>
+</tr>
+<tr>
+<td>
+<code>metadata</code>
+<em>
+<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta">
+Kubernetes meta/v1.ObjectMeta
+</a>
+</em>
+</td>
+<td>
+<p>Standard list metadata.
+More info: <a href="https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata">https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata</a></p>
+Refer to the Kubernetes API documentation for the fields of the
+<code>metadata</code> field.
+</td>
+</tr>
+<tr>
+<td>
+<code>spec</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageSpec">
+ResourceUsageSpec
+</a>
+</em>
+</td>
+<td>
+<p>Spec holds spec for resource usage.</p>
+<table>
+<tr>
+<td>
+<code>usages</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageContainer">
+[]ResourceUsageContainer
+</a>
+</em>
+</td>
+<td>
+<p>Usages is a list of resource usage for the pod.</p>
+</td>
+</tr>
+</table>
+</td>
+</tr>
+<tr>
+<td>
+<code>status</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageStatus">
+ResourceUsageStatus
+</a>
+</em>
+</td>
+<td>
+<p>Status holds status for resource usage</p>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="kwok.x-k8s.io/v1alpha1.Stage">
 Stage
 <a href="#kwok.x-k8s.io%2fv1alpha1.Stage"> #</a>
@@ -3846,6 +4049,87 @@ ClusterPortForwardStatus
 </tr>
 </tbody>
 </table>
+<h3 id="kwok.x-k8s.io/v1alpha1.ClusterResourceUsageSpec">
+ClusterResourceUsageSpec
+<a href="#kwok.x-k8s.io%2fv1alpha1.ClusterResourceUsageSpec"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.ClusterResourceUsage">ClusterResourceUsage</a>
+</p>
+<p>
+<p>ClusterResourceUsageSpec holds spec for cluster resource usage.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>selector</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ObjectSelector">
+ObjectSelector
+</a>
+</em>
+</td>
+<td>
+<p>Selector is a selector to filter pods to configure.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>usages</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageContainer">
+[]ResourceUsageContainer
+</a>
+</em>
+</td>
+<td>
+<p>Usages is a list of resource usage for the pod.</p>
+</td>
+</tr>
+</tbody>
+</table>
+<h3 id="kwok.x-k8s.io/v1alpha1.ClusterResourceUsageStatus">
+ClusterResourceUsageStatus
+<a href="#kwok.x-k8s.io%2fv1alpha1.ClusterResourceUsageStatus"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.ClusterResourceUsage">ClusterResourceUsage</a>
+</p>
+<p>
+<p>ClusterResourceUsageStatus holds status for cluster resource usage</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>conditions</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.Condition">
+[]Condition
+</a>
+</em>
+</td>
+<td>
+<p>Conditions holds conditions for cluster resource usage</p>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="kwok.x-k8s.io/v1alpha1.Condition">
 Condition
 <a href="#kwok.x-k8s.io%2fv1alpha1.Condition"> #</a>
@@ -3862,6 +4146,8 @@ Condition
 , 
 <a href="#kwok.x-k8s.io/v1alpha1.ClusterPortForwardStatus">ClusterPortForwardStatus</a>
 , 
+<a href="#kwok.x-k8s.io/v1alpha1.ClusterResourceUsageStatus">ClusterResourceUsageStatus</a>
+, 
 <a href="#kwok.x-k8s.io/v1alpha1.ExecStatus">ExecStatus</a>
 , 
 <a href="#kwok.x-k8s.io/v1alpha1.LogsStatus">LogsStatus</a>
@@ -3870,6 +4156,8 @@ Condition
 , 
 <a href="#kwok.x-k8s.io/v1alpha1.PortForwardStatus">PortForwardStatus</a>
 , 
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageStatus">ResourceUsageStatus</a>
+, 
 <a href="#kwok.x-k8s.io/v1alpha1.StageStatus">StageStatus</a>
 </p>
 <p>
@@ -4900,6 +5188,8 @@ ObjectSelector
 <a href="#kwok.x-k8s.io/v1alpha1.ClusterLogsSpec">ClusterLogsSpec</a>
 , 
 <a href="#kwok.x-k8s.io/v1alpha1.ClusterPortForwardSpec">ClusterPortForwardSpec</a>
+, 
+<a href="#kwok.x-k8s.io/v1alpha1.ClusterResourceUsageSpec">ClusterResourceUsageSpec</a>
 </p>
 <p>
 <p>ObjectSelector holds information how to match based on namespace and name.</p>
@@ -5006,6 +5296,164 @@ PortForwardStatus
 </tr>
 </tbody>
 </table>
+<h3 id="kwok.x-k8s.io/v1alpha1.ResourceUsageContainer">
+ResourceUsageContainer
+<a href="#kwok.x-k8s.io%2fv1alpha1.ResourceUsageContainer"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.ClusterResourceUsageSpec">ClusterResourceUsageSpec</a>
+, 
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageSpec">ResourceUsageSpec</a>
+</p>
+<p>
+<p>ResourceUsageContainer holds spec for resource usage container.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>containers</code>
+<em>
+[]string
+</em>
+</td>
+<td>
+<p>Containers is list of container names.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>usage</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageValue">
+map[string]sigs.k8s.io/kwok/pkg/apis/v1alpha1.ResourceUsageValue
+</a>
+</em>
+</td>
+<td>
+<p>Usage is a list of resource usage for the container.</p>
+</td>
+</tr>
+</tbody>
+</table>
+<h3 id="kwok.x-k8s.io/v1alpha1.ResourceUsageSpec">
+ResourceUsageSpec
+<a href="#kwok.x-k8s.io%2fv1alpha1.ResourceUsageSpec"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsage">ResourceUsage</a>
+</p>
+<p>
+<p>ResourceUsageSpec holds spec for resource usage.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>usages</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageContainer">
+[]ResourceUsageContainer
+</a>
+</em>
+</td>
+<td>
+<p>Usages is a list of resource usage for the pod.</p>
+</td>
+</tr>
+</tbody>
+</table>
+<h3 id="kwok.x-k8s.io/v1alpha1.ResourceUsageStatus">
+ResourceUsageStatus
+<a href="#kwok.x-k8s.io%2fv1alpha1.ResourceUsageStatus"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsage">ResourceUsage</a>
+</p>
+<p>
+<p>ResourceUsageStatus holds status for resource usage</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>conditions</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.Condition">
+[]Condition
+</a>
+</em>
+</td>
+<td>
+<p>Conditions holds conditions for resource usage</p>
+</td>
+</tr>
+</tbody>
+</table>
+<h3 id="kwok.x-k8s.io/v1alpha1.ResourceUsageValue">
+ResourceUsageValue
+<a href="#kwok.x-k8s.io%2fv1alpha1.ResourceUsageValue"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.ResourceUsageContainer">ResourceUsageContainer</a>
+</p>
+<p>
+<p>ResourceUsageValue holds value for resource usage.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>value</code>
+<em>
+k8s.io/apimachinery/pkg/api/resource.Quantity
+</em>
+</td>
+<td>
+<p>Value is the value for resource usage.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>expression</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Expression is the expression for resource usage.</p>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="kwok.x-k8s.io/v1alpha1.SecurityContext">
 SecurityContext
 <a href="#kwok.x-k8s.io%2fv1alpha1.SecurityContext"> #</a>


Test patch:
diff --git a/pkg/kwok/controllers/utils_test.go b/pkg/kwok/controllers/utils_test.go
index 080cff4f..984a071d 100644
--- a/pkg/kwok/controllers/utils_test.go
+++ b/pkg/kwok/controllers/utils_test.go
@@ -17,9 +17,15 @@ limitations under the License.
 package controllers
 
 import (
+	"encoding/json"
+	"errors"
 	"net"
 	"reflect"
+	"syscall"
 	"testing"
+
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/runtime/schema"
 )
 
 func Test_parseCIDR(t *testing.T) {
@@ -131,3 +137,54 @@ func Test_ipPool_new(t *testing.T) {
 		})
 	}
 }
+
+func genResource(resource string) schema.GroupResource {
+	return schema.GroupResource{Group: "", Resource: resource}
+}
+
+func Test_shouldRetry(t *testing.T) {
+	var testCases = []struct {
+		name     string
+		input    error
+		expected bool
+	}{
+		{
+			name:     "connection refused error",
+			input:    syscall.ECONNREFUSED,
+			expected: true,
+		},
+		{
+			name:     "too many request error",
+			input:    apierrors.NewTooManyRequests("foo", 1),
+			expected: true,
+		},
+		{
+			name:     "resource not found error",
+			input:    apierrors.NewNotFound(genResource("foo"), "bar"),
+			expected: false,
+		},
+		{
+			name:     "resource conflict error",
+			input:    apierrors.NewConflict(genResource("foo"), "bar", errors.New("message")),
+			expected: false,
+		},
+		{
+			name:     "marshal error",
+			input:    &json.UnsupportedTypeError{},
+			expected: false,
+		},
+		{
+			name:     "unknown error",
+			input:    errors.New("an weird error"),
+			expected: false,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			if shouldRetry(tc.input) != tc.expected {
+				t.Errorf("expected: %t", tc.expected)
+			}
+		})
+	}
+}
diff --git a/pkg/kwok/metrics/cel/evaluate_test.go b/pkg/kwok/metrics/cel/evaluate_test.go
index aaaaace8..8ce09abb 100644
--- a/pkg/kwok/metrics/cel/evaluate_test.go
+++ b/pkg/kwok/metrics/cel/evaluate_test.go
@@ -21,6 +21,7 @@ import (
 	"time"
 
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
@@ -62,3 +63,55 @@ func TestNodeEvaluation(t *testing.T) {
 		t.Errorf("expected %v, got %v", 17280, actual)
 	}
 }
+
+func TestResourceEvaluation(t *testing.T) {
+	n := &corev1.Node{
+		Status: corev1.NodeStatus{
+			Allocatable: map[corev1.ResourceName]resource.Quantity{
+				corev1.ResourceCPU: resource.MustParse("90m"),
+			},
+		},
+	}
+	p := &corev1.Pod{
+		ObjectMeta: metav1.ObjectMeta{
+			Annotations: map[string]string{
+				"cpu_usage": "10m",
+			},
+		},
+		Spec: corev1.PodSpec{
+			Containers: []corev1.Container{
+				{
+					Resources: corev1.ResourceRequirements{
+						Requests: map[corev1.ResourceName]resource.Quantity{
+							corev1.ResourceCPU: resource.MustParse("10m"),
+						},
+					},
+				},
+			},
+		},
+	}
+
+	exp := `( Quantity("10m") + Quantity(pod.metadata.annotations["cpu_usage"]) + node.status.allocatable["cpu"] + pod.spec.containers[0].resources.requests["cpu"] ) * 1.5 * 10`
+
+	env, err := NewEnvironment(NodeEvaluatorConfig{})
+	if err != nil {
+		t.Fatalf("failed to instantiate node Evaluator: %v", err)
+	}
+
+	eval, err := env.Compile(exp)
+	if err != nil {
+		t.Fatalf("failed to compile expression: %v", err)
+	}
+
+	actual, err := eval.EvaluateFloat64(Data{
+		Node: n,
+		Pod:  p,
+	})
+	if err != nil {
+		t.Fatalf("evaluation failed: %v", err)
+	}
+
+	if actual != 18 {
+		t.Errorf("expected %v, got %v", 18, actual)
+	}
+}
diff --git a/test/e2e/helper/node.go b/test/e2e/helper/node.go
index c2fe70c3..792a31e6 100644
--- a/test/e2e/helper/node.go
+++ b/test/e2e/helper/node.go
@@ -33,7 +33,8 @@ func NewNodeBuilder(name string) *NodeBuilder {
 			ObjectMeta: metav1.ObjectMeta{
 				Name: name,
 				Annotations: map[string]string{
-					"kwok.x-k8s.io/node": "fake",
+					"kwok.x-k8s.io/node":                   "fake",
+					"metrics.k8s.io/resource-metrics-path": "/metrics/nodes/" + name + "/metrics/resource",
 				},
 				Labels: map[string]string{
 					"type": "kwok",
diff --git a/test/e2e/kwokctl/binary/kubectl_test.go b/test/e2e/kwokctl/binary/kubectl_test.go
index 1950305a..9a763c47 100644
--- a/test/e2e/kwokctl/binary/kubectl_test.go
+++ b/test/e2e/kwokctl/binary/kubectl_test.go
@@ -65,3 +65,9 @@ func TestSnapshot(t *testing.T) {
 		Feature()
 	testEnv.Test(t, f0)
 }
+
+func TestResourceUsage(t *testing.T) {
+	f0 := e2e.CaseResourceUsage(kwokctlPath, clusterName).
+		Feature()
+	testEnv.Test(t, f0)
+}
diff --git a/test/e2e/kwokctl/binary/main_test.go b/test/e2e/kwokctl/binary/main_test.go
index 10b8a651..e7e558d1 100644
--- a/test/e2e/kwokctl/binary/main_test.go
+++ b/test/e2e/kwokctl/binary/main_test.go
@@ -44,6 +44,7 @@ var (
 	baseArgs    = []string{
 		"--kwok-controller-binary=" + kwokPath,
 		"--runtime=" + runtimeEnv,
+		"--enable-metrics-server",
 	}
 )
 
@@ -65,6 +66,8 @@ func TestMain(m *testing.M) {
 			"--config="+path.Join(rootDir, "test/e2e/logs.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/attach.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/exec.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/usage/usage-from-annotation.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/resource/metrics-resource.yaml"),
 		)...),
 		helper.CreateNamespace(namespace),
 	)
diff --git a/test/e2e/kwokctl/docker/kubectl_test.go b/test/e2e/kwokctl/docker/kubectl_test.go
index c890a672..873812a6 100644
--- a/test/e2e/kwokctl/docker/kubectl_test.go
+++ b/test/e2e/kwokctl/docker/kubectl_test.go
@@ -65,3 +65,9 @@ func TestSnapshot(t *testing.T) {
 		Feature()
 	testEnv.Test(t, f0)
 }
+
+func TestResourceUsage(t *testing.T) {
+	f0 := e2e.CaseResourceUsage(kwokctlPath, clusterName).
+		Feature()
+	testEnv.Test(t, f0)
+}
diff --git a/test/e2e/kwokctl/docker/main_test.go b/test/e2e/kwokctl/docker/main_test.go
index 5eb65f6b..b86ade7d 100644
--- a/test/e2e/kwokctl/docker/main_test.go
+++ b/test/e2e/kwokctl/docker/main_test.go
@@ -44,6 +44,7 @@ var (
 	baseArgs    = []string{
 		"--kwok-controller-image=" + testImage,
 		"--runtime=" + runtimeEnv,
+		"--enable-metrics-server",
 	}
 )
 
@@ -64,6 +65,8 @@ func TestMain(m *testing.M) {
 			"--config="+path.Join(rootDir, "test/e2e/logs.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/attach.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/exec.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/usage/usage-from-annotation.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/resource/metrics-resource.yaml"),
 		)...),
 		helper.CreateNamespace(namespace),
 	)
diff --git a/test/e2e/kwokctl/kind-podman/kubectl_test.go b/test/e2e/kwokctl/kind-podman/kubectl_test.go
index 1cfd4647..1f74bf2f 100644
--- a/test/e2e/kwokctl/kind-podman/kubectl_test.go
+++ b/test/e2e/kwokctl/kind-podman/kubectl_test.go
@@ -65,3 +65,9 @@ func TestSnapshot(t *testing.T) {
 		Feature()
 	testEnv.Test(t, f0)
 }
+
+func TestResourceUsage(t *testing.T) {
+	f0 := e2e.CaseResourceUsage(kwokctlPath, clusterName).
+		Feature()
+	testEnv.Test(t, f0)
+}
diff --git a/test/e2e/kwokctl/kind-podman/main_test.go b/test/e2e/kwokctl/kind-podman/main_test.go
index eeeb90f4..1ee35f95 100644
--- a/test/e2e/kwokctl/kind-podman/main_test.go
+++ b/test/e2e/kwokctl/kind-podman/main_test.go
@@ -44,6 +44,7 @@ var (
 	baseArgs    = []string{
 		"--kwok-controller-image=" + testImage,
 		"--runtime=" + runtimeEnv,
+		"--enable-metrics-server",
 	}
 )
 
@@ -64,6 +65,8 @@ func TestMain(m *testing.M) {
 			"--config="+path.Join(rootDir, "test/e2e/logs.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/attach.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/exec.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/usage/usage-from-annotation.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/resource/metrics-resource.yaml"),
 		)...),
 		helper.CreateNamespace(namespace),
 	)
diff --git a/test/e2e/kwokctl/kind/kubectl_test.go b/test/e2e/kwokctl/kind/kubectl_test.go
index 61883dd7..d1db8751 100644
--- a/test/e2e/kwokctl/kind/kubectl_test.go
+++ b/test/e2e/kwokctl/kind/kubectl_test.go
@@ -65,3 +65,9 @@ func TestSnapshot(t *testing.T) {
 		Feature()
 	testEnv.Test(t, f0)
 }
+
+func TestResourceUsage(t *testing.T) {
+	f0 := e2e.CaseResourceUsage(kwokctlPath, clusterName).
+		Feature()
+	testEnv.Test(t, f0)
+}
diff --git a/test/e2e/kwokctl/kind/main_test.go b/test/e2e/kwokctl/kind/main_test.go
index e04638b0..14a12056 100644
--- a/test/e2e/kwokctl/kind/main_test.go
+++ b/test/e2e/kwokctl/kind/main_test.go
@@ -44,6 +44,7 @@ var (
 	baseArgs    = []string{
 		"--kwok-controller-image=" + testImage,
 		"--runtime=" + runtimeEnv,
+		"--enable-metrics-server",
 	}
 )
 
@@ -64,6 +65,8 @@ func TestMain(m *testing.M) {
 			"--config="+path.Join(rootDir, "test/e2e/logs.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/attach.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/exec.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/usage/usage-from-annotation.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/resource/metrics-resource.yaml"),
 		)...),
 		helper.CreateNamespace(namespace),
 	)
diff --git a/test/e2e/kwokctl/nerdctl/kubectl_test.go b/test/e2e/kwokctl/nerdctl/kubectl_test.go
index 4d308d7d..3bb56384 100644
--- a/test/e2e/kwokctl/nerdctl/kubectl_test.go
+++ b/test/e2e/kwokctl/nerdctl/kubectl_test.go
@@ -65,3 +65,9 @@ func TestSnapshot(t *testing.T) {
 		Feature()
 	testEnv.Test(t, f0)
 }
+
+func TestResourceUsage(t *testing.T) {
+	f0 := e2e.CaseResourceUsage(kwokctlPath, clusterName).
+		Feature()
+	testEnv.Test(t, f0)
+}
diff --git a/test/e2e/kwokctl/nerdctl/main_test.go b/test/e2e/kwokctl/nerdctl/main_test.go
index ba3c85e6..3a83e7b1 100644
--- a/test/e2e/kwokctl/nerdctl/main_test.go
+++ b/test/e2e/kwokctl/nerdctl/main_test.go
@@ -44,6 +44,7 @@ var (
 	baseArgs    = []string{
 		"--kwok-controller-image=" + testImage,
 		"--runtime=" + runtimeEnv,
+		"--enable-metrics-server",
 	}
 )
 
@@ -64,6 +65,8 @@ func TestMain(m *testing.M) {
 			"--config="+path.Join(rootDir, "test/e2e/logs.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/attach.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/exec.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/usage/usage-from-annotation.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/resource/metrics-resource.yaml"),
 		)...),
 		helper.CreateNamespace(namespace),
 	)
diff --git a/test/e2e/kwokctl/podman/kubectl_test.go b/test/e2e/kwokctl/podman/kubectl_test.go
index 35223462..0709e81f 100644
--- a/test/e2e/kwokctl/podman/kubectl_test.go
+++ b/test/e2e/kwokctl/podman/kubectl_test.go
@@ -65,3 +65,9 @@ func TestSnapshot(t *testing.T) {
 		Feature()
 	testEnv.Test(t, f0)
 }
+
+func TestResourceUsage(t *testing.T) {
+	f0 := e2e.CaseResourceUsage(kwokctlPath, clusterName).
+		Feature()
+	testEnv.Test(t, f0)
+}
diff --git a/test/e2e/kwokctl/podman/main_test.go b/test/e2e/kwokctl/podman/main_test.go
index 5b716464..ed7cb319 100644
--- a/test/e2e/kwokctl/podman/main_test.go
+++ b/test/e2e/kwokctl/podman/main_test.go
@@ -44,6 +44,7 @@ var (
 	baseArgs    = []string{
 		"--kwok-controller-image=" + testImage,
 		"--runtime=" + runtimeEnv,
+		"--enable-metrics-server",
 	}
 )
 
@@ -64,6 +65,8 @@ func TestMain(m *testing.M) {
 			"--config="+path.Join(rootDir, "test/e2e/logs.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/attach.yaml"),
 			"--config="+path.Join(rootDir, "test/e2e/exec.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/usage/usage-from-annotation.yaml"),
+			"--config="+path.Join(rootDir, "kustomize/metrics/resource/metrics-resource.yaml"),
 		)...),
 		helper.CreateNamespace(namespace),
 	)
diff --git a/test/e2e/resource_usage.go b/test/e2e/resource_usage.go
new file mode 100644
index 00000000..7fa3551f
--- /dev/null
+++ b/test/e2e/resource_usage.go
@@ -0,0 +1,165 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package e2e
+
+import (
+	"bytes"
+	"context"
+	"strings"
+	"testing"
+	"time"
+
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/e2e-framework/klient/k8s"
+	"sigs.k8s.io/e2e-framework/klient/k8s/resources"
+	"sigs.k8s.io/e2e-framework/klient/wait"
+	"sigs.k8s.io/e2e-framework/pkg/envconf"
+	"sigs.k8s.io/e2e-framework/pkg/features"
+
+	"sigs.k8s.io/kwok/pkg/log"
+	"sigs.k8s.io/kwok/pkg/utils/exec"
+	"sigs.k8s.io/kwok/test/e2e/helper"
+)
+
+func CaseResourceUsage(kwokctlPath, clusterName string) *features.FeatureBuilder {
+	node0 := helper.NewNodeBuilder("node0").
+		Build()
+	pod0 := helper.NewPodBuilder("pod0").
+		WithNodeName(node0.Name).
+		Build()
+
+	node1 := helper.NewNodeBuilder("node1").
+		Build()
+	pod1 := helper.NewPodBuilder("pod1").
+		WithNodeName(node1.Name).
+		Build()
+
+	return features.New("ResourceUsage").
+		Setup(helper.CreateNode(node0)).
+		Setup(helper.CreatePod(pod0)).
+		Setup(helper.CreateNode(node1)).
+		Setup(helper.CreatePod(pod1)).
+		Assess("wait ready", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
+			_, err := helper.WaitForAllNodesReady()(ctx, cfg)
+			if err != nil {
+				t.Error(err)
+			}
+
+			_, err = helper.WaitForAllPodsReady()(ctx, cfg)
+			if err != nil {
+				t.Error(err)
+			}
+
+			return ctx
+		}).
+		Assess("test usage", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
+			logger := log.FromContext(ctx)
+
+			err := wait.For(
+				func(ctx context.Context) (done bool, err error) {
+					out := bytes.NewBuffer(nil)
+					_, err = exec.Command(exec.WithAllWriteTo(ctx, out), kwokctlPath, "--name", clusterName, "kubectl", "top", "pod")
+					if err != nil {
+						logger.Error("kubectl top pod", err)
+						return false, nil
+					}
+
+					output := out.String()
+					if strings.Contains(output, "Metrics API not available") || strings.Contains(output, "metrics not available yet") {
+						logger.Warn("kubectl top pod", "output", output)
+						return false, nil
+					}
+					if !strings.Contains(output, "1Mi") {
+						logger.Info("kubectl top pod", "output", output)
+						return false, nil
+					}
+					if !strings.Contains(output, "1m") && !strings.Contains(output, "2m") {
+						logger.Info("kubectl top pod", "output", output)
+						return false, nil
+					}
+					return true, nil
+				},
+				wait.WithContext(ctx),
+				wait.WithTimeout(5*time.Minute),
+			)
+			if err != nil {
+				t.Fatal(err)
+				return ctx
+			}
+
+			return ctx
+		}).
+		Assess("test modify usage", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
+			logger := log.FromContext(ctx)
+
+			client, err := resources.New(cfg.Client().RESTConfig())
+			if err != nil {
+				t.Error(err)
+			}
+
+			err = client.Patch(ctx, pod0, k8s.Patch{
+				PatchType: types.JSONPatchType,
+				Data:      []byte(`[{"op": "add", "path": "/metadata/annotations", "value": {"kwok.x-k8s.io/usage-cpu": "100m", "kwok.x-k8s.io/usage-memory": "100Mi"}}]`),
+			})
+			if err != nil {
+				t.Error(err)
+			}
+
+			err = wait.For(
+				func(ctx context.Context) (done bool, err error) {
+					out := bytes.NewBuffer(nil)
+					_, err = exec.Command(exec.WithAllWriteTo(ctx, out), kwokctlPath, "--name", clusterName, "kubectl", "top", "pod")
+					if err != nil {
+						logger.Error("kubectl top pod", err)
+						return false, nil
+					}
+
+					output := out.String()
+					if strings.Contains(output, "Metrics API not available") || strings.Contains(output, "metrics not available yet") {
+						logger.Warn("kubectl top pod", "output", output)
+						return false, nil
+					}
+					if !strings.Contains(output, "1Mi") {
+						logger.Info("kubectl top pod", "output", output)
+						return false, nil
+					}
+					if !strings.Contains(output, "1m") && !strings.Contains(output, "2m") {
+						logger.Info("kubectl top pod", "output", output)
+						return false, nil
+					}
+
+					if !strings.Contains(output, "100Mi") {
+						logger.Info("kubectl top pod", "output", output)
+						return false, nil
+					}
+					if !strings.Contains(output, "100m") && !strings.Contains(output, "101m") {
+						logger.Info("kubectl top pod", "output", output)
+						return false, nil
+					}
+					return true, nil
+				},
+				wait.WithContext(ctx),
+				wait.WithTimeout(5*time.Minute),
+			)
+			if err != nil {
+				t.Fatal(err)
+				return ctx
+			}
+
+			return ctx
+		})
+}
diff --git a/test/kwokctl/metric-node-resource.yaml b/test/kwokctl/metric-node-resource.yaml
deleted file mode 100644
index c3df72b9..00000000
--- a/test/kwokctl/metric-node-resource.yaml
+++ /dev/null
@@ -1,86 +0,0 @@
-kind: Metric
-apiVersion: kwok.x-k8s.io/v1alpha1
-metadata:
-  name: resource-default
-spec:
-  path: "/metrics/nodes/{nodeName}/metrics/resource"
-  metrics:
-  - name: scrape_error
-    dimension: node
-    help: "[ALPHA] 1 if there was an error while getting metrics from the node, 0 otherwise"
-    kind: gauge
-    value: '0'
-  - name: container_start_time_seconds
-    dimension: container
-    help: "[ALPHA] Start time of the container since unix epoch in seconds"
-    kind: gauge
-    labels:
-    - name: container
-      value: 'container.name'
-    - name: namespace
-      value: 'pod.metadata.namespace'
-    - name: pod
-      value: 'pod.metadata.name'
-    value: 'pod.SinceSecond()'
-
-  # CPU
-  - name: container_cpu_usage_seconds_total
-    dimension: container
-    help: "[ALPHA] Cumulative cpu time consumed by the container in core-seconds"
-    kind: counter
-    labels:
-    - name: container
-      value: 'container.name'
-    - name: namespace
-      value: 'pod.metadata.namespace'
-    - name: pod
-      value: 'pod.metadata.name'
-    value: 'pod.SinceSecond() / 10.0'
-  - name: pod_cpu_usage_seconds_total
-    dimension: pod
-    help: "[ALPHA] Cumulative cpu time consumed by the pod in core-seconds"
-    kind: counter
-    labels:
-    - name: namespace
-      value: 'pod.metadata.namespace'
-    - name: pod
-      value: 'pod.metadata.name'
-    # TODO: sum(container_cpu_usage_seconds_total) by (pod)
-    value: 'pod.SinceSecond()'
-  - name: node_cpu_usage_seconds_total
-    dimension: node
-    help: "[ALPHA] Cumulative cpu time consumed by the node in core-seconds"
-    kind: counter
-    # TODO: sum(pod_cpu_usage_seconds_total) by (node)
-    value: '10.0 * node.SinceSecond()'
-
-  # Memory
-  - name: container_memory_working_set_bytes
-    dimension: container
-    help: "[ALPHA] Current working set of the container in bytes"
-    kind: gauge
-    labels:
-    - name: container
-      value: 'container.name'
-    - name: namespace
-      value: 'pod.metadata.namespace'
-    - name: pod
-      value: 'pod.metadata.name'
-    value: '1024.0 * 1024.0 * 1024.0'
-  - name: pod_memory_working_set_bytes
-    dimension: pod
-    help: "[ALPHA] Current working set of the pod in bytes"
-    kind: gauge
-    labels:
-    - name: namespace
-      value: 'pod.metadata.namespace'
-    - name: pod
-      value: 'pod.metadata.name'
-    # TODO: sum(container_memory_working_set_bytes) by (pod)
-    value: '10.0 * 1024.0 * 1024.0 * 1024.0'
-  - name: node_memory_working_set_bytes
-    dimension: node
-    help: "[ALPHA] Current working set of the node in bytes"
-    kind: gauge
-    # TODO: sum(pod_memory_working_set_bytes) by (node)
-    value: '100.0 * 1024.0 * 1024.0 * 1024.0'
diff --git a/test/kwokctl/testdata/binary/create_cluster_with_verbosity.txt b/test/kwokctl/testdata/binary/create_cluster_with_verbosity.txt
index cd841e79..8983d8d5 100644
--- a/test/kwokctl/testdata/binary/create_cluster_with_verbosity.txt
+++ b/test/kwokctl/testdata/binary/create_cluster_with_verbosity.txt
@@ -159,7 +159,7 @@ cd <ROOT_DIR>/workdir/clusters/<CLUSTER_NAME> && kube-scheduler --config=<ROOT_D
 echo $! ><ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pids/kube-scheduler.pid
 cd <ROOT_DIR>/workdir/clusters/<CLUSTER_NAME> && kwok-controller --manage-all-nodes=true --kubeconfig=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig.yaml --config=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kwok.yaml --tls-cert-file=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt --tls-private-key-file=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key --node-ip= --node-name=localhost --node-port=32763 --server-address=0.0.0.0:32763 --node-lease-duration-seconds=40 --v=DEBUG ><ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/logs/kwok-controller.log 2>&1 &
 echo $! ><ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pids/kwok-controller.pid
-cd <ROOT_DIR>/workdir/clusters/<CLUSTER_NAME> && metrics-server --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --bind-address=0.0.0.0 --secure-port=32759 --kubeconfig=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig.yaml --authentication-kubeconfig=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig.yaml --authorization-kubeconfig=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig.yaml --tls-cert-file=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt --tls-private-key-file=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key --v=4 ><ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/logs/metrics-server.log 2>&1 &
+cd <ROOT_DIR>/workdir/clusters/<CLUSTER_NAME> && metrics-server --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --kubelet-insecure-tls --metric-resolution=15s --bind-address=0.0.0.0 --secure-port=32759 --kubeconfig=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig.yaml --authentication-kubeconfig=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig.yaml --authorization-kubeconfig=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig.yaml --tls-cert-file=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt --tls-private-key-file=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key --v=4 ><ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/logs/metrics-server.log 2>&1 &
 echo $! ><ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pids/metrics-server.pid
 cd <ROOT_DIR>/workdir/clusters/<CLUSTER_NAME> && prometheus --config.file=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/prometheus.yaml --web.listen-address=0.0.0.0:9090 --log.level=debug ><ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/logs/prometheus.log 2>&1 &
 echo $! ><ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pids/prometheus.pid
diff --git a/test/kwokctl/testdata/docker/create_cluster_with_verbosity.txt b/test/kwokctl/testdata/docker/create_cluster_with_verbosity.txt
index 412aad07..c83ddbb7 100644
--- a/test/kwokctl/testdata/docker/create_cluster_with_verbosity.txt
+++ b/test/kwokctl/testdata/docker/create_cluster_with_verbosity.txt
@@ -176,7 +176,7 @@ docker create --name=kwok-<CLUSTER_NAME>-kube-controller-manager --pull=never --
 docker create --name=kwok-<CLUSTER_NAME>-kube-scheduler --pull=never --entrypoint=kube-scheduler --network=kwok-<CLUSTER_NAME> --link=kwok-<CLUSTER_NAME>-kube-apiserver --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/scheduler.yaml:/etc/kubernetes/scheduler.yaml:ro registry.k8s.io/kube-scheduler:v1.28.0 --config=/etc/kubernetes/scheduler.yaml --authorization-always-allow-paths=/healthz,/readyz,/livez,/metrics --bind-address=0.0.0.0 --secure-port=10259 --kube-api-qps=5000 --kube-api-burst=10000 --v=4
 docker create --name=kwok-<CLUSTER_NAME>-kwok-controller --pull=never --entrypoint=kwok --network=kwok-<CLUSTER_NAME> --link=kwok-<CLUSTER_NAME>-kube-apiserver --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kwok.yaml:~/.kwok/kwok.yaml:ro localhost/kwok:test --manage-all-nodes=true --kubeconfig=~/.kube/config --config=~/.kwok/kwok.yaml --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --node-ip= --node-name=kwok-<CLUSTER_NAME>-kwok-controller --node-port=10247 --server-address=0.0.0.0:10247 --node-lease-duration-seconds=40 --v=DEBUG
 docker create --name=kwok-<CLUSTER_NAME>-dashboard --pull=never --network=kwok-<CLUSTER_NAME> --link=kwok-<CLUSTER_NAME>-kube-apiserver --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --publish=8000:8000/tcp --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro docker.io/kubernetesui/dashboard:v2.7.0 --insecure-bind-address=0.0.0.0 --bind-address=127.0.0.1 --port=0 --enable-insecure-login --enable-skip-login --disable-settings-authorizer --metrics-provider=none --system-banner=Welcome to kwok-<CLUSTER_NAME> --kubeconfig=~/.kube/config --insecure-port=8000
-docker create --name=kwok-<CLUSTER_NAME>-metrics-server --pull=never --entrypoint=/metrics-server --network=kwok-<CLUSTER_NAME> --user=root --link=kwok-<CLUSTER_NAME>-kwok-controller --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro registry.k8s.io/metrics-server/metrics-server:v0.7.0 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --bind-address=0.0.0.0 --secure-port=4443 --kubeconfig=~/.kube/config --authentication-kubeconfig=~/.kube/config --authorization-kubeconfig=~/.kube/config --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --v=4
+docker create --name=kwok-<CLUSTER_NAME>-metrics-server --pull=never --entrypoint=/metrics-server --network=kwok-<CLUSTER_NAME> --user=root --link=kwok-<CLUSTER_NAME>-kwok-controller --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro registry.k8s.io/metrics-server/metrics-server:v0.7.0 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --kubelet-insecure-tls --metric-resolution=15s --bind-address=0.0.0.0 --secure-port=4443 --kubeconfig=~/.kube/config --authentication-kubeconfig=~/.kube/config --authorization-kubeconfig=~/.kube/config --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --v=4
 docker create --name=kwok-<CLUSTER_NAME>-prometheus --pull=never --entrypoint=prometheus --network=kwok-<CLUSTER_NAME> --link=kwok-<CLUSTER_NAME>-etcd --link=kwok-<CLUSTER_NAME>-kube-apiserver --link=kwok-<CLUSTER_NAME>-kube-controller-manager --link=kwok-<CLUSTER_NAME>-kube-scheduler --link=kwok-<CLUSTER_NAME>-kwok-controller --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --publish=9090:9090/tcp --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro docker.io/prom/prometheus:v2.44.0 --config.file=/etc/prometheus/prometheus.yaml --web.listen-address=0.0.0.0:9090 --log.level=debug
 # Add context kwok-<CLUSTER_NAME> to ~/.kube/config
 docker start kwok-<CLUSTER_NAME>-etcd
diff --git a/test/kwokctl/testdata/kind-podman/create_cluster_with_verbosity.txt b/test/kwokctl/testdata/kind-podman/create_cluster_with_verbosity.txt
index 7c5a796e..cc3add41 100644
--- a/test/kwokctl/testdata/kind-podman/create_cluster_with_verbosity.txt
+++ b/test/kwokctl/testdata/kind-podman/create_cluster_with_verbosity.txt
@@ -264,6 +264,7 @@ spec:
   - args:
     - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
     - --kubelet-use-node-status-port
+    - --kubelet-insecure-tls
     - --metric-resolution=15s
     - --bind-address=0.0.0.0
     - --secure-port=4443
diff --git a/test/kwokctl/testdata/kind/create_cluster_with_verbosity.txt b/test/kwokctl/testdata/kind/create_cluster_with_verbosity.txt
index c8172a90..feb9da0b 100644
--- a/test/kwokctl/testdata/kind/create_cluster_with_verbosity.txt
+++ b/test/kwokctl/testdata/kind/create_cluster_with_verbosity.txt
@@ -264,6 +264,7 @@ spec:
   - args:
     - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
     - --kubelet-use-node-status-port
+    - --kubelet-insecure-tls
     - --metric-resolution=15s
     - --bind-address=0.0.0.0
     - --secure-port=4443
diff --git a/test/kwokctl/testdata/nerdctl/create_cluster_with_verbosity.txt b/test/kwokctl/testdata/nerdctl/create_cluster_with_verbosity.txt
index da8d22fc..7baee37f 100644
--- a/test/kwokctl/testdata/nerdctl/create_cluster_with_verbosity.txt
+++ b/test/kwokctl/testdata/nerdctl/create_cluster_with_verbosity.txt
@@ -176,7 +176,7 @@ nerdctl create --name=kwok-<CLUSTER_NAME>-kube-controller-manager --pull=never -
 nerdctl create --name=kwok-<CLUSTER_NAME>-kube-scheduler --pull=never --entrypoint=kube-scheduler --network=kwok-<CLUSTER_NAME> --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/scheduler.yaml:/etc/kubernetes/scheduler.yaml:ro registry.k8s.io/kube-scheduler:v1.28.0 --config=/etc/kubernetes/scheduler.yaml --authorization-always-allow-paths=/healthz,/readyz,/livez,/metrics --bind-address=0.0.0.0 --secure-port=10259 --kube-api-qps=5000 --kube-api-burst=10000 --v=4
 nerdctl create --name=kwok-<CLUSTER_NAME>-kwok-controller --pull=never --entrypoint=kwok --network=kwok-<CLUSTER_NAME> --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kwok.yaml:~/.kwok/kwok.yaml:ro localhost/kwok:test --manage-all-nodes=true --kubeconfig=~/.kube/config --config=~/.kwok/kwok.yaml --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --node-ip= --node-name=kwok-<CLUSTER_NAME>-kwok-controller --node-port=10247 --server-address=0.0.0.0:10247 --node-lease-duration-seconds=40 --v=DEBUG
 nerdctl create --name=kwok-<CLUSTER_NAME>-dashboard --pull=never --network=kwok-<CLUSTER_NAME> --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --publish=8000:8000/tcp --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro docker.io/kubernetesui/dashboard:v2.7.0 --insecure-bind-address=0.0.0.0 --bind-address=127.0.0.1 --port=0 --enable-insecure-login --enable-skip-login --disable-settings-authorizer --metrics-provider=none --system-banner=Welcome to kwok-<CLUSTER_NAME> --kubeconfig=~/.kube/config --insecure-port=8000
-nerdctl create --name=kwok-<CLUSTER_NAME>-metrics-server --pull=never --entrypoint=/metrics-server --network=kwok-<CLUSTER_NAME> --user=root --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro registry.k8s.io/metrics-server/metrics-server:v0.7.0 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --bind-address=0.0.0.0 --secure-port=4443 --kubeconfig=~/.kube/config --authentication-kubeconfig=~/.kube/config --authorization-kubeconfig=~/.kube/config --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --v=4
+nerdctl create --name=kwok-<CLUSTER_NAME>-metrics-server --pull=never --entrypoint=/metrics-server --network=kwok-<CLUSTER_NAME> --user=root --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro registry.k8s.io/metrics-server/metrics-server:v0.7.0 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --kubelet-insecure-tls --metric-resolution=15s --bind-address=0.0.0.0 --secure-port=4443 --kubeconfig=~/.kube/config --authentication-kubeconfig=~/.kube/config --authorization-kubeconfig=~/.kube/config --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --v=4
 nerdctl create --name=kwok-<CLUSTER_NAME>-prometheus --pull=never --entrypoint=prometheus --network=kwok-<CLUSTER_NAME> --restart=unless-stopped --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --publish=9090:9090/tcp --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro docker.io/prom/prometheus:v2.44.0 --config.file=/etc/prometheus/prometheus.yaml --web.listen-address=0.0.0.0:9090 --log.level=debug
 # Add context kwok-<CLUSTER_NAME> to ~/.kube/config
 nerdctl start kwok-<CLUSTER_NAME>-etcd
diff --git a/test/kwokctl/testdata/podman/create_cluster_with_verbosity.txt b/test/kwokctl/testdata/podman/create_cluster_with_verbosity.txt
index e97e457c..d207728a 100644
--- a/test/kwokctl/testdata/podman/create_cluster_with_verbosity.txt
+++ b/test/kwokctl/testdata/podman/create_cluster_with_verbosity.txt
@@ -176,7 +176,7 @@ podman create --name=kwok-<CLUSTER_NAME>-kube-controller-manager --pull=never --
 podman create --name=kwok-<CLUSTER_NAME>-kube-scheduler --pull=never --entrypoint=kube-scheduler --network=kwok-<CLUSTER_NAME> --requires=kwok-<CLUSTER_NAME>-kube-apiserver --restart=unless-stopped --label=io.podman.compose.project=kwok-<CLUSTER_NAME> --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/scheduler.yaml:/etc/kubernetes/scheduler.yaml:ro registry.k8s.io/kube-scheduler:v1.28.0 --config=/etc/kubernetes/scheduler.yaml --authorization-always-allow-paths=/healthz,/readyz,/livez,/metrics --bind-address=0.0.0.0 --secure-port=10259 --kube-api-qps=5000 --kube-api-burst=10000 --v=4
 podman create --name=kwok-<CLUSTER_NAME>-kwok-controller --pull=never --entrypoint=kwok --network=kwok-<CLUSTER_NAME> --requires=kwok-<CLUSTER_NAME>-kube-apiserver --restart=unless-stopped --label=io.podman.compose.project=kwok-<CLUSTER_NAME> --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kwok.yaml:~/.kwok/kwok.yaml:ro localhost/kwok:test --manage-all-nodes=true --kubeconfig=~/.kube/config --config=~/.kwok/kwok.yaml --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --node-ip= --node-name=kwok-<CLUSTER_NAME>-kwok-controller --node-port=10247 --server-address=0.0.0.0:10247 --node-lease-duration-seconds=40 --v=DEBUG
 podman create --name=kwok-<CLUSTER_NAME>-dashboard --pull=never --network=kwok-<CLUSTER_NAME> --requires=kwok-<CLUSTER_NAME>-kube-apiserver --restart=unless-stopped --label=io.podman.compose.project=kwok-<CLUSTER_NAME> --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --publish=8000:8000/tcp --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro docker.io/kubernetesui/dashboard:v2.7.0 --insecure-bind-address=0.0.0.0 --bind-address=127.0.0.1 --port=0 --enable-insecure-login --enable-skip-login --disable-settings-authorizer --metrics-provider=none --system-banner=Welcome to kwok-<CLUSTER_NAME> --kubeconfig=~/.kube/config --insecure-port=8000
-podman create --name=kwok-<CLUSTER_NAME>-metrics-server --pull=never --entrypoint=/metrics-server --network=kwok-<CLUSTER_NAME> --user=root --requires=kwok-<CLUSTER_NAME>-kwok-controller --restart=unless-stopped --label=io.podman.compose.project=kwok-<CLUSTER_NAME> --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro registry.k8s.io/metrics-server/metrics-server:v0.7.0 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --bind-address=0.0.0.0 --secure-port=4443 --kubeconfig=~/.kube/config --authentication-kubeconfig=~/.kube/config --authorization-kubeconfig=~/.kube/config --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --v=4
+podman create --name=kwok-<CLUSTER_NAME>-metrics-server --pull=never --entrypoint=/metrics-server --network=kwok-<CLUSTER_NAME> --user=root --requires=kwok-<CLUSTER_NAME>-kwok-controller --restart=unless-stopped --label=io.podman.compose.project=kwok-<CLUSTER_NAME> --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/kubeconfig:~/.kube/config:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/ca.crt:/etc/kubernetes/pki/ca.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro registry.k8s.io/metrics-server/metrics-server:v0.7.0 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --kubelet-insecure-tls --metric-resolution=15s --bind-address=0.0.0.0 --secure-port=4443 --kubeconfig=~/.kube/config --authentication-kubeconfig=~/.kube/config --authorization-kubeconfig=~/.kube/config --tls-cert-file=/etc/kubernetes/pki/admin.crt --tls-private-key-file=/etc/kubernetes/pki/admin.key --v=4
 podman create --name=kwok-<CLUSTER_NAME>-prometheus --pull=never --entrypoint=prometheus --network=kwok-<CLUSTER_NAME> --requires=kwok-<CLUSTER_NAME>-etcd --requires=kwok-<CLUSTER_NAME>-kube-apiserver --requires=kwok-<CLUSTER_NAME>-kube-controller-manager --requires=kwok-<CLUSTER_NAME>-kube-scheduler --requires=kwok-<CLUSTER_NAME>-kwok-controller --restart=unless-stopped --label=io.podman.compose.project=kwok-<CLUSTER_NAME> --label=com.docker.compose.project=kwok-<CLUSTER_NAME> --publish=9090:9090/tcp --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.crt:/etc/kubernetes/pki/admin.crt:ro --volume=<ROOT_DIR>/workdir/clusters/<CLUSTER_NAME>/pki/admin.key:/etc/kubernetes/pki/admin.key:ro docker.io/prom/prometheus:v2.44.0 --config.file=/etc/prometheus/prometheus.yaml --web.listen-address=0.0.0.0:9090 --log.level=debug
 # Add context kwok-<CLUSTER_NAME> to ~/.kube/config
 podman start kwok-<CLUSTER_NAME>-etcd