Repository: kubernetes-sigs/kwok
Pull request #555 (2023-04-28 11:39:39)

[kwok] Provide generic functions that enable it to simulate data for any indicator (Node)
#### What type of PR is this?

/kind feature

#### What this PR does / why we need it:

#### Which issue(s) this PR fixes:

Related to #199

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kwok/pull/555#" title="Author self-approved">nikola-jokic</a>*
**Once this PR has been reviewed and has the lgtm label**, please assign yibozhuang for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkwok).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/kwok/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["yibozhuang"]} -->
k8s-ci-robot: Hi @nikola-jokic. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkwok).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *k8s-kwok* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | eacaf0a2acb0ee8e24770ddbea0bb66e64162fb4 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/k8s-kwok/deploys/644bb07dbd63bb00075f695c |
nikola-jokic: This is currently just a proposed way of exposing metrics from the controller, to make sure package structure is okay.
Looking forward to your suggestions @wzshiming :smile: 
wzshiming: The package structure is good to me,
I would like to minimise hard-coded metrics and use configuration whenever possible 😄
wzshiming: /ok-to-test
k8s-ci-robot: @nikola-jokic: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | c785b9121655f26518a0472206a8400a5acaf67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1654068408943120384) | true | `/test pull-kwok-verify-main`
pull-kwok-unit-test-main | c785b9121655f26518a0472206a8400a5acaf67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-unit-test-main/1654068409047977984) | true | `/test pull-kwok-unit-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @nikola-jokic: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | c785b9121655f26518a0472206a8400a5acaf67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1654068408943120384) | true | `/test pull-kwok-verify-main`
pull-kwok-unit-test-main | c785b9121655f26518a0472206a8400a5acaf67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-unit-test-main/1654068409047977984) | true | `/test pull-kwok-unit-test-main`
pull-kwok-e2e-test-main | c785b9121655f26518a0472206a8400a5acaf67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-e2e-test-main/1654068409098309632) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
nikola-jokic: `startedContainersTotal` is kind of tricky. The fastest approach is to do it on the Pod Controller. Basically, this function would evaluate to true if `node.metadata.name` is pod's `NodeName`, and then increase counter on `true`.
Is there a better way I'm not seeing? 
k8s-ci-robot: @nikola-jokic: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-e2e-test-main | c785b9121655f26518a0472206a8400a5acaf67e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-e2e-test-main/1654068409098309632) | true | `/test pull-kwok-e2e-test-main`
pull-kwok-verify-main | 0631e9960de9cc42a4f582141095326efbef66ee | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1660603590621794304) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @nikola-jokic: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | 40eb730bfb6e5fc870aeed88bc1718a2fa640178 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1667160352816631808) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
nikola-jokic: /retest
k8s-ci-robot: @nikola-jokic: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | 40eb730bfb6e5fc870aeed88bc1718a2fa640178 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1667161082898157568) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
nikola-jokic: /retest
nikola-jokic: /retest
k8s-ci-robot: @nikola-jokic: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | 5ef07f739aff7b0e8aa847fa6fca7587ed989063 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-unit-test-main/1668245314122813440) | true | `/test pull-kwok-unit-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @nikola-jokic: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | 5ef07f739aff7b0e8aa847fa6fca7587ed989063 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-unit-test-main/1668245314122813440) | true | `/test pull-kwok-unit-test-main`
pull-kwok-e2e-test-main | 5ef07f739aff7b0e8aa847fa6fca7587ed989063 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-e2e-test-main/1668245314168950784) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @nikola-jokic: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | 5ef07f739aff7b0e8aa847fa6fca7587ed989063 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-unit-test-main/1668245314122813440) | true | `/test pull-kwok-unit-test-main`
pull-kwok-e2e-test-main | 5ef07f739aff7b0e8aa847fa6fca7587ed989063 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-e2e-test-main/1668245314168950784) | true | `/test pull-kwok-e2e-test-main`
pull-kwok-verify-main | 5ef07f739aff7b0e8aa847fa6fca7587ed989063 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1668245314030538752) | true | `/test pull-kwok-verify-main`
pull-kwok-build-main | 5ef07f739aff7b0e8aa847fa6fca7587ed989063 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-build-main/1668245314076676096) | true | `/test pull-kwok-build-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
wzshiming: BTY, commits are too much, squash it 😄
k8s-ci-robot: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @nikola-jokic: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1668820063223812096) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @nikola-jokic: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1668820063223812096) | true | `/test pull-kwok-verify-main`
pull-kwok-e2e-test-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-e2e-test-main/1668820063379001344) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @nikola-jokic: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1668820063223812096) | true | `/test pull-kwok-verify-main`
pull-kwok-e2e-test-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-e2e-test-main/1668820063379001344) | true | `/test pull-kwok-e2e-test-main`
pull-kwok-build-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-build-main/1668820063274143744) | true | `/test pull-kwok-build-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @nikola-jokic: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-verify-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-verify-main/1668820063223812096) | true | `/test pull-kwok-verify-main`
pull-kwok-e2e-test-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-e2e-test-main/1668820063379001344) | true | `/test pull-kwok-e2e-test-main`
pull-kwok-build-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-build-main/1668820063274143744) | true | `/test pull-kwok-build-main`
pull-kwok-unit-test-main | ac3cefd696ed994ab96bb295f9cb1c28b580efcc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/555/pull-kwok-unit-test-main/1668820063332864000) | true | `/test pull-kwok-unit-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=555). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anikola-jokic). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/go.mod b/go.mod
index 80c976cb..35b0dd1c 100644
--- a/go.mod
+++ b/go.mod
@@ -12,15 +12,18 @@ require (
 	github.com/creack/pty v1.1.18
 	github.com/emicklei/go-restful/v3 v3.10.2
 	github.com/fsnotify/fsnotify v1.6.0
+	github.com/google/cel-go v0.15.2
 	github.com/google/go-cmp v0.5.9
 	github.com/itchyny/gojq v0.12.12
 	github.com/nxadm/tail v1.4.8
 	github.com/prometheus/client_golang v1.15.0
+	github.com/prometheus/client_model v0.3.0
 	github.com/spf13/cobra v1.7.0
 	github.com/spf13/pflag v1.0.5
 	github.com/wzshiming/cmux v0.3.2
 	github.com/wzshiming/cron v0.2.1
 	github.com/wzshiming/ctc v1.2.3
+	github.com/wzshiming/easycel v0.4.0
 	golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1
 	golang.org/x/sync v0.2.0
 	golang.org/x/sys v0.8.0
@@ -38,6 +41,7 @@ require (
 )
 
 require (
+	github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230512164433-5d1fd1a340c9 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/containernetworking/cni v1.1.2 // indirect
@@ -72,22 +76,22 @@ require (
 	github.com/opencontainers/go-digest v1.0.0 // indirect
 	github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e // indirect
 	github.com/pkg/errors v0.9.1 // indirect
-	github.com/prometheus/client_model v0.3.0 // indirect
 	github.com/prometheus/common v0.42.0 // indirect
 	github.com/prometheus/procfs v0.9.0 // indirect
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
+	github.com/stoewer/go-strcase v1.3.0 // indirect
 	github.com/wzshiming/llrb v0.2.1 // indirect
 	github.com/wzshiming/trie v0.1.1 // indirect
 	github.com/wzshiming/winseq v0.0.0-20200112104235-db357dc107ae // indirect
 	golang.org/x/mod v0.9.0 // indirect
 	golang.org/x/net v0.8.0 // indirect
 	golang.org/x/oauth2 v0.5.0 // indirect
-	golang.org/x/text v0.8.0 // indirect
+	golang.org/x/text v0.9.0 // indirect
 	golang.org/x/time v0.2.0 // indirect
 	golang.org/x/tools v0.7.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
-	google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21 // indirect
-	google.golang.org/grpc v1.51.0 // indirect
+	google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
+	google.golang.org/grpc v1.54.0 // indirect
 	google.golang.org/protobuf v1.30.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
diff --git a/go.sum b/go.sum
index eac61773..27d90705 100644
--- a/go.sum
+++ b/go.sum
@@ -1,16 +1,15 @@
 cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
-cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/ahmetb/gen-crd-api-reference-docs v0.3.0 h1:+XfOU14S4bGuwyvCijJwhhBIjYN+YXS18jrCY2EzJaY=
 github.com/ahmetb/gen-crd-api-reference-docs v0.3.0/go.mod h1:TdjdkYhlOifCQWPs1UdTma97kQQMozf5h26hTuG70u8=
-github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
+github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230512164433-5d1fd1a340c9 h1:goHVqTbFX3AIo0tzGr14pgfAW2ZfPChKO21Z9MGf/gk=
+github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230512164433-5d1fd1a340c9/go.mod h1:pSwJ0fSY5KhvocuWSx4fz3BA8OrA1bQn+K1Eli3BRwM=
 github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
 github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
-github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
 github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
@@ -18,12 +17,6 @@ github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5P
 github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
 github.com/client9/misspell v0.3.4 h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
-github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
-github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
-github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
-github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
-github.com/cncf/xds/go v0.0.0-20211001041855-01bcc9b48dfe/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
-github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
 github.com/compose-spec/compose-go v1.8.2 h1:sUQvDxnPgpcOyoxC/lz7mFTrTlHeZ6LWyuASYetkOqw=
 github.com/compose-spec/compose-go v1.8.2/go.mod h1:Tb5Ae2PsYN3GTqYqzl2IRbTPiJtPZZjMw8UKUvmehFk=
 github.com/containerd/go-cni v1.1.9 h1:ORi7P1dYzCwVM6XPN4n3CbkuOx/NZ2DOqy+SHRdo9rU=
@@ -47,11 +40,7 @@ github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5Xh
 github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
 github.com/emicklei/go-restful/v3 v3.10.2 h1:hIovbnmBTLjHXkqEBUz3HGpXZdM7ZrE9fJIZIqlJLqE=
 github.com/emicklei/go-restful/v3 v3.10.2/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
-github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
 github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
-github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
-github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
-github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1/go.mod h1:KJwIaB5Mv44NWtYuAOFCVOjcI94vtpEz2JU/D2v6IjE=
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84=
 github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
@@ -60,7 +49,6 @@ github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMo
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
 github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
-github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU=
 github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0=
@@ -82,7 +70,6 @@ github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfb
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
 github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
 github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
 github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
@@ -91,20 +78,19 @@ github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:W
 github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
 github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
 github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
-github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
 github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
 github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
 github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
 github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
+github.com/google/cel-go v0.15.2 h1:jX8b/sSNoE1JBaxLAVDb+rTuEk9F7+Yd8UVKWjoz7SU=
+github.com/google/cel-go v0.15.2/go.mod h1:YzWEoI07MC/a/wj9in8GeVatqfypkldgBlwXh9bCwqY=
 github.com/google/gnostic v0.5.7-v3refs h1:FhTMOKj2VhjpouxvWJAV1TL304uMlb9zcDqkl6cEI54=
 github.com/google/gnostic v0.5.7-v3refs/go.mod h1:73MKFl6jIHelAJNaBGFzt3SPtZULs9dYrGFt8OiIsHQ=
 github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
 github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
-github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
-github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
 github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
@@ -114,11 +100,9 @@ github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLe
 github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 h1:K6RDEckDVWvDI9JAJYCmNdQXq6neHJOYx3V6jnqNEec=
 github.com/google/renameio/v2 v2.0.0 h1:UifI23ZTGY8Tt29JbYFiuyIU3eX+RNFtUwefq9qAhxg=
 github.com/google/renameio/v2 v2.0.0/go.mod h1:BtmJXm5YlszgC+TD4HOEEUFgkJP3nLxehU6hfe7jRt4=
-github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
 github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
-github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
 github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk=
@@ -188,7 +172,6 @@ github.com/prometheus/common v0.42.0 h1:EKsfXEYo4JpWMHH5cg+KOUWeuJSov1Id8zGR8eeI
 github.com/prometheus/common v0.42.0/go.mod h1:xBwqVerjNdUDjgODMpudtOMwlOwf2SaTr1yjz4b7Zbc=
 github.com/prometheus/procfs v0.9.0 h1:wzCHvIvM5SxWqYvwgVL7yJY8Lz3PKn49KQtpgMYJfhI=
 github.com/prometheus/procfs v0.9.0/go.mod h1:+pB4zwohETzFnmlpe6yd2lSc+0/46IYZRB/chUwxUZY=
-github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
 github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
 github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
@@ -199,13 +182,14 @@ github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRM
 github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8=
+github.com/stoewer/go-strcase v1.3.0 h1:g0eASXYtp+yvN9fK8sH94oCIk0fau9uV1/ZdJ0AVEzs=
+github.com/stoewer/go-strcase v1.3.0/go.mod h1:fAH5hQ5pehh+j3nZfvwdk2RgEgQjAoM8wodgtPmh1xo=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
-github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
 github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
@@ -216,6 +200,8 @@ github.com/wzshiming/cron v0.2.1 h1:jtJlyGG4Od+xE0A36DtqTeyLhGssE95dsghqLdemXxI=
 github.com/wzshiming/cron v0.2.1/go.mod h1:qXPkSl0AQLmjchtcqv7L6X8FSOLqy8hXYIXv9J1cKO4=
 github.com/wzshiming/ctc v1.2.3 h1:q+hW3IQNsjIlOFBTGZZZeIXTElFM4grF4spW/errh/c=
 github.com/wzshiming/ctc v1.2.3/go.mod h1:2tVAtIY7SUyraSk0JxvwmONNPFL4ARavPuEsg5+KA28=
+github.com/wzshiming/easycel v0.4.0 h1:flPpJYS2bXQj1Jgk2yOp5oB6FvflvPM0VCHXx/qOJDk=
+github.com/wzshiming/easycel v0.4.0/go.mod h1:qg3oAkmPOLJEUFlFsxBxYbp0cXHgbq2sZEHVq7SmUp8=
 github.com/wzshiming/llrb v0.2.1 h1:+T/PE0lrrB2LE8487fYcRlkZct5E9L/Gi8mf6mUtEK8=
 github.com/wzshiming/llrb v0.2.1/go.mod h1:xlAM5hpCBIT3dTy/6CbNPxHMR1JfIMSyk2lV68wz3Yw=
 github.com/wzshiming/trie v0.1.1 h1:02AaBSZGhs6Aqljp8fz4xq/Mg8omFBPIlrUS0pJ11ks=
@@ -224,7 +210,6 @@ github.com/wzshiming/winseq v0.0.0-20200112104235-db357dc107ae h1:tpXvBXC3hpQBDC
 github.com/wzshiming/winseq v0.0.0-20200112104235-db357dc107ae/go.mod h1:VTAq37rkGeV+WOybvZwjXiJOicICdpLCN8ifpISjK20=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
-go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
@@ -241,7 +226,6 @@ golang.org/x/mod v0.9.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@@ -249,14 +233,11 @@ golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
-golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
 golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
 golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ=
 golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
-golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.5.0 h1:HuArIo48skDwlrvM3sEdHXElYslAMsf3KwRkkW4MC4s=
 golang.org/x/oauth2 v0.5.0/go.mod h1:9/XBHVqLaWO3/BRHs5jbpYCnOZVjj5V0ndyaAM7KB4I=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -280,10 +261,7 @@ golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
 golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -293,10 +271,9 @@ golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=
-golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
+golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE=
+golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
 golang.org/x/time v0.2.0 h1:52I/1L54xyEQAYdtcSuxtiT84KGYTBGXwayxmIpNJhE=
 golang.org/x/time v0.2.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -321,20 +298,15 @@ google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6
 google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
 google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
 google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
-google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
 google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
 google.golang.org/genproto v0.0.0-20201019141844-1ed22bb0c154/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
-google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21 h1:hrbNEivu7Zn1pxvHk6MBrq9iE22woVILTHqexqBxe6I=
-google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4=
+google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 h1:KpwkzHKEF7B9Zxg18WzOa7djJ+Ha5DzthMyZYQfEn2A=
+google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1/go.mod h1:nKE/iIaLqn2bQwXBg8f1g2Ylh6r5MN5CmZvuzZCgsCU=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
-google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
 google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
-google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
-google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
-google.golang.org/grpc v1.46.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk=
-google.golang.org/grpc v1.51.0 h1:E1eGv1FTqoLIdnBCZufiSHgKjlqG6fKFf6pPWtMTh8U=
-google.golang.org/grpc v1.51.0/go.mod h1:wgNDFcnuBGmxLKI/qn4T+m5BtEBYXJPvibbUPsAIPww=
+google.golang.org/grpc v1.54.0 h1:EhTqbhiYeixwWQtAEZAxmV9MGqcjEU2mFx52xCzNyag=
+google.golang.org/grpc v1.54.0/go.mod h1:PUSEXI6iWghWaB6lXM4knEgpJNu2qUcKfDtNci3EC2g=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -344,11 +316,8 @@ google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2
 google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
 google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
 google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
-google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
 google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
-google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
-google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
 google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
 google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -361,7 +330,6 @@ gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
diff --git a/pkg/apis/internalversion/conversion.go b/pkg/apis/internalversion/conversion.go
index f61dc412..21b85771 100644
--- a/pkg/apis/internalversion/conversion.go
+++ b/pkg/apis/internalversion/conversion.go
@@ -299,3 +299,25 @@ func ConvertToInternalAttach(in *v1alpha1.Attach) (*Attach, error) {
 	}
 	return &out, nil
 }
+
+// ConvertToV1Alpha1Metric converts an internal version Metric to a v1alpha1.Metric.
+func ConvertToV1Alpha1Metric(in *Metric) (*v1alpha1.Metric, error) {
+	var out v1alpha1.Metric
+	out.APIVersion = v1alpha1.GroupVersion.String()
+	out.Kind = v1alpha1.MetricKind
+	err := Convert_internalversion_Metric_To_v1alpha1_Metric(in, &out, nil)
+	if err != nil {
+		return nil, err
+	}
+	return &out, nil
+}
+
+// ConvertToInternalMetric converts a v1alpha1.Metric to an internal version.
+func ConvertToInternalMetric(in *v1alpha1.Metric) (*Metric, error) {
+	var out Metric
+	err := Convert_v1alpha1_Metric_To_internalversion_Metric(in, &out, nil)
+	if err != nil {
+		return nil, err
+	}
+	return &out, nil
+}
diff --git a/pkg/apis/internalversion/metric_type.go b/pkg/apis/internalversion/metric_type.go
new file mode 100644
index 00000000..2c6ac030
--- /dev/null
+++ b/pkg/apis/internalversion/metric_type.go
@@ -0,0 +1,73 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package internalversion
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// Metric provides metrics configuration.
+type Metric struct {
+	// Standard list metadata.
+	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
+	metav1.ObjectMeta
+	// Spec holds spec for metrics.
+	Spec MetricSpec
+}
+
+// MetricSpec holds spec for metrics.
+type MetricSpec struct {
+	// Path is a restful service path.
+	Path string
+	// Metrics is a list of metric configurations.
+	Metrics []MetricConfig
+}
+
+// MetricConfig provides metric configuration to a single metric
+type MetricConfig struct {
+	// Name is the fully-qualified name of the metric.
+	Name string
+	// Help provides information about this metric.
+	Help string
+	// Kind is kind of metric (ex. counter, gauge, histogram).
+	Kind string
+	// Labels are metric labels.
+	Labels []MetricLabel
+	// Value is a CEL expression.
+	Value string
+	// Buckets is a list of buckets for a histogram metric.
+	Buckets []MetricBucket
+}
+
+// MetricLabel holds label name and the value of the label.
+type MetricLabel struct {
+	// Name is a label name.
+	Name string
+	// Value is a CEL expression.
+	Value string
+}
+
+// MetricBucket is a single bucket for a metric.
+type MetricBucket struct {
+	// Le is less-than or equal.
+	Le float64
+	// Value is a CEL expression.
+	Value string
+	// Hidden is means that this bucket not shown in the metric.
+	// but value will be calculated and cumulative into the next bucket.
+	Hidden bool
+}
diff --git a/pkg/apis/internalversion/zz_generated.conversion.go b/pkg/apis/internalversion/zz_generated.conversion.go
index 355dac16..f0fcd24f 100644
--- a/pkg/apis/internalversion/zz_generated.conversion.go
+++ b/pkg/apis/internalversion/zz_generated.conversion.go
@@ -348,6 +348,56 @@ func RegisterConversions(s *runtime.Scheme) error {
 	}); err != nil {
 		return err
 	}
+	if err := s.AddGeneratedConversionFunc((*Metric)(nil), (*v1alpha1.Metric)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_Metric_To_v1alpha1_Metric(a.(*Metric), b.(*v1alpha1.Metric), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.Metric)(nil), (*Metric)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_Metric_To_internalversion_Metric(a.(*v1alpha1.Metric), b.(*Metric), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*MetricBucket)(nil), (*v1alpha1.MetricBucket)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_MetricBucket_To_v1alpha1_MetricBucket(a.(*MetricBucket), b.(*v1alpha1.MetricBucket), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.MetricBucket)(nil), (*MetricBucket)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_MetricBucket_To_internalversion_MetricBucket(a.(*v1alpha1.MetricBucket), b.(*MetricBucket), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*MetricConfig)(nil), (*v1alpha1.MetricConfig)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_MetricConfig_To_v1alpha1_MetricConfig(a.(*MetricConfig), b.(*v1alpha1.MetricConfig), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.MetricConfig)(nil), (*MetricConfig)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_MetricConfig_To_internalversion_MetricConfig(a.(*v1alpha1.MetricConfig), b.(*MetricConfig), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*MetricLabel)(nil), (*v1alpha1.MetricLabel)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_MetricLabel_To_v1alpha1_MetricLabel(a.(*MetricLabel), b.(*v1alpha1.MetricLabel), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.MetricLabel)(nil), (*MetricLabel)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_MetricLabel_To_internalversion_MetricLabel(a.(*v1alpha1.MetricLabel), b.(*MetricLabel), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*MetricSpec)(nil), (*v1alpha1.MetricSpec)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_internalversion_MetricSpec_To_v1alpha1_MetricSpec(a.(*MetricSpec), b.(*v1alpha1.MetricSpec), scope)
+	}); err != nil {
+		return err
+	}
+	if err := s.AddGeneratedConversionFunc((*v1alpha1.MetricSpec)(nil), (*MetricSpec)(nil), func(a, b interface{}, scope conversion.Scope) error {
+		return Convert_v1alpha1_MetricSpec_To_internalversion_MetricSpec(a.(*v1alpha1.MetricSpec), b.(*MetricSpec), scope)
+	}); err != nil {
+		return err
+	}
 	if err := s.AddGeneratedConversionFunc((*ObjectSelector)(nil), (*v1alpha1.ObjectSelector)(nil), func(a, b interface{}, scope conversion.Scope) error {
 		return Convert_internalversion_ObjectSelector_To_v1alpha1_ObjectSelector(a.(*ObjectSelector), b.(*v1alpha1.ObjectSelector), scope)
 	}); err != nil {
@@ -1507,6 +1557,131 @@ func Convert_v1alpha1_LogsSpec_To_internalversion_LogsSpec(in *v1alpha1.LogsSpec
 	return autoConvert_v1alpha1_LogsSpec_To_internalversion_LogsSpec(in, out, s)
 }
 
+func autoConvert_internalversion_Metric_To_v1alpha1_Metric(in *Metric, out *v1alpha1.Metric, s conversion.Scope) error {
+	out.ObjectMeta = in.ObjectMeta
+	if err := Convert_internalversion_MetricSpec_To_v1alpha1_MetricSpec(&in.Spec, &out.Spec, s); err != nil {
+		return err
+	}
+	return nil
+}
+
+// Convert_internalversion_Metric_To_v1alpha1_Metric is an autogenerated conversion function.
+func Convert_internalversion_Metric_To_v1alpha1_Metric(in *Metric, out *v1alpha1.Metric, s conversion.Scope) error {
+	return autoConvert_internalversion_Metric_To_v1alpha1_Metric(in, out, s)
+}
+
+func autoConvert_v1alpha1_Metric_To_internalversion_Metric(in *v1alpha1.Metric, out *Metric, s conversion.Scope) error {
+	// INFO: in.TypeMeta opted out of conversion generation
+	out.ObjectMeta = in.ObjectMeta
+	if err := Convert_v1alpha1_MetricSpec_To_internalversion_MetricSpec(&in.Spec, &out.Spec, s); err != nil {
+		return err
+	}
+	return nil
+}
+
+// Convert_v1alpha1_Metric_To_internalversion_Metric is an autogenerated conversion function.
+func Convert_v1alpha1_Metric_To_internalversion_Metric(in *v1alpha1.Metric, out *Metric, s conversion.Scope) error {
+	return autoConvert_v1alpha1_Metric_To_internalversion_Metric(in, out, s)
+}
+
+func autoConvert_internalversion_MetricBucket_To_v1alpha1_MetricBucket(in *MetricBucket, out *v1alpha1.MetricBucket, s conversion.Scope) error {
+	out.Le = in.Le
+	out.Value = in.Value
+	out.Hidden = in.Hidden
+	return nil
+}
+
+// Convert_internalversion_MetricBucket_To_v1alpha1_MetricBucket is an autogenerated conversion function.
+func Convert_internalversion_MetricBucket_To_v1alpha1_MetricBucket(in *MetricBucket, out *v1alpha1.MetricBucket, s conversion.Scope) error {
+	return autoConvert_internalversion_MetricBucket_To_v1alpha1_MetricBucket(in, out, s)
+}
+
+func autoConvert_v1alpha1_MetricBucket_To_internalversion_MetricBucket(in *v1alpha1.MetricBucket, out *MetricBucket, s conversion.Scope) error {
+	out.Le = in.Le
+	out.Value = in.Value
+	out.Hidden = in.Hidden
+	return nil
+}
+
+// Convert_v1alpha1_MetricBucket_To_internalversion_MetricBucket is an autogenerated conversion function.
+func Convert_v1alpha1_MetricBucket_To_internalversion_MetricBucket(in *v1alpha1.MetricBucket, out *MetricBucket, s conversion.Scope) error {
+	return autoConvert_v1alpha1_MetricBucket_To_internalversion_MetricBucket(in, out, s)
+}
+
+func autoConvert_internalversion_MetricConfig_To_v1alpha1_MetricConfig(in *MetricConfig, out *v1alpha1.MetricConfig, s conversion.Scope) error {
+	out.Name = in.Name
+	out.Help = in.Help
+	out.Kind = in.Kind
+	out.Labels = *(*[]v1alpha1.MetricLabel)(unsafe.Pointer(&in.Labels))
+	out.Value = in.Value
+	out.Buckets = *(*[]v1alpha1.MetricBucket)(unsafe.Pointer(&in.Buckets))
+	return nil
+}
+
+// Convert_internalversion_MetricConfig_To_v1alpha1_MetricConfig is an autogenerated conversion function.
+func Convert_internalversion_MetricConfig_To_v1alpha1_MetricConfig(in *MetricConfig, out *v1alpha1.MetricConfig, s conversion.Scope) error {
+	return autoConvert_internalversion_MetricConfig_To_v1alpha1_MetricConfig(in, out, s)
+}
+
+func autoConvert_v1alpha1_MetricConfig_To_internalversion_MetricConfig(in *v1alpha1.MetricConfig, out *MetricConfig, s conversion.Scope) error {
+	out.Name = in.Name
+	out.Help = in.Help
+	out.Kind = in.Kind
+	out.Labels = *(*[]MetricLabel)(unsafe.Pointer(&in.Labels))
+	out.Value = in.Value
+	out.Buckets = *(*[]MetricBucket)(unsafe.Pointer(&in.Buckets))
+	return nil
+}
+
+// Convert_v1alpha1_MetricConfig_To_internalversion_MetricConfig is an autogenerated conversion function.
+func Convert_v1alpha1_MetricConfig_To_internalversion_MetricConfig(in *v1alpha1.MetricConfig, out *MetricConfig, s conversion.Scope) error {
+	return autoConvert_v1alpha1_MetricConfig_To_internalversion_MetricConfig(in, out, s)
+}
+
+func autoConvert_internalversion_MetricLabel_To_v1alpha1_MetricLabel(in *MetricLabel, out *v1alpha1.MetricLabel, s conversion.Scope) error {
+	out.Name = in.Name
+	out.Value = in.Value
+	return nil
+}
+
+// Convert_internalversion_MetricLabel_To_v1alpha1_MetricLabel is an autogenerated conversion function.
+func Convert_internalversion_MetricLabel_To_v1alpha1_MetricLabel(in *MetricLabel, out *v1alpha1.MetricLabel, s conversion.Scope) error {
+	return autoConvert_internalversion_MetricLabel_To_v1alpha1_MetricLabel(in, out, s)
+}
+
+func autoConvert_v1alpha1_MetricLabel_To_internalversion_MetricLabel(in *v1alpha1.MetricLabel, out *MetricLabel, s conversion.Scope) error {
+	out.Name = in.Name
+	out.Value = in.Value
+	return nil
+}
+
+// Convert_v1alpha1_MetricLabel_To_internalversion_MetricLabel is an autogenerated conversion function.
+func Convert_v1alpha1_MetricLabel_To_internalversion_MetricLabel(in *v1alpha1.MetricLabel, out *MetricLabel, s conversion.Scope) error {
+	return autoConvert_v1alpha1_MetricLabel_To_internalversion_MetricLabel(in, out, s)
+}
+
+func autoConvert_internalversion_MetricSpec_To_v1alpha1_MetricSpec(in *MetricSpec, out *v1alpha1.MetricSpec, s conversion.Scope) error {
+	out.Path = in.Path
+	out.Metrics = *(*[]v1alpha1.MetricConfig)(unsafe.Pointer(&in.Metrics))
+	return nil
+}
+
+// Convert_internalversion_MetricSpec_To_v1alpha1_MetricSpec is an autogenerated conversion function.
+func Convert_internalversion_MetricSpec_To_v1alpha1_MetricSpec(in *MetricSpec, out *v1alpha1.MetricSpec, s conversion.Scope) error {
+	return autoConvert_internalversion_MetricSpec_To_v1alpha1_MetricSpec(in, out, s)
+}
+
+func autoConvert_v1alpha1_MetricSpec_To_internalversion_MetricSpec(in *v1alpha1.MetricSpec, out *MetricSpec, s conversion.Scope) error {
+	out.Path = in.Path
+	out.Metrics = *(*[]MetricConfig)(unsafe.Pointer(&in.Metrics))
+	return nil
+}
+
+// Convert_v1alpha1_MetricSpec_To_internalversion_MetricSpec is an autogenerated conversion function.
+func Convert_v1alpha1_MetricSpec_To_internalversion_MetricSpec(in *v1alpha1.MetricSpec, out *MetricSpec, s conversion.Scope) error {
+	return autoConvert_v1alpha1_MetricSpec_To_internalversion_MetricSpec(in, out, s)
+}
+
 func autoConvert_internalversion_ObjectSelector_To_v1alpha1_ObjectSelector(in *ObjectSelector, out *v1alpha1.ObjectSelector, s conversion.Scope) error {
 	out.MatchNamespaces = *(*[]string)(unsafe.Pointer(&in.MatchNamespaces))
 	out.MatchNames = *(*[]string)(unsafe.Pointer(&in.MatchNames))
diff --git a/pkg/apis/internalversion/zz_generated.deepcopy.go b/pkg/apis/internalversion/zz_generated.deepcopy.go
index 6e0186aa..616642b2 100644
--- a/pkg/apis/internalversion/zz_generated.deepcopy.go
+++ b/pkg/apis/internalversion/zz_generated.deepcopy.go
@@ -708,6 +708,105 @@ func (in *LogsSpec) DeepCopy() *LogsSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Metric) DeepCopyInto(out *Metric) {
+	*out = *in
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Metric.
+func (in *Metric) DeepCopy() *Metric {
+	if in == nil {
+		return nil
+	}
+	out := new(Metric)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MetricBucket) DeepCopyInto(out *MetricBucket) {
+	*out = *in
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MetricBucket.
+func (in *MetricBucket) DeepCopy() *MetricBucket {
+	if in == nil {
+		return nil
+	}
+	out := new(MetricBucket)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MetricConfig) DeepCopyInto(out *MetricConfig) {
+	*out = *in
+	if in.Labels != nil {
+		in, out := &in.Labels, &out.Labels
+		*out = make([]MetricLabel, len(*in))
+		copy(*out, *in)
+	}
+	if in.Buckets != nil {
+		in, out := &in.Buckets, &out.Buckets
+		*out = make([]MetricBucket, len(*in))
+		copy(*out, *in)
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MetricConfig.
+func (in *MetricConfig) DeepCopy() *MetricConfig {
+	if in == nil {
+		return nil
+	}
+	out := new(MetricConfig)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MetricLabel) DeepCopyInto(out *MetricLabel) {
+	*out = *in
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MetricLabel.
+func (in *MetricLabel) DeepCopy() *MetricLabel {
+	if in == nil {
+		return nil
+	}
+	out := new(MetricLabel)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MetricSpec) DeepCopyInto(out *MetricSpec) {
+	*out = *in
+	if in.Metrics != nil {
+		in, out := &in.Metrics, &out.Metrics
+		*out = make([]MetricConfig, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MetricSpec.
+func (in *MetricSpec) DeepCopy() *MetricSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(MetricSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *ObjectSelector) DeepCopyInto(out *ObjectSelector) {
 	*out = *in
diff --git a/pkg/apis/v1alpha1/metric_types.go b/pkg/apis/v1alpha1/metric_types.go
new file mode 100644
index 00000000..d01ec743
--- /dev/null
+++ b/pkg/apis/v1alpha1/metric_types.go
@@ -0,0 +1,82 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const (
+	// MetricKind is the kind for metrics.
+	MetricKind = "Metric"
+)
+
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+
+// Metric provides metrics configuration.
+type Metric struct {
+	//+k8s:conversion-gen=false
+	metav1.TypeMeta `json:",inline"`
+	// Standard list metadata.
+	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
+	metav1.ObjectMeta `json:"metadata"`
+	// Spec holds spec for metrics.
+	Spec MetricSpec `json:"spec"`
+}
+
+// MetricSpec holds spec for metrics.
+type MetricSpec struct {
+	// Path is a restful service path.
+	Path string `json:"path"`
+	// Metrics is a list of metric configurations.
+	Metrics []MetricConfig `json:"metrics"`
+}
+
+// MetricConfig provides metric configuration to a single metric
+type MetricConfig struct {
+	// Name is the fully-qualified name of the metric.
+	Name string `json:"name"`
+	// Help provides information about this metric.
+	Help string `json:"help"`
+	// Kind is kind of metric (ex. counter, gauge, histogram).
+	Kind string `json:"kind"`
+	// Labels are metric labels.
+	Labels []MetricLabel `json:"labels"`
+	// Value is a CEL expression.
+	Value string `json:"value"`
+	// Buckets is a list of buckets for a histogram metric.
+	Buckets []MetricBucket `json:"buckets"`
+}
+
+// MetricLabel holds label name and the value of the label.
+type MetricLabel struct {
+	// Name is a label name.
+	Name string `json:"name"`
+	// Value is a CEL expression.
+	Value string `json:"value"`
+}
+
+// MetricBucket is a single bucket for a metric.
+type MetricBucket struct {
+	// Le is less-than or equal.
+	Le float64 `json:"le"`
+	// Value is a CEL expression.
+	Value string `json:"value"`
+	// Hidden is means that this bucket not shown in the metric.
+	// but value will be calculated and cumulative into the next bucket.
+	Hidden bool `json:"hidden"`
+}
diff --git a/pkg/apis/v1alpha1/zz_generated.deepcopy.go b/pkg/apis/v1alpha1/zz_generated.deepcopy.go
index 7056a88a..cb50fb7c 100644
--- a/pkg/apis/v1alpha1/zz_generated.deepcopy.go
+++ b/pkg/apis/v1alpha1/zz_generated.deepcopy.go
@@ -579,6 +579,114 @@ func (in *LogsSpec) DeepCopy() *LogsSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Metric) DeepCopyInto(out *Metric) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Metric.
+func (in *Metric) DeepCopy() *Metric {
+	if in == nil {
+		return nil
+	}
+	out := new(Metric)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *Metric) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MetricBucket) DeepCopyInto(out *MetricBucket) {
+	*out = *in
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MetricBucket.
+func (in *MetricBucket) DeepCopy() *MetricBucket {
+	if in == nil {
+		return nil
+	}
+	out := new(MetricBucket)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MetricConfig) DeepCopyInto(out *MetricConfig) {
+	*out = *in
+	if in.Labels != nil {
+		in, out := &in.Labels, &out.Labels
+		*out = make([]MetricLabel, len(*in))
+		copy(*out, *in)
+	}
+	if in.Buckets != nil {
+		in, out := &in.Buckets, &out.Buckets
+		*out = make([]MetricBucket, len(*in))
+		copy(*out, *in)
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MetricConfig.
+func (in *MetricConfig) DeepCopy() *MetricConfig {
+	if in == nil {
+		return nil
+	}
+	out := new(MetricConfig)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MetricLabel) DeepCopyInto(out *MetricLabel) {
+	*out = *in
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MetricLabel.
+func (in *MetricLabel) DeepCopy() *MetricLabel {
+	if in == nil {
+		return nil
+	}
+	out := new(MetricLabel)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MetricSpec) DeepCopyInto(out *MetricSpec) {
+	*out = *in
+	if in.Metrics != nil {
+		in, out := &in.Metrics, &out.Metrics
+		*out = make([]MetricConfig, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MetricSpec.
+func (in *MetricSpec) DeepCopy() *MetricSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(MetricSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *ObjectSelector) DeepCopyInto(out *ObjectSelector) {
 	*out = *in
diff --git a/pkg/config/config.go b/pkg/config/config.go
index f419087a..746bef95 100644
--- a/pkg/config/config.go
+++ b/pkg/config/config.go
@@ -144,6 +144,12 @@ var configHandlers = map[string]configHandler{
 		MutateToInternal: mutateToInternalConfig(internalversion.ConvertToInternalClusterAttach),
 		MutateToVersiond: mutateToVersiondConfig(internalversion.ConvertToV1Alpha1ClusterAttach),
 	},
+	v1alpha1.MetricKind: {
+		Unmarshal:        unmarshalConfig[*v1alpha1.Metric],
+		Marshal:          marshalConfig,
+		MutateToInternal: mutateToInternalConfig(internalversion.ConvertToInternalMetric),
+		MutateToVersiond: mutateToVersiondConfig(internalversion.ConvertToV1Alpha1Metric),
+	},
 }
 
 func unmarshalConfig[T versiondObject](raw []byte) (versiondObject, error) {
diff --git a/pkg/kwok/cmd/root.go b/pkg/kwok/cmd/root.go
index a6541781..14667a33 100644
--- a/pkg/kwok/cmd/root.go
+++ b/pkg/kwok/cmd/root.go
@@ -19,6 +19,7 @@ package cmd
 
 import (
 	"context"
+	"fmt"
 	"os"
 	"time"
 
@@ -169,6 +170,8 @@ func runE(ctx context.Context, flags *flagpole) error {
 	}
 	ctx = log.NewContext(ctx, logger.With("id", id))
 
+	metrics := config.FilterWithTypeFromContext[*internalversion.Metric](ctx)
+
 	ctr, err := controllers.NewController(controllers.Config{
 		Clock:                                 clock.RealClock{},
 		ClientSet:                             typedClient,
@@ -217,9 +220,13 @@ func runE(ctx context.Context, flags *flagpole) error {
 			Logs:                logs,
 			ClusterAttaches:     clusterAttaches,
 			Attaches:            attaches,
+			Metrics:             metrics,
+			Controller:          ctr,
 		}
 		svc := server.NewServer(config)
-		svc.InstallMetrics()
+		if err := svc.InstallMetrics(); err != nil {
+			return fmt.Errorf("failed to install metrics: %w", err)
+		}
 		svc.InstallHealthz()
 
 		if flags.Options.EnableDebuggingHandlers {
diff --git a/pkg/kwok/controllers/controller.go b/pkg/kwok/controllers/controller.go
index 9265fdd2..65aa9afa 100644
--- a/pkg/kwok/controllers/controller.go
+++ b/pkg/kwok/controllers/controller.go
@@ -98,6 +98,7 @@ type Config struct {
 	NodeLeaseDurationSeconds              uint
 	NodeLeaseParallelism                  uint
 	ID                                    string
+	Metrics                               []*internalversion.Metric
 }
 
 // NewController creates a new fake kubelet controller
@@ -194,6 +195,10 @@ func NewController(conf Config) (*Controller, error) {
 		return nil, fmt.Errorf("failed to create nodes controller: %w", err)
 	}
 
+	nodeHasMetric := func(nodeName string) bool {
+		return len(conf.Metrics) != 0
+	}
+
 	pods, err := NewPodController(PodControllerConfig{
 		Clock:                                 conf.Clock,
 		EnableCNI:                             conf.EnableCNI,
@@ -206,6 +211,7 @@ func NewController(conf Config) (*Controller, error) {
 		PlayStageParallelism:                  conf.PodPlayStageParallelism,
 		Namespace:                             corev1.NamespaceAll,
 		NodeGetFunc:                           nodes.Get,
+		NodeHasMetric:                         nodeHasMetric,
 		FuncMap:                               defaultFuncMap,
 		Recorder:                              recorder,
 		ReadOnlyFunc:                          readOnlyFunc,
@@ -270,6 +276,11 @@ func (c *Controller) Start(ctx context.Context) error {
 	return nil
 }
 
+// GetNode returns the node with the given name
+func (c *Controller) GetNode(nodeName string) (*NodeInfo, bool) {
+	return c.nodes.Get(nodeName)
+}
+
 // Identity returns a unique identifier for this controller
 func Identity() (string, error) {
 	hostname, err := os.Hostname()
diff --git a/pkg/kwok/controllers/node_controller.go b/pkg/kwok/controllers/node_controller.go
index 78196424..b23481d0 100644
--- a/pkg/kwok/controllers/node_controller.go
+++ b/pkg/kwok/controllers/node_controller.go
@@ -22,6 +22,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"net"
+	"sync/atomic"
 	"text/template"
 	"time"
 
@@ -132,10 +133,11 @@ type NodeControllerConfig struct {
 
 // NodeInfo is the collection of necessary node information
 type NodeInfo struct {
-	Node            *corev1.Node
-	HostIPs         []string
-	PodCIDRs        []string
-	OwnerReferences []metav1.OwnerReference
+	Node             *corev1.Node
+	HostIPs          []string
+	PodCIDRs         []string
+	OwnerReferences  []metav1.OwnerReference
+	StartedContainer atomic.Int64
 }
 
 // NewNodeController creates a new fake nodes controller
diff --git a/pkg/kwok/controllers/pod_controller.go b/pkg/kwok/controllers/pod_controller.go
index b147c95f..72844602 100644
--- a/pkg/kwok/controllers/pod_controller.go
+++ b/pkg/kwok/controllers/pod_controller.go
@@ -62,6 +62,7 @@ type PodController struct {
 	defaultCIDR                           string
 	namespace                             string
 	nodeGetFunc                           func(nodeName string) (*NodeInfo, bool)
+	nodeHasMetric                         func(nodeName string) bool
 	ipPools                               maps.SyncMap[string, *ipPool]
 	renderer                              *renderer
 	preprocessChan                        chan *corev1.Pod
@@ -86,6 +87,7 @@ type PodControllerConfig struct {
 	CIDR                                  string
 	Namespace                             string
 	NodeGetFunc                           func(nodeName string) (*NodeInfo, bool)
+	NodeHasMetric                         func(nodeName string) bool
 	Stages                                []*internalversion.Stage
 	PlayStageParallelism                  uint
 	FuncMap                               template.FuncMap
@@ -507,6 +509,15 @@ func (c *PodController) watchResources(ctx context.Context, opt metav1.ListOptio
 						)
 					}
 
+					if event.Type == watch.Added &&
+						c.nodeHasMetric != nil &&
+						c.nodeHasMetric(pod.Spec.NodeName) &&
+						c.nodeGetFunc != nil {
+						nodeInfo, ok := c.nodeGetFunc(pod.Spec.NodeName)
+						if ok {
+							nodeInfo.StartedContainer.Add(int64(len(pod.Spec.Containers)))
+						}
+					}
 				case watch.Deleted:
 					pod := event.Object.(*corev1.Pod)
 					if c.need(pod) {
diff --git a/pkg/kwok/metrics/cel/doc.go b/pkg/kwok/metrics/cel/doc.go
new file mode 100644
index 00000000..94939459
--- /dev/null
+++ b/pkg/kwok/metrics/cel/doc.go
@@ -0,0 +1,18 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Package cel implements a metrics evaluator using the Common Expression Language (CEL).
+package cel
diff --git a/pkg/kwok/metrics/cel/evaluate.go b/pkg/kwok/metrics/cel/evaluate.go
new file mode 100644
index 00000000..7967a5d2
--- /dev/null
+++ b/pkg/kwok/metrics/cel/evaluate.go
@@ -0,0 +1,211 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package cel
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/google/cel-go/cel"
+	"github.com/google/cel-go/common/types"
+	"github.com/wzshiming/easycel"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// NodeEvaluatorConfig holds configuration for a cel program
+type NodeEvaluatorConfig struct {
+	Now                    func() time.Time
+	StartedContainersTotal func(nodeName string) int64
+}
+
+// NewEnvironment returns a MetricEvaluator that is able to evaluate node metrics
+func NewEnvironment(conf NodeEvaluatorConfig) (*Environment, error) {
+	registry := easycel.NewRegistry("kwok.metric.ext.node",
+		easycel.WithTagName("json"),
+	)
+
+	e := &Environment{
+		registry:       registry,
+		conf:           conf,
+		cacheEvaluator: make(map[string]*Evaluator),
+	}
+	err := e.init()
+	if err != nil {
+		return nil, err
+	}
+
+	env, err := easycel.NewEnvironment(cel.Lib(registry))
+	if err != nil {
+		return nil, fmt.Errorf("failed to create CEL environment: %w", err)
+	}
+	e.env = env
+
+	return e, nil
+}
+
+// Environment is environment in which cel programs are executed
+type Environment struct {
+	registry       *easycel.Registry
+	env            *easycel.Environment
+	conf           NodeEvaluatorConfig
+	cacheEvaluator map[string]*Evaluator
+}
+
+func (e *Environment) init() error {
+	conversions := []any{
+		func(t metav1.Time) types.Timestamp {
+			return types.Timestamp{Time: t.Time}
+		},
+	}
+	types := []any{
+		corev1.Node{},
+		metav1.ObjectMeta{},
+	}
+
+	vars := map[string]any{
+		"node": corev1.Node{},
+	}
+
+	funcs := map[string][]any{}
+
+	methods := map[string][]any{}
+
+	if e.conf.Now != nil {
+		funcs["now"] = append(funcs["now"], e.conf.Now)
+	} else {
+		funcs["now"] = append(funcs["now"], time.Now)
+	}
+
+	unixSecond := func(t time.Time) int64 {
+		return t.Unix()
+	}
+	methods["unixSecond"] = append(methods["unixSecond"], unixSecond)
+	funcs["unixSecond"] = append(funcs["unixSecond"], unixSecond)
+
+	if e.conf.StartedContainersTotal != nil {
+		startedContainersTotal := e.conf.StartedContainersTotal
+		startedContainersTotalByNode := func(node corev1.Node) int64 {
+			return e.conf.StartedContainersTotal(node.Name)
+		}
+		methods["startedContainersTotal"] = append(methods["startedContainersTotal"], startedContainersTotal, startedContainersTotalByNode)
+		funcs["startedContainersTotal"] = append(funcs["startedContainersTotal"], startedContainersTotal, startedContainersTotalByNode)
+	}
+
+	for _, convert := range conversions {
+		err := e.registry.RegisterConversion(convert)
+		if err != nil {
+			return fmt.Errorf("failed to register convert %T: %w", convert, err)
+		}
+	}
+	for _, typ := range types {
+		err := e.registry.RegisterType(typ)
+		if err != nil {
+			return fmt.Errorf("failed to register type %T: %w", typ, err)
+		}
+	}
+	for name, val := range vars {
+		err := e.registry.RegisterVariable(name, val)
+		if err != nil {
+			return fmt.Errorf("failed to register variable %s: %w", name, err)
+		}
+	}
+	for name, list := range funcs {
+		for _, fun := range list {
+			err := e.registry.RegisterFunction(name, fun)
+			if err != nil {
+				return fmt.Errorf("failed to register function %s: %w", name, err)
+			}
+		}
+	}
+	for name, list := range methods {
+		for _, fun := range list {
+			err := e.registry.RegisterMethod(name, fun)
+			if err != nil {
+				return fmt.Errorf("failed to register method %s: %w", name, err)
+			}
+		}
+	}
+
+	return nil
+}
+
+// Compile is responsible for compiling a cel program
+func (e *Environment) Compile(src string) (*Evaluator, error) {
+	if evaluator, ok := e.cacheEvaluator[src]; ok {
+		return evaluator, nil
+	}
+	program, err := e.env.Program(src)
+	if err != nil {
+		return nil, fmt.Errorf("failed to compile metric expression: %w", err)
+	}
+
+	evaluator := &Evaluator{
+		program: program,
+	}
+	e.cacheEvaluator[src] = evaluator
+	return evaluator, nil
+}
+
+// Evaluator evaluates a cel program
+type Evaluator struct {
+	program cel.Program
+}
+
+// EvaluateFloat64 evaluates a cel program and returns a metric value and returns float64.
+func (e *Evaluator) EvaluateFloat64(node *corev1.Node) (float64, error) {
+	refVal, _, err := e.program.Eval(map[string]any{
+		"node": node,
+	})
+	if err != nil {
+		return 0, fmt.Errorf("failed to evaluate metric expression: %w", err)
+	}
+
+	switch v := refVal.(type) {
+	case types.Duration:
+		return float64(v.Duration), nil
+	case types.Int:
+		return float64(v), nil
+	case types.Double:
+		return float64(v), nil
+	case types.Uint:
+		return float64(v), nil
+	case types.Bool:
+		if v {
+			return 1, nil
+		}
+		return 0, nil
+	default:
+		return 0, fmt.Errorf("unsupported metric value type: %T", v)
+	}
+}
+
+// EvaluateString evaluates a cel program and returns a string
+func (e *Evaluator) EvaluateString(node *corev1.Node) (string, error) {
+	refVal, _, err := e.program.Eval(map[string]any{
+		"node": node,
+	})
+	if err != nil {
+		return "", fmt.Errorf("failed to evaluate metric expression: %w", err)
+	}
+
+	v, ok := refVal.(types.String)
+	if !ok {
+		return "", fmt.Errorf("unsupported metric type: %T", v)
+	}
+	return string(v), nil
+}
diff --git a/pkg/kwok/metrics/doc.go b/pkg/kwok/metrics/doc.go
new file mode 100644
index 00000000..9a79a546
--- /dev/null
+++ b/pkg/kwok/metrics/doc.go
@@ -0,0 +1,18 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Package metrics implements the metrics server.
+package metrics
diff --git a/pkg/kwok/metrics/histogram.go b/pkg/kwok/metrics/histogram.go
new file mode 100644
index 00000000..d9e0eacc
--- /dev/null
+++ b/pkg/kwok/metrics/histogram.go
@@ -0,0 +1,160 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package metrics
+
+import (
+	"math"
+	"sort"
+
+	"github.com/prometheus/client_golang/prometheus"
+	dto "github.com/prometheus/client_model/go"
+
+	"sigs.k8s.io/kwok/pkg/utils/format"
+	"sigs.k8s.io/kwok/pkg/utils/maps"
+	"sigs.k8s.io/kwok/pkg/utils/slices"
+)
+
+// HistogramOpts provides configuration options for Histogram.
+type HistogramOpts struct {
+	// Namespace, Subsystem, and Name are components of the fully-qualified
+	// name of the Histogram (created by joining these components with
+	// "_"). Only Name is mandatory, the others merely help structuring the
+	// name. Note that the fully-qualified name of the Histogram must be a
+	// valid Prometheus metric name.
+	Namespace string
+	Subsystem string
+	Name      string
+
+	// Help provides information about this Histogram.
+	//
+	// Metrics with the same fully-qualified name must have the same Help
+	// string.
+	Help string
+
+	// ConstLabels are used to attach fixed labels to this metric. Metrics
+	// with the same fully-qualified name must have the same label names in
+	// their ConstLabels.
+	//
+	// ConstLabels are only used rarely. In particular, do not use them to
+	// attach the same labels to all your metrics. Those use cases are
+	// better covered by target labels set by the scraping Prometheus
+	// server, or by one specific metric (e.g. a build_info or a
+	// machine_role metric). See also
+	// https://prometheus.io/docs/instrumenting/writing_exporters/#target-labels-not-static-scraped-labels
+	ConstLabels prometheus.Labels
+	Buckets     []float64
+}
+
+// Histogram is custom type emulating prometheus.Histogram
+type Histogram struct {
+	desc *prometheus.Desc
+
+	// buckets are the upper bounds of the buckets
+	buckets []float64
+
+	// stored is a map of le -> count
+	stored maps.SyncMap[float64, uint64]
+}
+
+// NewHistogram creates new Histogram based on Histogram options
+func NewHistogram(opts HistogramOpts) *Histogram {
+	desc := prometheus.NewDesc(
+		prometheus.BuildFQName(opts.Namespace, opts.Subsystem, opts.Name),
+		opts.Help,
+		nil,
+		opts.ConstLabels,
+	)
+
+	buckets := opts.Buckets
+	sort.Float64s(buckets)
+
+	his := &Histogram{
+		desc:    desc,
+		buckets: buckets,
+	}
+	prometheus.NewHistogram(prometheus.HistogramOpts{})
+	return his
+}
+
+// Desc returns prometheus.Desc used by every Prometheus Metric.
+func (h *Histogram) Desc() *prometheus.Desc {
+	return h.desc
+}
+
+var inf = math.Inf(1)
+
+// Write writes out Histogram data to the Metric dto.
+func (h *Histogram) Write(out *dto.Metric) error {
+	buckets := slices.Map(h.buckets, func(le float64) *dto.Bucket {
+		return &dto.Bucket{
+			CumulativeCount: format.Ptr[uint64](0),
+			UpperBound:      format.Ptr(le),
+		}
+	})
+	buckets = append(buckets, &dto.Bucket{
+		CumulativeCount: format.Ptr[uint64](0),
+		UpperBound:      format.Ptr(inf),
+	})
+
+	bucketsIndex := 0
+
+	keys := h.stored.Keys()
+	sort.Float64s(keys)
+
+	var count uint64
+	var sum float64
+	for _, le := range keys {
+		// cumulative count of previous buckets
+		for bucketsIndex < len(buckets) && le > *buckets[bucketsIndex].UpperBound {
+			bucketsIndex++
+			buckets[bucketsIndex].CumulativeCount = format.Ptr(*buckets[bucketsIndex].CumulativeCount + count)
+		}
+
+		val, _ := h.stored.Load(le)
+		// cumulative count of current bucket
+		buckets[bucketsIndex].CumulativeCount = format.Ptr(*buckets[bucketsIndex].CumulativeCount + val)
+
+		// cumulative count and sum
+		count += val
+		sum += le * float64(val)
+	}
+
+	his := &dto.Histogram{
+		Bucket:      buckets,
+		SampleCount: format.Ptr(count),
+		SampleSum:   format.Ptr(sum),
+	}
+
+	out.Histogram = his
+
+	return nil
+}
+
+// Describe sends metric description to a channel.
+func (h *Histogram) Describe(ch chan<- *prometheus.Desc) {
+	ch <- h.desc
+}
+
+// Collect sends histogram to a prometheus Metric channel.
+func (h *Histogram) Collect(ch chan<- prometheus.Metric) {
+	ch <- h
+}
+
+// Set sets value for a given le.
+func (h *Histogram) Set(le float64, val uint64) {
+	h.stored.Store(le, val)
+}
diff --git a/pkg/kwok/metrics/metrics.go b/pkg/kwok/metrics/metrics.go
new file mode 100644
index 00000000..f7834981
--- /dev/null
+++ b/pkg/kwok/metrics/metrics.go
@@ -0,0 +1,355 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package metrics
+
+import (
+	"fmt"
+	"net/http"
+	"sync"
+	"sync/atomic"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promhttp"
+	corev1 "k8s.io/api/core/v1"
+
+	"sigs.k8s.io/kwok/pkg/apis/internalversion"
+	"sigs.k8s.io/kwok/pkg/kwok/controllers"
+	"sigs.k8s.io/kwok/pkg/kwok/metrics/cel"
+	"sigs.k8s.io/kwok/pkg/log"
+)
+
+// Constants holding metric kinds.
+const (
+	KindGauge     = "gauge"
+	KindHistogram = "histogram"
+	KindCounter   = "counter"
+)
+
+// UpdateHandler handles updating metrics on request
+type UpdateHandler struct {
+	name        string
+	metric      *internalversion.Metric
+	controller  *controllers.Controller
+	environment *cel.Environment
+
+	handler  http.Handler
+	registry *prometheus.Registry
+	updates  []func() error
+}
+
+// UpdateHandlerConfig is configuration for a single node
+type UpdateHandlerConfig struct {
+	NodeName    string
+	Metrics     *internalversion.Metric
+	Controller  *controllers.Controller
+	Environment *cel.Environment
+}
+
+// NewMetricsUpdateHandler creates new metric update handler based on the config
+func NewMetricsUpdateHandler(conf UpdateHandlerConfig) (*UpdateHandler, error) {
+	registry := prometheus.NewRegistry()
+	handler := promhttp.InstrumentMetricHandler(
+		prometheus.DefaultRegisterer, promhttp.HandlerFor(registry, promhttp.HandlerOpts{Registry: registry}),
+	)
+
+	h := &UpdateHandler{
+		name:        conf.NodeName,
+		metric:      conf.Metrics,
+		controller:  conf.Controller,
+		environment: conf.Environment,
+		registry:    registry,
+		handler:     handler,
+	}
+	err := h.init()
+	if err != nil {
+		return nil, fmt.Errorf("failed to initialize metrics update handler: %w", err)
+	}
+	return h, nil
+}
+
+func (h *UpdateHandler) getNodeInfo(nodeName string) (*controllers.NodeInfo, bool) {
+	nodeInfo, ok := h.controller.GetNode(nodeName)
+	return nodeInfo, ok
+}
+
+func (h *UpdateHandler) init() error {
+	metric := h.metric
+
+	for _, m := range metric.Spec.Metrics {
+		m := m
+
+		preparedLabels, err := h.prepareLabels(&m)
+		if err != nil {
+			return fmt.Errorf("failed to prepare labels for metric %q: %w", m.Name, err)
+		}
+
+		switch m.Kind {
+		case KindGauge:
+			var val prometheus.Gauge
+
+			registerFunc := func(node *corev1.Node) error {
+				labels, err := h.createLabels(preparedLabels, node)
+				if err != nil {
+					return fmt.Errorf("failed to evaluate labels: %w", err)
+				}
+				val = prometheus.NewGauge(
+					prometheus.GaugeOpts{
+						Name:        m.Name,
+						Help:        m.Help,
+						ConstLabels: labels,
+					},
+				)
+
+				err = h.registry.Register(val)
+				if err != nil {
+					return fmt.Errorf("failed to register gauge %q: %w", m.Name, err)
+				}
+				return nil
+			}
+
+			eval, err := h.environment.Compile(m.Value)
+			if err != nil {
+				return fmt.Errorf("failed to compile metric %q: %w", metric.Name, err)
+			}
+
+			var once sync.Once
+			registered := false
+			updateFunc := func() error {
+				nodeInfo, ok := h.getNodeInfo(metric.Name)
+				if !ok {
+					if !registered {
+						return fmt.Errorf("failed to find node %q", metric.Name)
+					}
+					val.Set(0)
+					return nil
+				}
+
+				var err error
+				once.Do(func() {
+					err = registerFunc(nodeInfo.Node)
+					if err != nil {
+						registered = true
+					}
+				})
+				if err != nil {
+					return err
+				}
+
+				result, err := eval.EvaluateFloat64(nodeInfo.Node)
+				if err != nil {
+					return fmt.Errorf("failed to evaluate metric %q: %w", m.Name, err)
+				}
+				val.Set(result)
+				return nil
+			}
+			h.updates = append(h.updates, updateFunc)
+		case KindCounter:
+			counter := atomic.Int64{}
+
+			registerFunc := func(node *corev1.Node) error {
+				labels, err := h.createLabels(preparedLabels, node)
+				if err != nil {
+					return fmt.Errorf("failed to evaluate labels: %w", err)
+				}
+				val := prometheus.NewCounterFunc(
+					prometheus.CounterOpts{
+						Name:        m.Name,
+						Help:        m.Help,
+						ConstLabels: labels,
+					},
+					func() float64 {
+						return float64(counter.Load())
+					},
+				)
+
+				err = h.registry.Register(val)
+				if err != nil {
+					return fmt.Errorf("failed to register counter %q: %w", m.Name, err)
+				}
+				return nil
+			}
+
+			eval, err := h.environment.Compile(m.Value)
+			if err != nil {
+				return fmt.Errorf("failed to compile metric %q: %w", metric.Name, err)
+			}
+
+			var once sync.Once
+			registered := false
+			updateFunc := func() error {
+				nodeInfo, ok := h.getNodeInfo(metric.Name)
+				if !ok {
+					if !registered {
+						return fmt.Errorf("failed to find node %q", metric.Name)
+					}
+
+					counter.Store(0)
+					return nil
+				}
+
+				var err error
+				once.Do(func() {
+					err = registerFunc(nodeInfo.Node)
+					if err != nil {
+						registered = true
+					}
+				})
+				if err != nil {
+					return err
+				}
+
+				result, err := eval.EvaluateFloat64(nodeInfo.Node)
+				if err != nil {
+					return fmt.Errorf("failed to evaluate metric %q: %w", m.Name, err)
+				}
+				counter.Store(int64(result))
+				return nil
+			}
+			h.updates = append(h.updates, updateFunc)
+		case KindHistogram:
+			var his *Histogram
+			registerFunc := func(node *corev1.Node) error {
+				labels, err := h.createLabels(preparedLabels, node)
+				if err != nil {
+					return fmt.Errorf("failed to evaluate labels: %w", err)
+				}
+
+				buckets := make([]float64, 0, len(m.Buckets))
+				for _, b := range m.Buckets {
+					if b.Hidden {
+						continue
+					}
+					buckets = append(buckets, b.Le)
+				}
+
+				his = NewHistogram(
+					HistogramOpts{
+						Name:        m.Name,
+						Help:        m.Help,
+						ConstLabels: labels,
+						Buckets:     buckets,
+					})
+
+				err = h.registry.Register(his)
+				if err != nil {
+					return fmt.Errorf("failed to register histogram %q: %w", m.Name, err)
+				}
+				return nil
+			}
+
+			programs := make(map[float64]*cel.Evaluator, len(m.Buckets))
+			for _, b := range m.Buckets {
+				eval, err := h.environment.Compile(b.Value)
+				if err != nil {
+					return fmt.Errorf("failed to compile program for Le(%v) %q: %w", b.Le, b.Value, err)
+				}
+
+				programs[b.Le] = eval
+			}
+
+			var once sync.Once
+			registered := false
+			updateFunc := func() error {
+				nodeInfo, ok := h.getNodeInfo(metric.Name)
+				if !ok {
+					if !registered {
+						return fmt.Errorf("failed to find node %q", metric.Name)
+					}
+
+					for _, b := range m.Buckets {
+						his.Set(b.Le, 0)
+					}
+
+					return nil
+				}
+
+				var err error
+				once.Do(func() {
+					err = registerFunc(nodeInfo.Node)
+					if err != nil {
+						registered = true
+					}
+				})
+				if err != nil {
+					return err
+				}
+
+				for _, b := range m.Buckets {
+					prog, ok := programs[b.Le]
+					if !ok {
+						return fmt.Errorf("failed to find program for metric %v", b.Le)
+					}
+					value, err := prog.EvaluateFloat64(nodeInfo.Node)
+					if err != nil {
+						return fmt.Errorf("failed to evaluate metric with Le(%v): %w", b.Le, err)
+					}
+					his.Set(b.Le, uint64(value))
+				}
+				return nil
+			}
+			h.updates = append(h.updates, updateFunc)
+		}
+	}
+
+	return nil
+}
+
+func (h *UpdateHandler) prepareLabels(m *internalversion.MetricConfig) (map[string]*cel.Evaluator, error) {
+	var labels map[string]*cel.Evaluator
+	if len(m.Labels) > 0 {
+		labels = make(map[string]*cel.Evaluator, len(m.Labels))
+	}
+
+	for _, lbl := range m.Labels {
+		eval, err := h.environment.Compile(lbl.Value)
+		if err != nil {
+			return nil, fmt.Errorf("failed to compile metric label value %q: %w", lbl.Value, err)
+		}
+		labels[lbl.Name] = eval
+	}
+
+	return labels, nil
+}
+
+func (h *UpdateHandler) createLabels(labels map[string]*cel.Evaluator, node *corev1.Node) (prometheus.Labels, error) {
+	var constLabels prometheus.Labels
+	if labels != nil {
+		constLabels = make(prometheus.Labels, len(labels))
+	}
+	for name, eval := range labels {
+		var err error
+		constLabels[name], err = eval.EvaluateString(node)
+		if err != nil {
+			return nil, fmt.Errorf("failed to evaluate metric label %q for node \"%s/%s\": %w", name, node.Namespace, node.Name, err)
+		}
+	}
+	return constLabels, nil
+}
+
+func (h *UpdateHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	logger := log.FromContext(r.Context())
+	// Update metrics
+	for _, u := range h.updates {
+		err := u()
+		if err != nil {
+			logger.Error("failed to update metrics", err)
+		}
+	}
+
+	// Serve metrics
+	h.handler.ServeHTTP(w, r)
+}
diff --git a/pkg/kwok/server/metrics.go b/pkg/kwok/server/metrics.go
index 9b1f6528..d972a486 100644
--- a/pkg/kwok/server/metrics.go
+++ b/pkg/kwok/server/metrics.go
@@ -17,11 +17,43 @@ limitations under the License.
 package server
 
 import (
+	"fmt"
+
 	"github.com/prometheus/client_golang/prometheus/promhttp"
+
+	"sigs.k8s.io/kwok/pkg/kwok/metrics"
+	"sigs.k8s.io/kwok/pkg/kwok/metrics/cel"
 )
 
 // InstallMetrics registers the metrics handler on the given mux.
-func (s *Server) InstallMetrics() {
+func (s *Server) InstallMetrics() error {
 	promHandler := promhttp.Handler()
 	s.restfulCont.Handle("/metrics", promHandler)
+
+	controller := s.config.Controller
+	env, err := cel.NewEnvironment(cel.NodeEvaluatorConfig{
+		StartedContainersTotal: func(nodeName string) int64 {
+			nodeInfo, ok := controller.GetNode(nodeName)
+			if !ok {
+				return 0
+			}
+			return nodeInfo.StartedContainer.Load()
+		},
+	})
+	if err != nil {
+		return fmt.Errorf("failed to create CEL environment: %w", err)
+	}
+	for _, m := range s.config.Metrics {
+		handler, err := metrics.NewMetricsUpdateHandler(metrics.UpdateHandlerConfig{
+			NodeName:    m.Name,
+			Metrics:     m,
+			Controller:  controller,
+			Environment: env,
+		})
+		if err != nil {
+			return fmt.Errorf("failed to create metrics update handler: %w", err)
+		}
+		s.restfulCont.Handle(m.Spec.Path, handler)
+	}
+	return nil
 }
diff --git a/pkg/kwok/server/server.go b/pkg/kwok/server/server.go
index a4b4a5b4..e3efe5b9 100644
--- a/pkg/kwok/server/server.go
+++ b/pkg/kwok/server/server.go
@@ -29,6 +29,7 @@ import (
 	remotecommandconsts "k8s.io/apimachinery/pkg/util/remotecommand"
 
 	"sigs.k8s.io/kwok/pkg/apis/internalversion"
+	"sigs.k8s.io/kwok/pkg/kwok/controllers"
 	"sigs.k8s.io/kwok/pkg/log"
 	"sigs.k8s.io/kwok/pkg/utils/pools"
 )
@@ -57,6 +58,8 @@ type Config struct {
 	Logs                []*internalversion.Logs
 	ClusterAttaches     []*internalversion.ClusterAttach
 	Attaches            []*internalversion.Attach
+	Metrics             []*internalversion.Metric
+	Controller          *controllers.Controller
 }
 
 // NewServer creates a new Server.
diff --git a/pkg/kwokctl/runtime/binary/cluster.go b/pkg/kwokctl/runtime/binary/cluster.go
index 5bc4b84a..069c9fa6 100644
--- a/pkg/kwokctl/runtime/binary/cluster.go
+++ b/pkg/kwokctl/runtime/binary/cluster.go
@@ -435,6 +435,7 @@ func (c *Cluster) Install(ctx context.Context) error {
 			KubeControllerManagerPort: conf.KubeControllerManagerPort,
 			KubeSchedulerPort:         conf.KubeSchedulerPort,
 			KwokControllerPort:        conf.KwokControllerPort,
+			Metrics:                   runtime.GetMetrics(ctx),
 		})
 		if err != nil {
 			return fmt.Errorf("failed to generate prometheus yaml: %w", err)
diff --git a/pkg/kwokctl/runtime/binary/prometheus.go b/pkg/kwokctl/runtime/binary/prometheus.go
index 68bbf769..d9443bb4 100644
--- a/pkg/kwokctl/runtime/binary/prometheus.go
+++ b/pkg/kwokctl/runtime/binary/prometheus.go
@@ -21,6 +21,8 @@ import (
 	"fmt"
 	"text/template"
 
+	"sigs.k8s.io/kwok/pkg/apis/internalversion"
+
 	_ "embed"
 )
 
@@ -51,4 +53,5 @@ type BuildPrometheusConfig struct {
 	KubeControllerManagerPort uint32
 	KubeSchedulerPort         uint32
 	KwokControllerPort        uint32
+	Metrics                   []*internalversion.Metric
 }
diff --git a/pkg/kwokctl/runtime/binary/prometheus.yaml.tpl b/pkg/kwokctl/runtime/binary/prometheus.yaml.tpl
index a77680f4..def88b32 100644
--- a/pkg/kwokctl/runtime/binary/prometheus.yaml.tpl
+++ b/pkg/kwokctl/runtime/binary/prometheus.yaml.tpl
@@ -39,7 +39,18 @@ scrape_configs:
   static_configs:
   - targets:
     - localhost:{{ .KwokControllerPort }}
-
+{{ $kwokControllerPort := .KwokControllerPort }}
+{{ range .Metrics }}
+- job_name: "kwok-metric-{{ .Name }}"
+  scheme: http
+  honor_timestamps: true
+  metrics_path: {{ .Spec.Path }}
+  follow_redirects: true
+  enable_http2: true
+  static_configs:
+  - targets:
+    - localhost:{{ $kwokControllerPort }}
+{{ end }}
 {{ if .SecurePort }}
 - job_name: "kube-apiserver"
   scheme: https
diff --git a/pkg/kwokctl/runtime/compose/cluster.go b/pkg/kwokctl/runtime/compose/cluster.go
index ee707aa5..e1ef8643 100644
--- a/pkg/kwokctl/runtime/compose/cluster.go
+++ b/pkg/kwokctl/runtime/compose/cluster.go
@@ -453,6 +453,7 @@ func (c *Cluster) Install(ctx context.Context) error {
 			SecurePort:   conf.SecurePort,
 			AdminCrtPath: inClusterAdminCertPath,
 			AdminKeyPath: inClusterAdminKeyPath,
+			Metrics:      runtime.GetMetrics(ctx),
 		})
 		if err != nil {
 			return fmt.Errorf("failed to generate prometheus yaml: %w", err)
diff --git a/pkg/kwokctl/runtime/compose/prometheus.go b/pkg/kwokctl/runtime/compose/prometheus.go
index 74aa78af..54fcebb5 100644
--- a/pkg/kwokctl/runtime/compose/prometheus.go
+++ b/pkg/kwokctl/runtime/compose/prometheus.go
@@ -21,6 +21,8 @@ import (
 	"fmt"
 	"text/template"
 
+	"sigs.k8s.io/kwok/pkg/apis/internalversion"
+
 	_ "embed"
 )
 
@@ -45,4 +47,5 @@ type BuildPrometheusConfig struct {
 	SecurePort   bool
 	AdminCrtPath string
 	AdminKeyPath string
+	Metrics      []*internalversion.Metric
 }
diff --git a/pkg/kwokctl/runtime/compose/prometheus.yaml.tpl b/pkg/kwokctl/runtime/compose/prometheus.yaml.tpl
index a35b6627..36717024 100644
--- a/pkg/kwokctl/runtime/compose/prometheus.yaml.tpl
+++ b/pkg/kwokctl/runtime/compose/prometheus.yaml.tpl
@@ -40,6 +40,18 @@ scrape_configs:
   - targets:
     - "{{ .ProjectName }}-kwok-controller:10247"
 
+{{ $projectName := .ProjectName }}
+{{ range .Metrics }}
+- job_name: "kwok-metric-{{ .Name }}"
+  scheme: http
+  honor_timestamps: true
+  metrics_path: {{ .Spec.Path }}
+  follow_redirects: true
+  enable_http2: true
+  static_configs:
+  - targets:
+    - "{{ $projectName }}-kwok-controller:10247"
+{{ end }}
 {{ if .SecurePort }}
 - job_name: "kube-apiserver"
   scheme: https
diff --git a/pkg/kwokctl/runtime/kind/cluster.go b/pkg/kwokctl/runtime/kind/cluster.go
index 0a812e53..793132ea 100644
--- a/pkg/kwokctl/runtime/kind/cluster.go
+++ b/pkg/kwokctl/runtime/kind/cluster.go
@@ -194,6 +194,7 @@ func (c *Cluster) Install(ctx context.Context) error {
 			Name:            c.Name(),
 			ExtraArgs:       prometheusPatches.ExtraArgs,
 			ExtraVolumes:    prometheusPatches.ExtraVolumes,
+			Metrics:         runtime.GetMetrics(ctx),
 		}
 		if verbosity != log.LevelInfo {
 			prometheusConf.LogLevel = log.ToLogSeverityLevel(verbosity)
diff --git a/pkg/kwokctl/runtime/kind/prometheus_deployment.go b/pkg/kwokctl/runtime/kind/prometheus_deployment.go
index d08c4c2e..22eba106 100644
--- a/pkg/kwokctl/runtime/kind/prometheus_deployment.go
+++ b/pkg/kwokctl/runtime/kind/prometheus_deployment.go
@@ -56,4 +56,5 @@ type BuildPrometheusDeploymentConfig struct {
 	LogLevel        string
 	ExtraArgs       []internalversion.ExtraArgs
 	ExtraVolumes    []internalversion.Volume
+	Metrics         []*internalversion.Metric
 }
diff --git a/pkg/kwokctl/runtime/kind/prometheus_deployment.yaml.tpl b/pkg/kwokctl/runtime/kind/prometheus_deployment.yaml.tpl
index 5e32867d..b7045481 100644
--- a/pkg/kwokctl/runtime/kind/prometheus_deployment.yaml.tpl
+++ b/pkg/kwokctl/runtime/kind/prometheus_deployment.yaml.tpl
@@ -77,6 +77,17 @@ data:
         static_configs:
           - targets:
               - "localhost:10247"
+      {{ range .Metrics }}
+      - job_name: "kwok-metric-{{ .Name }}"
+        scheme: http
+        honor_timestamps: true
+        metrics_path: {{ .Spec.Path }}
+        follow_redirects: true
+        enable_http2: true
+        static_configs:
+        - targets:
+            - "localhost:10247"
+      {{ end }}
       - job_name: "kube-apiserver"
         scheme: https
         honor_timestamps: true
diff --git a/pkg/kwokctl/runtime/util.go b/pkg/kwokctl/runtime/util.go
index 203c7518..771b65fc 100644
--- a/pkg/kwokctl/runtime/util.go
+++ b/pkg/kwokctl/runtime/util.go
@@ -98,3 +98,8 @@ func GetLogVolumes(ctx context.Context) []internalversion.Volume {
 	}
 	return volumes
 }
+
+// GetMetrics returns metrics for Metrics resource.
+func GetMetrics(ctx context.Context) []*internalversion.Metric {
+	return config.FilterWithTypeFromContext[*internalversion.Metric](ctx)
+}
diff --git a/pkg/utils/maps/sync.go b/pkg/utils/maps/sync.go
index 56be819f..1c893772 100644
--- a/pkg/utils/maps/sync.go
+++ b/pkg/utils/maps/sync.go
@@ -79,3 +79,13 @@ func (m *SyncMap[K, V]) Size() int {
 	})
 	return size
 }
+
+// Keys returns all the keys in the map.
+func (m *SyncMap[K, V]) Keys() []K {
+	keys := []K{}
+	m.m.Range(func(key, value interface{}) bool {
+		keys = append(keys, key.(K))
+		return true
+	})
+	return keys
+}
diff --git a/site/content/en/docs/generated/apis.md b/site/content/en/docs/generated/apis.md
index 84e6b1c9..7421275d 100644
--- a/site/content/en/docs/generated/apis.md
+++ b/site/content/en/docs/generated/apis.md
@@ -212,6 +212,9 @@ Resource Types:
 <a href="#kwok.x-k8s.io/v1alpha1.Logs">Logs</a>
 </li>
 <li>
+<a href="#kwok.x-k8s.io/v1alpha1.Metric">Metric</a>
+</li>
+<li>
 <a href="#kwok.x-k8s.io/v1alpha1.PortForward">PortForward</a>
 </li>
 <li>
@@ -822,6 +825,96 @@ LogsSpec
 </tr>
 </tbody>
 </table>
+<h3 id="kwok.x-k8s.io/v1alpha1.Metric">
+Metric
+<a href="#kwok.x-k8s.io%2fv1alpha1.Metric"> #</a>
+</h3>
+<p>
+<p>Metric provides metrics configuration.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>apiVersion</code>
+string
+</td>
+<td>
+<code>
+kwok.x-k8s.io/v1alpha1
+</code>
+</td>
+</tr>
+<tr>
+<td>
+<code>kind</code>
+string
+</td>
+<td><code>Metric</code></td>
+</tr>
+<tr>
+<td>
+<code>metadata</code>
+<em>
+<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta">
+Kubernetes meta/v1.ObjectMeta
+</a>
+</em>
+</td>
+<td>
+<p>Standard list metadata.
+More info: <a href="https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata">https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata</a></p>
+Refer to the Kubernetes API documentation for the fields of the
+<code>metadata</code> field.
+</td>
+</tr>
+<tr>
+<td>
+<code>spec</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.MetricSpec">
+MetricSpec
+</a>
+</em>
+</td>
+<td>
+<p>Spec holds spec for metrics.</p>
+<table>
+<tr>
+<td>
+<code>path</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Path is a restful service path.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>metrics</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.MetricConfig">
+[]MetricConfig
+</a>
+</em>
+</td>
+<td>
+<p>Metrics is a list of metric configurations.</p>
+</td>
+</tr>
+</table>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="kwok.x-k8s.io/v1alpha1.PortForward">
 PortForward
 <a href="#kwok.x-k8s.io%2fv1alpha1.PortForward"> #</a>
@@ -3329,6 +3422,240 @@ LogsSpec
 </tr>
 </tbody>
 </table>
+<h3 id="kwok.x-k8s.io/v1alpha1.MetricBucket">
+MetricBucket
+<a href="#kwok.x-k8s.io%2fv1alpha1.MetricBucket"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.MetricConfig">MetricConfig</a>
+</p>
+<p>
+<p>MetricBucket is a single bucket for a metric.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>le</code>
+<em>
+float64
+</em>
+</td>
+<td>
+<p>Le is less-than or equal.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>value</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Value is a CEL expression.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>hidden</code>
+<em>
+bool
+</em>
+</td>
+<td>
+<p>Hidden is means that this bucket not shown in the metric.
+but value will be calculated and cumulative into the next bucket.</p>
+</td>
+</tr>
+</tbody>
+</table>
+<h3 id="kwok.x-k8s.io/v1alpha1.MetricConfig">
+MetricConfig
+<a href="#kwok.x-k8s.io%2fv1alpha1.MetricConfig"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.MetricSpec">MetricSpec</a>
+</p>
+<p>
+<p>MetricConfig provides metric configuration to a single metric</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>name</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Name is the fully-qualified name of the metric.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>help</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Help provides information about this metric.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>kind</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Kind is kind of metric (ex. counter, gauge, histogram).</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>labels</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.MetricLabel">
+[]MetricLabel
+</a>
+</em>
+</td>
+<td>
+<p>Labels are metric labels.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>value</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Value is a CEL expression.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>buckets</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.MetricBucket">
+[]MetricBucket
+</a>
+</em>
+</td>
+<td>
+<p>Buckets is a list of buckets for a histogram metric.</p>
+</td>
+</tr>
+</tbody>
+</table>
+<h3 id="kwok.x-k8s.io/v1alpha1.MetricLabel">
+MetricLabel
+<a href="#kwok.x-k8s.io%2fv1alpha1.MetricLabel"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.MetricConfig">MetricConfig</a>
+</p>
+<p>
+<p>MetricLabel holds label name and the value of the label.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>name</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Name is a label name.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>value</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Value is a CEL expression.</p>
+</td>
+</tr>
+</tbody>
+</table>
+<h3 id="kwok.x-k8s.io/v1alpha1.MetricSpec">
+MetricSpec
+<a href="#kwok.x-k8s.io%2fv1alpha1.MetricSpec"> #</a>
+</h3>
+<p>
+<em>Appears on: </em>
+<a href="#kwok.x-k8s.io/v1alpha1.Metric">Metric</a>
+</p>
+<p>
+<p>MetricSpec holds spec for metrics.</p>
+</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>path</code>
+<em>
+string
+</em>
+</td>
+<td>
+<p>Path is a restful service path.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>metrics</code>
+<em>
+<a href="#kwok.x-k8s.io/v1alpha1.MetricConfig">
+[]MetricConfig
+</a>
+</em>
+</td>
+<td>
+<p>Metrics is a list of metric configurations.</p>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="kwok.x-k8s.io/v1alpha1.ObjectSelector">
 ObjectSelector
 <a href="#kwok.x-k8s.io%2fv1alpha1.ObjectSelector"> #</a>


Test patch:
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
index a657279e..e28e9746 100644
--- a/.github/workflows/test.yaml
+++ b/.github/workflows/test.yaml
@@ -321,6 +321,11 @@ jobs:
         run: |
           ./hack/e2e-test.sh kwokctl/kwokctl_${{ matrix.kwokctl-runtime }}_extra
 
+      - name: Test Metrics
+        shell: bash
+        run: |
+          ./hack/e2e-test.sh kwokctl/kwokctl_${{ matrix.kwokctl-runtime }}_metric
+
       - name: Test Benchmark
         if: ${{ matrix.os == 'ubuntu-latest' && matrix.kwokctl-runtime != 'kind' && matrix.kwokctl-runtime != 'kind-podman' }}
         shell: bash
diff --git a/pkg/kwok/controllers/pod_controller_test.go b/pkg/kwok/controllers/pod_controller_test.go
index c3003633..805c5881 100644
--- a/pkg/kwok/controllers/pod_controller_test.go
+++ b/pkg/kwok/controllers/pod_controller_test.go
@@ -161,6 +161,9 @@ func TestPodController(t *testing.T) {
 		}
 		return nodeInfo, true
 	}
+	nodeHasMetric := func(nodeName string) bool {
+		return false
+	}
 	podStages, _ := NewStagesFromYaml([]byte(stages.DefaultPodStages))
 	annotationSelector, _ := labels.Parse("fake=custom")
 	pods, err := NewPodController(PodControllerConfig{
@@ -170,6 +173,7 @@ func TestPodController(t *testing.T) {
 		DisregardStatusWithAnnotationSelector: annotationSelector.String(),
 		Stages:                                podStages,
 		NodeGetFunc:                           nodeGetFunc,
+		NodeHasMetric:                         nodeHasMetric,
 		FuncMap:                               defaultFuncMap,
 		PlayStageParallelism:                  2,
 	})
diff --git a/pkg/kwok/metrics/cel/evaluate_test.go b/pkg/kwok/metrics/cel/evaluate_test.go
new file mode 100644
index 00000000..bf7c7478
--- /dev/null
+++ b/pkg/kwok/metrics/cel/evaluate_test.go
@@ -0,0 +1,62 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package cel
+
+import (
+	"testing"
+	"time"
+
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+func TestNodeEvaluation(t *testing.T) {
+	now := time.Date(2022, 1, 1, 0, 0, 0, 0, time.UTC)
+
+	n := &corev1.Node{
+		ObjectMeta: metav1.ObjectMeta{
+			CreationTimestamp: metav1.Time{Time: now.Add(-24 * time.Hour)},
+		},
+	}
+	exp := "( now().unixSecond() - node.metadata.creationTimestamp.unixSecond() ) * node.startedContainersTotal() / 10"
+
+	env, err := NewEnvironment(NodeEvaluatorConfig{
+		Now: func() time.Time {
+			return now
+		},
+		StartedContainersTotal: func(nodeName string) int64 {
+			return 2
+		},
+	})
+	if err != nil {
+		t.Fatalf("failed to instantiate node Evaluator: %v", err)
+	}
+
+	eval, err := env.Compile(exp)
+	if err != nil {
+		t.Fatalf("failed to compile expression: %v", err)
+	}
+
+	actual, err := eval.EvaluateFloat64(n)
+	if err != nil {
+		t.Fatalf("evaluation failed: %v", err)
+	}
+
+	if actual != 17280 {
+		t.Errorf("expected %v, got %v", 17280, actual)
+	}
+}
diff --git a/pkg/kwok/metrics/histogram_test.go b/pkg/kwok/metrics/histogram_test.go
new file mode 100644
index 00000000..dd933602
--- /dev/null
+++ b/pkg/kwok/metrics/histogram_test.go
@@ -0,0 +1,95 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package metrics
+
+import (
+	"testing"
+
+	"github.com/google/go-cmp/cmp"
+	dto "github.com/prometheus/client_model/go"
+
+	"sigs.k8s.io/kwok/pkg/utils/format"
+)
+
+func TestHistogramObserve(t *testing.T) {
+	opts := HistogramOpts{
+		Name:    "name",
+		Help:    "help",
+		Buckets: []float64{0.5, 1, 2.5, 5, 10},
+	}
+
+	data := map[float64]uint64{
+		0.9:  0b_00_00_01,
+		1.0:  0b_00_00_10,
+		1.1:  0b_00_01_00,
+		2.4:  0b_00_10_00,
+		10.0: 0b_01_00_00,
+		11.0: 0b_10_00_00,
+	}
+
+	his := NewHistogram(opts)
+	for le, count := range data {
+		his.Set(le, count)
+	}
+
+	var out dto.Metric
+	if err := his.Write(&out); err != nil {
+		t.Fatalf("Failed to write metric: %v", err)
+	}
+
+	sampleCount := uint64(0)
+	sampleSum := float64(0)
+	for le, count := range data {
+		sampleCount += count
+		sampleSum += le * float64(count)
+	}
+
+	want := &dto.Histogram{
+		SampleCount: format.Ptr(sampleCount),
+		SampleSum:   format.Ptr(sampleSum),
+		Bucket: []*dto.Bucket{
+			{
+				CumulativeCount: format.Ptr[uint64](0b_00_00_00),
+				UpperBound:      format.Ptr(0.5),
+			},
+			{
+				CumulativeCount: format.Ptr[uint64](0b_00_00_11),
+				UpperBound:      format.Ptr(1.0),
+			},
+			{
+				CumulativeCount: format.Ptr[uint64](0b_00_11_11),
+				UpperBound:      format.Ptr(2.5),
+			},
+			{
+				CumulativeCount: format.Ptr[uint64](0b_00_11_11),
+				UpperBound:      format.Ptr(5.0),
+			},
+			{
+				CumulativeCount: format.Ptr[uint64](0b_01_11_11),
+				UpperBound:      format.Ptr(10.0),
+			},
+			{
+				CumulativeCount: format.Ptr[uint64](0b_11_11_11),
+				UpperBound:      format.Ptr(inf),
+			},
+		},
+	}
+
+	if diff := cmp.Diff(out.Histogram, want); diff != "" {
+		t.Errorf("Histogram mismatch (-want +got):\n%s", diff)
+	}
+}
diff --git a/test/kwokctl/kwokctl_binary_metric.test.sh b/test/kwokctl/kwokctl_binary_metric.test.sh
new file mode 100755
index 00000000..8f940902
--- /dev/null
+++ b/test/kwokctl/kwokctl_binary_metric.test.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Copyright 2023 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+DIR="$(dirname "${BASH_SOURCE[0]}")"
+
+DIR="$(realpath "${DIR}")"
+
+source "${DIR}/helper.sh"
+
+function main() {
+  local all_releases=("${@}")
+
+  test_all "binary" "metric" "${all_releases[@]}" || exit 1
+}
+
+requirements_for_binary
+
+mapfile -t releases < <(supported_releases)
+main "${releases[@]}"
diff --git a/test/kwokctl/kwokctl_docker_metric.test.sh b/test/kwokctl/kwokctl_docker_metric.test.sh
new file mode 100755
index 00000000..ffa8741a
--- /dev/null
+++ b/test/kwokctl/kwokctl_docker_metric.test.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Copyright 2023 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+DIR="$(dirname "${BASH_SOURCE[0]}")"
+
+DIR="$(realpath "${DIR}")"
+
+source "${DIR}/helper.sh"
+
+function main() {
+  local all_releases=("${@}")
+
+  test_all "docker" "metric" "${all_releases[@]}" || exit 1
+}
+
+requirements
+
+mapfile -t releases < <(supported_releases)
+main "${releases[@]}"
diff --git a/test/kwokctl/kwokctl_kind-podman_metric.test.sh b/test/kwokctl/kwokctl_kind-podman_metric.test.sh
new file mode 100755
index 00000000..0d24622b
--- /dev/null
+++ b/test/kwokctl/kwokctl_kind-podman_metric.test.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Copyright 2023 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+DIR="$(dirname "${BASH_SOURCE[0]}")"
+
+DIR="$(realpath "${DIR}")"
+
+source "${DIR}/helper.sh"
+
+function main() {
+  local all_releases=("${@}")
+
+  test_all "kind-podman" "metric" "${all_releases[@]}" || exit 1
+}
+
+requirements_for_podman
+
+mapfile -t releases < <(supported_releases)
+main "${releases[@]}"
diff --git a/test/kwokctl/kwokctl_kind_metric.test.sh b/test/kwokctl/kwokctl_kind_metric.test.sh
new file mode 100755
index 00000000..6e2b44bc
--- /dev/null
+++ b/test/kwokctl/kwokctl_kind_metric.test.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Copyright 2023 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+DIR="$(dirname "${BASH_SOURCE[0]}")"
+
+DIR="$(realpath "${DIR}")"
+
+source "${DIR}/helper.sh"
+
+function main() {
+  local all_releases=("${@}")
+
+  test_all "kind" "metric" "${all_releases[@]}" || exit 1
+}
+
+requirements
+
+mapfile -t releases < <(supported_releases)
+main "${releases[@]}"
diff --git a/test/kwokctl/kwokctl_metric_test.sh b/test/kwokctl/kwokctl_metric_test.sh
new file mode 100755
index 00000000..28636930
--- /dev/null
+++ b/test/kwokctl/kwokctl_metric_test.sh
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# Copyright 2023 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+DIR="$(dirname "${BASH_SOURCE[0]}")"
+
+DIR="$(realpath "${DIR}")"
+
+source "${DIR}/suite.sh"
+
+RELEASES=()
+
+function usage() {
+  echo "Usage: $0 <kube-version...>"
+  echo "  <kube-version> is the version of kubernetes to test against."
+}
+
+function args() {
+  if [[ $# -eq 0 ]]; then
+    usage
+    exit 1
+  fi
+  while [[ $# -gt 0 ]]; do
+    RELEASES+=("${1}")
+    shift
+  done
+}
+
+function test_apply_node_and_pod() {
+  local name="${1}"
+  if ! kwokctl --name "${name}" kubectl apply -f "${DIR}/fake-node.yaml"; then
+    echo "Error: fake-node apply failed"
+    return 1
+  fi
+  if ! retry 120 kwokctl --name "${name}" kubectl apply -f "${DIR}/fake-pod-in-other-ns.yaml"; then
+    echo "Error: fake-pod apply failed"
+    return 1
+  fi
+  if ! kwokctl --name "${name}" kubectl apply -f "${DIR}/fake-deployment.yaml"; then
+    echo "Error: fake-deployment apply failed"
+    return 1
+  fi
+  if ! kwokctl --name "${name}" kubectl wait pod -A --all --for=condition=Ready --timeout=60s; then
+    echo "Error: fake-pod wait failed"
+    echo kwokctl --name "${name}" kubectl get pod -A --all
+    kwokctl --name "${name}" kubectl get pod -A --all
+    return 1
+  fi
+}
+
+function metric_value() {
+  local result="${1}"
+  local metric_name="${2}"
+  echo "${result}" | grep "^${metric_name}" | cut -d " " -f 2
+}
+
+function test_node_metrics() {
+  local result
+  result="$(curl http://localhost:10250/metrics/nodes/fake-node)"
+  local found_metric=0
+  for ((i = 0; i < 15; i++)); do
+    if [[ "$(metric_value "${result}" 'kubelet_node_name{node="fake-node"}')" -eq 1 ]]; then
+      found_metric=1
+      break
+    fi
+    sleep 1
+  done
+
+  if [[ "${found_metric}" -ne 1 ]]; then
+    echo "Error: kubelet_node_name of fake-node is not equal to 1"
+    return 1
+  fi
+
+  found_metric=0
+  for ((i = 0; i < 15; i++)); do
+    if [[ "$(metric_value "${result}" "kubelet_started_containers_total")" -ne 1 ]]; then
+      found_metric=1
+      break
+    fi
+    sleep 1
+  done
+
+  if [[ "${found_metric}" -ne 1 ]]; then
+    echo "Error: started containers total is not equal to 1"
+    return 1
+  fi
+
+  if [[ -z "$(metric_value "${result}" "kubelet_pleg_relist_duration_seconds_count")" ]]; then
+    echo "Error: kubelet_pleg_relist_duration_seconds_count not set"
+    return 1
+  fi
+}
+
+function main() {
+  local failed=()
+
+  for release in "${RELEASES[@]}"; do
+    echo "------------------------------"
+    echo "Testing metrics on ${KWOK_RUNTIME} for ${release}"
+    name="metric-cluster-${KWOK_RUNTIME}-${release//./-}"
+    create_cluster "${name}" "${release}" --config "${DIR}/metric.yaml" --prometheus-port "9090" --controller-port "10250"
+    test_apply_node_and_pod "${name}" || failed+=("apply_node_and_pod")
+    test_node_metrics "${name}" || failed+=("test_node_metrics")
+    delete_cluster "${name}"
+  done
+
+  if [[ "${#failed[@]}" -ne 0 ]]; then
+    echo "------------------------------"
+    echo "Error: Some tests failed"
+    for test in "${failed[@]}"; do
+      echo " - ${test}"
+    done
+    exit 1
+  fi
+}
+
+args "$@"
+
+main
diff --git a/test/kwokctl/kwokctl_nerdctl_metric.test.sh b/test/kwokctl/kwokctl_nerdctl_metric.test.sh
new file mode 100755
index 00000000..4bb7128b
--- /dev/null
+++ b/test/kwokctl/kwokctl_nerdctl_metric.test.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Copyright 2023 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+DIR="$(dirname "${BASH_SOURCE[0]}")"
+
+DIR="$(realpath "${DIR}")"
+
+source "${DIR}/helper.sh"
+
+function main() {
+  local all_releases=("${@}")
+
+  test_all "nerdctl" "metric" "${all_releases[@]}" || exit 1
+}
+
+requirements_for_nerdctl
+
+mapfile -t releases < <(supported_releases)
+main "${releases[@]}"
diff --git a/test/kwokctl/kwokctl_podman_metric.test.sh b/test/kwokctl/kwokctl_podman_metric.test.sh
new file mode 100755
index 00000000..fd4d38ca
--- /dev/null
+++ b/test/kwokctl/kwokctl_podman_metric.test.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Copyright 2023 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+DIR="$(dirname "${BASH_SOURCE[0]}")"
+
+DIR="$(realpath "${DIR}")"
+
+source "${DIR}/helper.sh"
+
+function main() {
+  local all_releases=("${@}")
+
+  test_all "podman" "metric" "${all_releases[@]}" || exit 1
+}
+
+requirements_for_podman
+
+mapfile -t releases < <(supported_releases)
+main "${releases[@]}"
diff --git a/test/kwokctl/metric.yaml b/test/kwokctl/metric.yaml
new file mode 100644
index 00000000..f14ad965
--- /dev/null
+++ b/test/kwokctl/metric.yaml
@@ -0,0 +1,51 @@
+kind: Metric
+apiVersion: kwok.x-k8s.io/v1alpha1
+metadata:
+  name: fake-node
+spec:
+  path: "/metrics/nodes/fake-node"
+  metrics:
+    - name: kubelet_node_name
+      help: "[ALPHA] The node's name. The count is always 1."
+      kind: gauge
+      labels:
+        - name: node
+          value: 'node.metadata.name'
+      value: '1.0'
+    - name: kubelet_started_containers_total
+      help: "[ALPHA] Cumulative number of containers started"
+      kind: counter
+      value: 'startedContainersTotal( node.metadata.name )'
+    - name: kubelet_pleg_relist_duration_seconds
+      help: "[ALPHA] Duration in seconds of a single pod list and pod events list call."
+      kind: histogram
+      buckets:
+        - le: 0.005
+          value: '0'
+
+        - le: 0.01
+          value: '(unixSecond(now()) - unixSecond(node.metadata.creationTimestamp)) / 10'
+        - le: 0.025
+          value: '(unixSecond(now()) - unixSecond(node.metadata.creationTimestamp)) / 9'
+        - le: 0.05
+          value: '(unixSecond(now()) - unixSecond(node.metadata.creationTimestamp)) / 9'
+        - le: 0.1
+          value: '(unixSecond(now()) - unixSecond(node.metadata.creationTimestamp)) / 8'
+        - le: 0.25
+          value: '(unixSecond(now()) - unixSecond(node.metadata.creationTimestamp)) / 8'
+        - le: 0.5
+          value: 'unixSecond(now()) - unixSecond(node.metadata.creationTimestamp)'
+        - le: 1
+          value: 'unixSecond(now()) - unixSecond(node.metadata.creationTimestamp)'
+        - le: 2.5
+          value: 'unixSecond(now()) - unixSecond(node.metadata.creationTimestamp)'
+        - le: 5
+          value: '1'
+        - le: 7.5
+          value: '1'
+          hidden: true
+        - le: 10
+          value: '1'
+        - le: 20
+          value: '1'
+          hidden: true