Repository: kubernetes-sigs/kueue

Issue:
Dashboard support in Kueue
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:

It would be great if we can have an insight about what's our queueing system looks like at real time
- for administrators, it helps to understand the total resource usages in-between cluster queues and whether we should make them a cohort
- for batch users, they will have an overview about the job queueing, how many jobs are pending for scheduling, how long jobs are waiting.

Overall, it's a great enhancement especially for production env.

**Why is this needed**:

A big enhancement and a great insight of kueue system.

**Completion requirements**:

This enhancement requires the following artifacts:

- [x] Design doc
- [ ] API change
- [x] Docs update

The artifacts should be linked in subsequent comments.

Some advices here:
- For administrators:
  - clusterQueue resource groups
  - clusterQueue resource utilizations
  - localQueues
  - cohorts
- For users:
  - jobs with their status

Hints from issue comments:
@alculquicondor: Do you have a high level idea of how to get there?

For example, would metrics + grafana yamls be enough for the administrative side?

For end users, certainly grafana wouldn't be viable. But what could be the MVP that would keep kueue largely non-opinionated and reusable (so you can integrate it with your own UI, if you already have one). Could we offer a CLI instead?@moficodes: Kueue already spits out prometheus metrics. Building a UI based on that can be useful and the UI should be optional to deploy. 

I do wonder if it is more useful for us to provide general purpose grafana dashboard and make it available in https://grafana.com/grafana/dashboards/

@kerthcet: Building on metrics is helpful I think, but the dashboard is more than that, like it will display the basic information about the system, how many queues there, what their names are, how many jobs inside the queue, it can be interactive. We can get the information via the apis directly or we can have a lightweight database inside for cache, like sqlite. 

We may need some frontend volunteers if we want to finish this work. As a MVP, IMHO, I think it should include
- Most of the API objects(clusterQueue, localQueue, Job, resourceFlavor, workload) at least, also including their relationships
- Some exported metrics
@ahg-g: +1000, this is a much needed experience gap, I would be happy to review proposals.

@kerthcet may be we start by looking at similar batch schedulers and see what "screens" they offer to inform and help seed what we need to build?@kerthcet: > @kerthcet may be we start by looking at similar batch schedulers and see what "screens" they offer to inform and help seed what we need to build?

Yes, that's a good approach, let me make a research first and then I'll share with your guys. I know YuniKorn has a dashboard. Also cc @BinL233 @zeddit: @ahg-g 
it's a core feature which will make kueue easy to use.
A alternative would be airflow, and its UI looks like below
![image](https://github.com/kubernetes-sigs/kueue/assets/30164206/afc164a3-2b6f-4285-b000-ef1a197f07cf)
it contains task status, code, and audit logs which would be useful information for user to inspect their jobs.
besides, airflow integrates with idp like ldap or oidc and provides access control and permission managment features.

However, airflow doesn't provide an API to submit one-time run jobs like ml training jobs, which is the core application for kueue.@kerthcet: cc @B1F030 we also did some research around the popular queueing systems, I think we can provide a summary about the   essential elements in dashboard, or even a prototype. Can you help with this @B1F030 ?@samzong: Hi guys, I want to try involved in the prototyping part of the dashbaord Desgin, and provide the prototype like figma.@kerthcet: Thanks @samzong 
We can provide a based design, and share with the community for feedbacks and then involve the developments, any concerns? @alculquicondor @alculquicondor: Maybe we can start with a list of views you would like to have and do priority sorting @kerthcet: > Maybe we can start with a list of views you would like to have and do priority sorting

@B1F030 is doing this.@Sharpz7: Hey folks, https://github.com/armadaproject/armada has a UI in the form of lookout.

Our demo UI is here: https://ui.demo.armadaproject.io/

Let us know what you think of it - I think many parts of it could be suitable for lookout and we would be interested in contributing.

Thanks!@kerthcet: Thanks @Sharpz7 that's helpful, and we have a general idea now, @samzong is doing the prototyping, once we've done, we'll share a google doc/figma with your guys, hope to work together.@Sharpz7: Great, Thank you :))@alculquicondor: @Sharpz7 what is a lookout in this context?@k8s-triage-robot: The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@tenzen-y: /remove-lifecycle stale@k8s-triage-robot: The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@kerthcet: Are you still following this @samzong ?@samzong: @kerthcet 

Absolutely, I'm still very interested in this. The good news is that I'll have more time to contribute to open source in the coming period. I'll make sure to push this forward as soon as possible.@trashadewan: Hey, 
there is a grafana dashboard shown in https://www.youtube.com/watch?v=B63vT2_UYE4, would you know if this is avialble for use. somewhere?@alculquicondor: I think it might be this one https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/best-practices/gke-batch-refarch/02_platform/monitoring/deploy-dashboard.yaml

but @moficodes and @alizaidis to confirm@alizaidis: Yep that's the one!@k8s-triage-robot: The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle rotten`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle rotten

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@k8s-triage-robot: The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Reopen this issue with `/reopen`
- Mark this issue as fresh with `/remove-lifecycle rotten`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/close not-planned

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@k8s-ci-robot: @k8s-triage-robot: Closing this issue, marking it as "Not Planned".

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/940#issuecomment-2353851432):

>The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
>
>This bot triages issues according to the following rules:
>- After 90d of inactivity, `lifecycle/stale` is applied
>- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
>- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed
>
>You can:
>- Reopen this issue with `/reopen`
>- Mark this issue as fresh with `/remove-lifecycle rotten`
>- Offer to help out with [Issue Triage][1]
>
>Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).
>
>/close not-planned
>
>[1]: https://www.kubernetes.dev/docs/guide/issue-triage/


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@akram: Hi everyone,

As a side project to just run a demo I have contributed kueue-viz : as a kueue dashboard.
The project is availabe here: https://github.com/akram/kueue-viz

It is still very basic , but every contribution and feedback are welcome.


<img width="1511" alt="image" src="https://github.com/user-attachments/assets/5728e0cf-2aad-4c27-9511-78da1d995347">



@kannon92: /reopen@k8s-ci-robot: @kannon92: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/940#issuecomment-2483413102):

>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@kannon92: @mwielgus has mentioned that this is still of interest to the Kueue project. @kerthcet: Thanks @akram for the work!@mimowo: /reopen
let's use this issue to continue with the next steps of productazing the kueue-viz sub-project. I post some steps in the [comment](https://github.com/kubernetes-sigs/kueue/pull/3727#issuecomment-2519804009):
- I think it would be good to publish the image
- move it out of the cmd/experimental
- align the release process with the main Kueue
(possibly more)
- reference the project from the kueue main documentation page (https://kueue.sigs.k8s.io/), for starter we can do similar as for kueuectl plugin, but later it deserves more advertisement I think
- e2e tests for the dashboard backend and frontend

I'm also open to track the improvements as dedicated issue, just listing them here as a starting point.@k8s-ci-robot: @mimowo: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/940#issuecomment-2519815489):

>/reopen
>let's use this issue to continue with the next steps of productazing the kueue-viz sub-project. I post some steps in the [comment](https://github.com/kubernetes-sigs/kueue/pull/3727#issuecomment-2519804009)


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@mimowo: /remove-lifecycle rotten@k8s-triage-robot: The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@mimowo: /remove-lifecycle stale
This is being actively worked on@mbobrovskyi: @mimowo @akram Looks like we have the Dashboard now. Can we close this issue, or is there still something pending?@mimowo: For the functional requirements I will defer to @akram and @kannon92 the question. For the technical I would like to complete this https://github.com/kubernetes-sigs/kueue/issues/3796 too.@kannon92: Let's keep this open until 0.12 release. I think technically we don't yet have this as it is not released.@shaharelys: Hey guys! We would really like to try this one for our use case. I saw this is on the roadman for 2025 but I realize now this is already available? Is that correct?

Would love to be an alpha user. Could also contibute from the usecase POV of an administrator for a 50 H100s k8s cluster with 10 ClusterQueues :)

Is that possible? Thanks!@kannon92: > Hey guys! We would really like to try this one for our use case. I saw this is on the roadman for 2025 but I realize now this is already available? Is that correct?
> 
> Would love to be an alpha user. Could also contibute from the usecase POV of an administrator for a 50 H100s k8s cluster with 10 ClusterQueues :)
> 
> Is that possible? Thanks!

It is available in main right now. We do not yet publish this in a release. For 0.12 it will be released.

Please feel free to test or be an alpha user. Feedback is welcome!@kannon92: I change my mind on this. We should close it. It will be released in 0.12.@kannon92: /close@mimowo: /close@k8s-ci-robot: @mimowo: Closing this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/940#issuecomment-2884562246):

>/close


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

Solution patch:
diff --git a/apis/config/v1beta1/configuration_types.go b/apis/config/v1beta1/configuration_types.go
index b398b91b4..fb2d02321 100644
--- a/apis/config/v1beta1/configuration_types.go
+++ b/apis/config/v1beta1/configuration_types.go
@@ -82,6 +82,14 @@ type ControllerManager struct {
 	// +optional
 	Health ControllerHealth `json:"health,omitempty"`
 
+	// PprofBindAddress is the TCP address that the controller should bind to
+	// for serving pprof.
+	// It can be set to "" or "0" to disable the pprof serving.
+	// Since pprof may contain sensitive information, make sure to protect it
+	// before exposing it to public.
+	// +optional
+	PprofBindAddress string `json:"pprofBindAddress,omitempty"`
+
 	// Controller contains global configuration options for controllers
 	// registered within this manager.
 	// +optional
diff --git a/charts/kueue/values.yaml b/charts/kueue/values.yaml
index 48731518c..6882afe59 100644
--- a/charts/kueue/values.yaml
+++ b/charts/kueue/values.yaml
@@ -61,6 +61,7 @@ managerConfig:
     clientConnection:
       qps: 50
       burst: 100
+    #pprofBindAddress: :8082
     #waitForPodsReady:
     #  enable: true
     #manageJobsWithoutQueueName: true
diff --git a/config/components/manager/controller_manager_config.yaml b/config/components/manager/controller_manager_config.yaml
index 6075370c8..6d99cbef0 100644
--- a/config/components/manager/controller_manager_config.yaml
+++ b/config/components/manager/controller_manager_config.yaml
@@ -19,6 +19,7 @@ controller:
 clientConnection:
   qps: 50
   burst: 100
+#pprofBindAddress: :8082
 #waitForPodsReady:
 #  enable: true
 #manageJobsWithoutQueueName: true
diff --git a/pkg/config/config.go b/pkg/config/config.go
index 6aada3dd1..c8d9f7b51 100644
--- a/pkg/config/config.go
+++ b/pkg/config/config.go
@@ -52,6 +52,10 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.MetricsBindAddress = cfg.Metrics.BindAddress
 	}
 
+	if o.PprofBindAddress == "" && cfg.PprofBindAddress != "" {
+		o.PprofBindAddress = cfg.PprofBindAddress
+	}
+
 	if o.HealthProbeBindAddress == "" && cfg.Health.HealthProbeBindAddress != "" {
 		o.HealthProbeBindAddress = cfg.Health.HealthProbeBindAddress
 	}
diff --git a/site/content/en/docs/tasks/run_python_jobs.md b/site/content/en/docs/tasks/run_python_jobs.md
index c8a189740..d29668dec 100644
--- a/site/content/en/docs/tasks/run_python_jobs.md
+++ b/site/content/en/docs/tasks/run_python_jobs.md
@@ -43,7 +43,7 @@ The following examples demonstrate different use cases for using Kueue in Python
 This example demonstrates installing Kueue to an existing cluster. You can save this
 script to your local machine as `install-kueue-queues.py`. 
 
-{{% include "python/sample-job.py" "python" %}}
+{{% include "python/install-kueue-queues.py" "python" %}}
 
 And then run as follows:
 
@@ -98,3 +98,64 @@ Use:
 
 You can also change the container image with `--image` and args with `--args`.
 For more customization, you can edit the example script.
+
+### Interact with Queues and Jobs
+
+If you are developing an application that submits jobs and needs to interact
+with and check on them, you likely want to interact with queues or jobs directly.
+After running the example above, you can test the following example to interact
+with the results. Write the following to a script called `sample-queue-control.py`.
+
+{{% include "python/sample-queue-control.py" "python" %}}
+
+To make the output more interesting, we can run a few random jobs first:
+
+```bash
+python sample-job.py
+python sample-job.py
+python sample-job.py --job-name tacos
+```
+
+And then run the script to see your queue and sample job that you submit previously.
+
+```bash
+python sample-queue-control.py
+```
+```console
+⛑️  Local Queues
+Found queue user-queue
+  Admitted workloads: 3
+  Pending workloads: 0
+  Flavor default-flavor has resources [{'name': 'cpu', 'total': '3'}, {'name': 'memory', 'total': '600Mi'}]
+
+💼️ Jobs
+Found job sample-job-8n5sb
+  Succeeded: 3
+  Ready: 0
+Found job sample-job-gnxtl
+  Succeeded: 1
+  Ready: 0
+Found job tacos46bqw
+  Succeeded: 1
+  Ready: 1
+```
+
+If you wanted to filter jobs to a specific queue, you can do this via the job labels
+under `job["metadata"]["labels"]["kueue.x-k8s.io/queue-name"]'. To list a specific job by
+name, you can do:
+
+```python
+from kubernetes import client, config
+
+# Interact with batch
+config.load_kube_config()
+batch_api = client.BatchV1Api()
+
+# This is providing the name, and namespace
+job = batch_api.read_namespaced_job("tacos46bqw", "default")
+print(job)
+```
+
+See the [BatchV1](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/BatchV1Api.md)
+API documentation for more calls.
+
diff --git a/site/static/examples/python/install-kueue-queues.py b/site/static/examples/python/install-kueue-queues.py
index d1796269f..bf6bf1d51 100644
--- a/site/static/examples/python/install-kueue-queues.py
+++ b/site/static/examples/python/install-kueue-queues.py
@@ -6,7 +6,7 @@ import requests
 import argparse
 
 # install-kueue-queues.py will:
-# 1. install queue from the latest on GitHub
+# 1. install queue from the latest or a specific version on GitHub
 # This example will demonstrate installing Kueue and applying a YAML file (local) to install Kueue
 
 # Make sure your cluster is running!
@@ -22,8 +22,8 @@ def get_parser():
     )
     parser.add_argument(
         "--version",
-        help="Version of Kueue to install",
-        default="0.4.0",
+        help="Version of Kueue to install (if undefined, will install from master branch)",
+        default=None,
     )
     return parser
 
@@ -39,12 +39,24 @@ def main():
     install_kueue(args.version)
 
 
+def get_install_url(version):
+    """
+    Get the install version.
+
+    If a version is specified, use it. Otherwise install
+    from the main branch.
+    """
+    if version is not None:
+        return f"https://github.com/kubernetes-sigs/kueue/releases/download/v{version}/manifests.yaml"
+    return "https://github.com/kubernetes-sigs/kueue/config/default?ref=main"
+
+
 def install_kueue(version):
     """
     Install Kueue of a particular version.
     """
     print("⭐️ Installing Kueue...")
-    url = f"https://github.com/kubernetes-sigs/kueue/releases/download/v{version}/manifests.yaml"
+    url = get_install_url(version)
     with tempfile.NamedTemporaryFile(delete=True) as install_yaml:
         res = requests.get(url)
         assert res.status_code == 200
diff --git a/site/static/examples/python/sample-queue-control.py b/site/static/examples/python/sample-queue-control.py
new file mode 100644
index 000000000..98b8ba70f
--- /dev/null
+++ b/site/static/examples/python/sample-queue-control.py
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+
+import argparse
+from kubernetes import config, client
+
+# sample-queue-control.py
+# This will show how to interact with queues
+
+# Make sure your cluster is running!
+config.load_kube_config()
+crd_api = client.CustomObjectsApi()
+api_client = crd_api.api_client
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(
+        description="Interact with Queues e",
+        formatter_class=argparse.RawTextHelpFormatter,
+    )
+    parser.add_argument(
+        "--namespace",
+        help="namespace to list for",
+        default="default",
+    )
+    return parser
+
+
+def main():
+    """
+    Get a listing of jobs in the queue
+    """
+    parser = get_parser()
+    args, _ = parser.parse_known_args()
+
+    listing = crd_api.list_namespaced_custom_object(
+        group="kueue.x-k8s.io",
+        version="v1beta1",
+        namespace=args.namespace,
+        plural="localqueues",
+    )
+    list_queues(listing)
+
+    listing = crd_api.list_namespaced_custom_object(
+        group="batch",
+        version="v1",
+        namespace=args.namespace,
+        plural="jobs",
+    )
+    list_jobs(listing)
+
+
+def list_jobs(listing):
+    """
+    Iterate and show job metadata.
+    """
+    if not listing:
+        print("💼️ There are no jobs.")
+        return
+
+    print("\n💼️ Jobs")
+    for job in listing["items"]:
+        jobname = job["metadata"]["name"]
+        status = (
+            "TBA" if "succeeded" not in job["status"] else job["status"]["succeeded"]
+        )
+        ready = job["status"]["ready"]
+        print(f"Found job {jobname}")
+        print(f"  Succeeded: {status}")
+        print(f"  Ready: {ready}")
+
+
+def list_queues(listing):
+    """
+    Helper function to iterate over and list queues.
+    """
+    if not listing:
+        print("⛑️  There are no queues.")
+        return
+
+    print("\n⛑️  Local Queues")
+
+    # This is listing queues
+    for q in listing["items"]:
+        print(f'Found queue {q["metadata"]["name"]}')
+        print(f"  Admitted workloads: {q['status']['admittedWorkloads']}")
+        print(f"  Pending workloads: {q['status']['pendingWorkloads']}")
+
+        # And flavors with resources
+        for f in q["status"]["flavorUsage"]:
+            print(f'  Flavor {f["name"]} has resources {f["resources"]}')
+
+
+if __name__ == "__main__":
+    main()



Test patch:
diff --git a/pkg/config/config_test.go b/pkg/config/config_test.go
index 931d31f54..8d804c8b5 100644
--- a/pkg/config/config_test.go
+++ b/pkg/config/config_test.go
@@ -189,6 +189,7 @@ health:
   livenessEndpointName: live
 metrics:
   bindAddress: :8080
+pprofBindAddress: :8082
 leaderElection:
   leaderElect: true
   resourceName: c1f6bfd2.kueue.x-k8s.io
@@ -474,6 +475,7 @@ integrations:
 				ReadinessEndpointName:      "ready",
 				LivenessEndpointName:       "live",
 				MetricsBindAddress:         configapi.DefaultMetricsBindAddress,
+				PprofBindAddress:           ":8082",
 				LeaderElection:             true,
 				LeaderElectionID:           configapi.DefaultLeaderElectionID,
 				LeaderElectionNamespace:    "namespace",