Repository: PrefectHQ/prefect

Issue:
Add an official Kubernetes agent template
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar request and didn't find it.
- [X] I searched the Prefect documentation for this feature.

### Prefect Version

2.x

### Describe the current behavior

Currently, `prefect kubernetes manifest` only provides manifest for orion and flow-run-job. 

### Describe the proposed behavior

It would be great to have an official template for Cloud agents deployable to Kubernetes.

```
prefect kubernetes manifest agent
```

Example template we could use:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent
  namespace: default
spec:
  selector:
    matchLabels:
      app: agent
  replicas: 1
  template:
    metadata:
      labels:
        app: agent
    spec:
      containers:
        - name: agent
          image: prefecthq/prefect:2-python3.10
          command: ["prefect", "agent", "start", "-q", "default"]
          imagePullPolicy: "IfNotPresent"
          env:
            - name: PREFECT_API_URL
              valueFrom:
                secretKeyRef:
                  name: your_k8s_secret_name
                  key: api-url
            - name: PREFECT_API_KEY
              valueFrom:
                secretKeyRef:
                  name: your_k8s_secret_name
                  key: api-key
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: infra-blocks-role
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: infra-blocks-role-binding
  namespace: default
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default
roleRef:
  kind: Role
  name: infra-blocks
  apiGroup: rbac.authorization.k8s.io
```



### Example Use

_No response_

### Additional context

_No response_


Discussion:
I agree!

The current manifest output by `prefect kubernetes manifest orion` includes both the orion server _and an agent container as a "sidecar"_.  As a user and cluster operator, I find this surprising:

```
$ prefect kubernetes manifest orion
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orion
spec:
  selector:
    matchLabels:
      app: orion
  replicas: 1  # We're using SQLite, so we should only run 1 pod
  template:
    metadata:
      labels:
        app: orion
    spec:
      containers:
      - name: api
        image: prefecthq/prefect:dev-python3.10
        command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 4200
      - name: agent <---- here's the agent, as a second container in the pod
        image: prefecthq/prefect:dev-python3.10
        command: ["prefect", "agent", "start", "-q", "kubernetes"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: http://orion:4200/api
```

What I'd expect is that you can ask for a complete manifest for the orion deployment, and another complete manifest for an agent.  Additionally, it would be great to add a few more options, like the namespace and API URL to use.

I think a common use case (this is what I do in my cluster) would be to run a single orion server in its own namespace, and then several independent agents in other namespaces that listen to work queues from the orion instance and create `Job`s in their own namespaces.

Example for a cluster where there are agents handling different applications in different namespaces, coordinating with a single orion server in the cluster:

```
$ prefect kubernetes manifest orion --namespace the-orion-namespace
$ prefect kubernetes manifest agent --namespace app-1 --api-url http://orion.the-orion-namespace:4200 --queue app-1-flows
$ prefect kubernetes manifest agent --namespace app-2 --api-url http://orion.the-orion-namespace:4200 --queue app-2-flows
```

Example for a Cloud customer running agents for two different apps in different namespaces:

```
$ prefect kubernetes manifest agent --namespace app-1 --api-url https://api.prefect.cloud/api/accounts/.... --queue app-1-flows
$ prefect kubernetes manifest agent --namespace app-2 --api-url https://api.prefect.cloud/api/accounts/.... --queue app-2-flows
```
👍 this makes sense to me, especially splitting it out of the existing orion deployment. I'd stick with `--api-url` instead of `--orion-api` to match our settings though.

Resolving patch:
diff --git a/src/prefect/cli/kubernetes.py b/src/prefect/cli/kubernetes.py
--- a/src/prefect/cli/kubernetes.py
+++ b/src/prefect/cli/kubernetes.py
@@ -3,6 +3,7 @@
 """
 from string import Template
 
+import typer
 import yaml
 
 import prefect
@@ -10,7 +11,7 @@
 from prefect.cli.root import app
 from prefect.docker import get_prefect_image_name
 from prefect.infrastructure import KubernetesJob
-from prefect.settings import PREFECT_LOGGING_SERVER_LEVEL
+from prefect.settings import PREFECT_API_URL, PREFECT_LOGGING_SERVER_LEVEL
 
 kubernetes_app = PrefectTyper(
     name="kubernetes",
@@ -27,7 +28,18 @@
 
 @manifest_app.command("orion")
 def manifest_orion(
-    image_tag: str = None,
+    image_tag: str = typer.Option(
+        get_prefect_image_name(),
+        "-i",
+        "--image-tag",
+        help="The tag of a Docker image to use for the Orion.",
+    ),
+    namespace: str = typer.Option(
+        "default",
+        "-n",
+        "--namespace",
+        help="A Kubernetes namespace to create Orion in.",
+    ),
     log_level: str = SettingsOption(PREFECT_LOGGING_SERVER_LEVEL),
 ):
     """
@@ -38,17 +50,65 @@ def manifest_orion(
     """
 
     template = Template(
-        (prefect.__module_path__ / "cli" / "templates" / "kubernetes.yaml").read_text()
+        (
+            prefect.__module_path__ / "cli" / "templates" / "kubernetes-orion.yaml"
+        ).read_text()
     )
     manifest = template.substitute(
         {
-            "image_name": image_tag or get_prefect_image_name(),
+            "image_name": image_tag,
+            "namespace": namespace,
             "log_level": log_level,
         }
     )
     print(manifest)
 
 
+@manifest_app.command("agent")
+def manifest_agent(
+    api_url: str = SettingsOption(PREFECT_API_URL),
+    image_tag: str = typer.Option(
+        get_prefect_image_name(),
+        "-i",
+        "--image-tag",
+        help="The tag of a Docker image to use for the Agent.",
+    ),
+    namespace: str = typer.Option(
+        "default",
+        "-n",
+        "--namespace",
+        help="A Kubernetes namespace to create agent in.",
+    ),
+    work_queue: str = typer.Option(
+        "kubernetes",
+        "-q",
+        "--work-queue",
+        help="A work queue name for the agent to pull from.",
+    ),
+):
+    """
+    Generates a manifest for deploying Agent on Kubernetes.
+
+    Example:
+        $ prefect kubernetes manifest agent | kubectl apply -f -
+    """
+
+    template = Template(
+        (
+            prefect.__module_path__ / "cli" / "templates" / "kubernetes-agent.yaml"
+        ).read_text()
+    )
+    manifest = template.substitute(
+        {
+            "api_url": api_url,
+            "image_name": image_tag,
+            "namespace": namespace,
+            "work_queue": work_queue,
+        }
+    )
+    print(manifest)
+
+
 @manifest_app.command("flow-run-job")
 async def manifest_flow_run_job():
     """


Test patch:
diff --git a/tests/cli/test_deployment_kubernetes.py b/tests/cli/test_deployment_kubernetes.py
deleted file mode 100644
--- a/tests/cli/test_deployment_kubernetes.py
+++ /dev/null
@@ -1,41 +0,0 @@
-import yaml
-
-from prefect.infrastructure import KubernetesJob
-from prefect.testing.cli import invoke_and_assert
-
-
-def test_printing_the_orion_manifest():
-    """`prefect kubernetes manifest orion` should print a valid YAML file
-    representing a basic Orion deployment to a cluster"""
-    result = invoke_and_assert(
-        ["kubernetes", "manifest", "orion"],
-        expected_output_contains="kind: Deployment",
-    )
-    manifests = yaml.load_all(result.stdout, yaml.SafeLoader)
-
-    # Spot-check a few things.  This test is mostly just confirming that the output
-    # looks roughly like a set of Kubernetes manifests in YAML, not that this is a
-    # valid and working Orion deployment
-    assert manifests
-
-    deployment = next(m for m in manifests if m["kind"] == "Deployment")
-    assert deployment["metadata"]["name"] == "orion"
-
-    orion_container = deployment["spec"]["template"]["spec"]["containers"][0]
-    assert orion_container["command"][0:3] == ["prefect", "orion", "start"]
-
-
-def test_printing_the_job_base_manifest():
-    """`prefect kubernetes manifest flow-run-job` should print a valid YAML file
-    representing the minimum starting point for a KubernetesJob"""
-    result = invoke_and_assert(
-        ["kubernetes", "manifest", "flow-run-job"],
-        expected_output_contains="kind: Job",
-    )
-
-    # check for the presence of helpful comments
-    assert "# the first container is required" in result.stdout
-
-    parsed = yaml.load(result.stdout, yaml.SafeLoader)
-
-    assert parsed == KubernetesJob.base_job_manifest()
diff --git a/tests/cli/test_kubernetes_manifest.py b/tests/cli/test_kubernetes_manifest.py
--- a/tests/cli/test_kubernetes_manifest.py
+++ b/tests/cli/test_kubernetes_manifest.py
@@ -1,10 +1,12 @@
 import yaml
 
+from prefect.docker import get_prefect_image_name
 from prefect.infrastructure.kubernetes import KubernetesJob
+from prefect.settings import PREFECT_API_URL, PREFECT_LOGGING_SERVER_LEVEL
 from prefect.testing.cli import invoke_and_assert
 
 
-def test_printing_the_orion_manifest():
+def test_printing_the_orion_manifest_with_no_args():
     """`prefect kubernetes manifest orion` should print a valid YAML file
     representing a basic Orion deployment to a cluster"""
     result = invoke_and_assert(
@@ -13,16 +15,142 @@ def test_printing_the_orion_manifest():
     )
     manifests = yaml.load_all(result.stdout, yaml.SafeLoader)
 
-    # Spot-check a few things.  This test is mostly just confirming that the output
+    # Spot-check a few things. This test is mostly just confirming that the output
     # looks roughly like a set of Kubernetes manifests in YAML, not that this is a
-    # valid and working Orion deployment
+    # valid and working Orion deployment.
+    assert manifests
+
+    for manifest in manifests:
+        assert manifest["metadata"]["namespace"] == "default"
+
+        if manifest["kind"] == "Deployment":
+            assert manifest["metadata"]["name"] == "prefect-orion"
+            assert len(manifest["spec"]["template"]["spec"]["containers"]) == 1
+
+            orion_container = manifest["spec"]["template"]["spec"]["containers"][0]
+            assert orion_container["image"] == get_prefect_image_name()
+            assert orion_container["command"][0:3] == ["prefect", "orion", "start"]
+            assert orion_container["command"][0:3] == ["prefect", "orion", "start"]
+            assert orion_container["command"][5:] == [
+                "--log-level",
+                str(PREFECT_LOGGING_SERVER_LEVEL.value()),
+            ]
+
+
+def test_printing_the_orion_manifest_with_image_tag_and_log_level():
+    result = invoke_and_assert(
+        [
+            "kubernetes",
+            "manifest",
+            "orion",
+            "-i",
+            "test_image_tag",
+            "--log-level",
+            "test_log_level",
+        ],
+        expected_output_contains="kind: Deployment",
+    )
+    manifests = yaml.load_all(result.stdout, yaml.SafeLoader)
+    assert manifests
+
+    manifests = yaml.load_all(result.stdout, yaml.SafeLoader)
     assert manifests
 
     deployment = next(m for m in manifests if m["kind"] == "Deployment")
-    assert deployment["metadata"]["name"] == "orion"
+    assert deployment["metadata"]["name"] == "prefect-orion"
+    assert len(deployment["spec"]["template"]["spec"]["containers"]) == 1
 
     orion_container = deployment["spec"]["template"]["spec"]["containers"][0]
-    assert orion_container["command"][0:3] == ["prefect", "orion", "start"]
+    assert orion_container["image"] == "test_image_tag"
+    assert orion_container["command"][5:] == ["--log-level", "test_log_level"]
+
+
+def test_printing_the_orion_manifest_with_namespace():
+    result = invoke_and_assert(
+        ["kubernetes", "manifest", "orion", "-n", "test_namespace"],
+        expected_output_contains="kind: Deployment",
+    )
+    manifests = yaml.load_all(result.stdout, yaml.SafeLoader)
+    assert manifests
+
+    for manifest in manifests:
+        assert manifest["metadata"]["namespace"] == "test_namespace"
+
+
+def test_printing_the_agent_manifest_with_no_args():
+    """`prefect kubernetes manifest agent` should print a valid YAML file
+    representing a basic agent deployment to a cluster"""
+    result = invoke_and_assert(
+        ["kubernetes", "manifest", "agent"],
+        expected_output_contains="kind: Deployment",
+    )
+    manifests = yaml.load_all(result.stdout, yaml.SafeLoader)
+
+    # Spot-check a few things. This test is mostly just confirming that the output
+    # looks roughly like a set of Kubernetes manifests in YAML.
+    assert manifests
+
+    for manifest in manifests:
+        assert manifest["metadata"]["namespace"] == "default"
+
+        if manifest["kind"] == "Deployment":
+            assert manifest["metadata"]["name"] == "prefect-agent"
+            assert len(manifest["spec"]["template"]["spec"]["containers"]) == 1
+
+            agent_container = manifest["spec"]["template"]["spec"]["containers"][0]
+            assert agent_container["image"] == get_prefect_image_name()
+            assert agent_container["command"] == [
+                "prefect",
+                "agent",
+                "start",
+                "-q",
+                "kubernetes",
+            ]
+            assert len(agent_container["env"]) == 1
+            assert agent_container["env"][0]["name"] == "PREFECT_API_URL"
+            assert agent_container["env"][0]["value"] == str(PREFECT_API_URL.value())
+
+
+def test_printing_the_agent_manifest_with_api_url_image_tag_and_work_queue():
+    result = invoke_and_assert(
+        [
+            "kubernetes",
+            "manifest",
+            "agent",
+            "--api-url",
+            "test_api_url",
+            "-i",
+            "test_image_tag",
+            "-q",
+            "test_work_queue",
+        ],
+        expected_output_contains="kind: Deployment",
+    )
+    manifests = yaml.load_all(result.stdout, yaml.SafeLoader)
+    assert manifests
+
+    deployment = next(m for m in manifests if m["kind"] == "Deployment")
+    assert deployment["metadata"]["name"] == "prefect-agent"
+    assert len(deployment["spec"]["template"]["spec"]["containers"]) == 1
+
+    agent_container = deployment["spec"]["template"]["spec"]["containers"][0]
+    assert agent_container["image"] == "test_image_tag"
+    assert agent_container["command"][3:5] == ["-q", "test_work_queue"]
+    assert len(agent_container["env"]) == 1
+    assert agent_container["env"][0]["name"] == "PREFECT_API_URL"
+    assert agent_container["env"][0]["value"] == "test_api_url"
+
+
+def test_printing_the_agent_manifest_with_namespace():
+    result = invoke_and_assert(
+        ["kubernetes", "manifest", "agent", "-n", "test_namespace"],
+        expected_output_contains="kind: Deployment",
+    )
+    manifests = yaml.load_all(result.stdout, yaml.SafeLoader)
+    assert manifests
+
+    for manifest in manifests:
+        assert manifest["metadata"]["namespace"] == "test_namespace"
 
 
 def test_printing_the_job_base_manifest():