Repository: PrefectHQ/prefect

Issue:
`executable not found` when not setting `command` in KubernetesRun job template
## Opened from the [Prefect Public Slack Community](https://join.slack.com/t/prefect-public/shared_invite/enQtNzE5OTU3OTQwNzc1LTQ5M2FkZmQzZjI0ODg1ZTBmOTc0ZjVjYWFjMWExZDAyYzBmYjVmMTE1NTQ1Y2IxZTllOTc4MmI3NzYxMDlhYWU)

**clemens**: Hi Prefect Team,

We have a prefect-server running on a Kubernetes cluster and just updated to `0.13.19.` . As environments are being deprecated I am trying to move the flows over to run_config setup.

I am using the Docker storage and KubernetesRun for run_config. I noticed that the container that is built with `KubernetesRun()` is configured with:

```
command: /bin/sh -c
command arguments: prefect execute flow-run
```

but when I inject a job spec file with`KubernetesRun(job_template_path="job_specs.yaml")` (as I need to to access the Kubernetes secrets that I need for the flow to run) only the `command arguments` are written by the prefect build and the `command` is left blank. That leads to the container failing with an `"executable not found"`

A workaround is to specifically include the `command` section in the job specs file `command: ["/bin/sh", "-c"]`.

That doesn’t seem to be expected behaviour, or am I missing something?

**josh**: Hey <@U01DD3HGDK8> I believe this is actually intentional (although it could probably be improved) in the <https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/kubernetes/job_template.yaml|default job template> it sets the `command` and I think that by using a custom job template it is assuming you have the command set. I’m going to open an issue on the repo to verify if that is intended behavior and if it is I think we might be able to make the experience better :slightly_smiling_face:

**josh**: <@ULVA73B9P> open “`executable not found` when not setting `command` in KubernetesRun job template”

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1607526089388000?thread_ts=1607526089.388000&cid=CL09KU1K7).




Discussion:
@jcrist Opening because I _think_ this is intended behavior and I'm wondering if we can improve the experience here. Perhaps adding a note to the KubernetesRun docs about the default job template is enough.
This isn't unintended, but it could be cleaner.

In kubernetes, the `command` overrides an image's `ENTRYPOINT`, and `args` overrides an image's `CMD`. Usually you want to let user's configure their own entrypoint for an image (allows for setup scripts in an image), and only set the cmd (`args` in k8s). We do that in prefect, but we pass in a string instead of a list of args, which is a bit odd (I left this for backwards compatibility when I was doing the run-config stuff). It's not wrong, but usually you want to pass around a list of args - this composes better with existing tooling. For example, using tini as an entrypoint in an image, you usually do

```
ENTRYPOINT ["tini", "-g", "--"]
```

and prefect would pass in args at runtime as

```
["prefect", "execute", "flow-run"]
```

As is, the user would need to set

```
ENTRYPOINT ["tini", "-g", "--", "/usr/bin/env", "bash", "-c"]
```

I think we should:

- Document the `ENTRYPOINT` requirements expected for prefect images. Whether these should expect a string (as currently written) or a list of args is up for grabs.
- Maybe switch entrypoint conventions. I'm not sure if we can do this in a non-breaking way, but it shouldn't affect most users and there's a straightforward upgrade strategy so we maybe should do it anyway. Switching would drop the need for an entrypoint (or `command` in k8s) to be set, since the provided CMD should work either way (since it's a list of args not a string).

I also think we should add `tini` to the built prefect image, and set it as the `ENTRYPOINT`. This isn't a backwards incompatible change, and help prevent zombie processes (which has been a problem in the past).

Resolving patch:
diff --git a/src/prefect/agent/kubernetes/agent.py b/src/prefect/agent/kubernetes/agent.py
--- a/src/prefect/agent/kubernetes/agent.py
+++ b/src/prefect/agent/kubernetes/agent.py
@@ -561,6 +561,9 @@ def generate_job_spec_from_run_config(
         if image_pull_secrets is not None:
             pod_template["imagePullSecrets"] = [{"name": s} for s in image_pull_secrets]
 
+        # Default restartPolicy to Never
+        _get_or_create(job, "spec.template.spec.restartPolicy", "Never")
+
         # Get the first container, which is used for the prefect job
         containers = _get_or_create(job, "spec.template.spec.containers", [])
         if not containers:
@@ -571,7 +574,7 @@ def generate_job_spec_from_run_config(
         container["image"] = image = get_flow_image(flow_run)
 
         # Set flow run command
-        container["args"] = [get_flow_run_command(flow_run)]
+        container["args"] = get_flow_run_command(flow_run).split()
 
         # Populate environment variables from the following sources,
         # with precedence:


Test patch:
diff --git a/tests/agent/test_k8s_agent.py b/tests/agent/test_k8s_agent.py
--- a/tests/agent/test_k8s_agent.py
+++ b/tests/agent/test_k8s_agent.py
@@ -1071,7 +1071,7 @@ def read_default_template(self):
         with open(DEFAULT_JOB_TEMPLATE_PATH) as f:
             return yaml.safe_load(f)
 
-    def build_flow_run(self, config, storage=None):
+    def build_flow_run(self, config, storage=None, core_version="0.13.0"):
         if storage is None:
             storage = Local()
         return GraphQLResult(
@@ -1081,7 +1081,7 @@ def build_flow_run(self, config, storage=None):
                         "storage": storage.serialize(),
                         "run_config": None if config is None else config.serialize(),
                         "id": "new_id",
-                        "core_version": "0.13.0",
+                        "core_version": core_version,
                     }
                 ),
                 "id": "id",
@@ -1163,6 +1163,7 @@ def test_generate_job_spec_metadata(self, tmpdir):
         assert job["spec"]["template"]["metadata"]["labels"] == dict(
             POD_LABEL="VALUE2", **labels
         )
+        assert job["spec"]["template"]["spec"]["restartPolicy"] == "Never"
 
     @pytest.mark.parametrize(
         "run_config, storage, expected",
@@ -1183,6 +1184,19 @@ def test_generate_job_spec_image(self, run_config, storage, expected):
         image = job["spec"]["template"]["spec"]["containers"][0]["image"]
         assert image == expected
 
+    @pytest.mark.parametrize(
+        "core_version, expected",
+        [
+            ("0.12.0", "prefect execute cloud-flow"),
+            ("0.14.0", "prefect execute flow-run"),
+        ],
+    )
+    def test_generate_job_spec_container_args(self, core_version, expected):
+        flow_run = self.build_flow_run(KubernetesRun(), core_version=core_version)
+        job = self.agent.generate_job_spec(flow_run)
+        args = job["spec"]["template"]["spec"]["containers"][0]["args"]
+        assert args == expected.split()
+
     def test_generate_job_spec_environment_variables(self, tmpdir):
         """Check that environment variables are set in precedence order