Repository: PrefectHQ/prefect

Issue:
Make the `job_watch_timeout_seconds` optional to watch Kubernetes resources as long as required
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar request and didn't find it.
- [X] I searched the Prefect documentation for this feature.

### Prefect Version

2.x

### Describe the current behavior

User request: "The default for job watch timeout is really short at 5 seconds, shouldn't this be bumped up?"

The docstring is misleading - this timeout doesn't only apply to job startup, but also to job completion


```python
class KubernetesJob(Infrastructure):
...
    # controls the behavior of execution
    job_watch_timeout_seconds: int = Field(
        default=5,
        description="Number of seconds to watch for job creation before timing out.",
    )
    pod_watch_timeout_seconds: int = Field(
        default=60,
        description="Number of seconds to watch for pod creation before timing out.",
    )
```

### Describe the proposed behavior

There's no reason to enforce a timeout during the watch

Behavior proposed by @madkinsz: user can add it if they want to, but it shouldn’t be the default. If there’s no state change in N seconds, there are no events and it looks like things have timed out. 

### Example Use

any deployment that uses `KubernetesJob` infra block




Discussion:
Do we want to do the same with the `pod_watch_timeout_seconds`?
you got my intention right, Jeff
@zzstoatzz would you be able to look at this one?
> @zzstoatzz would you be able to look at this one?

Yes! I'll look at this today 

Resolving patch:
diff --git a/src/prefect/infrastructure/kubernetes.py b/src/prefect/infrastructure/kubernetes.py
--- a/src/prefect/infrastructure/kubernetes.py
+++ b/src/prefect/infrastructure/kubernetes.py
@@ -59,7 +59,9 @@ class KubernetesJob(Infrastructure):
             Defaults to the Prefect image.
         image_pull_policy: The Kubernetes image pull policy to use for job containers.
         job: The base manifest for the Kubernetes Job.
-        job_watch_timeout_seconds: Number of seconds to watch for job creation before timing out (default 5).
+        job_watch_timeout_seconds: Number of seconds to wait for each event emitted by the job before
+            timing out. Defaults to `None`, which means no timeout will be enforced
+            while waiting for each event in the job lifecycle.
         labels: An optional dictionary of labels to add to the job.
         name: An optional name for the job.
         namespace: An optional string signifying the Kubernetes namespace to use.
@@ -117,8 +119,12 @@ class KubernetesJob(Infrastructure):
 
     # controls the behavior of execution
     job_watch_timeout_seconds: int = Field(
-        default=5,
-        description="Number of seconds to watch for job creation before timing out.",
+        default=None,
+        description=(
+            "Number of seconds to wait for each event emitted by the job before "
+            "timing out. Defaults to `None`, which means no timeout will be enforced "
+            "while waiting for each event in the job lifecycle."
+        ),
     )
     pod_watch_timeout_seconds: int = Field(
         default=60,
@@ -519,6 +525,7 @@ def _get_job_pod(self, job_name: str) -> "V1Pod":
         """Get the first running pod for a job."""
 
         # Wait until we find a running pod for the job
+        # if `pod_watch_timeout_seconds` is None, no timeout will be enforced
         watch = kubernetes.watch.Watch()
         self.logger.debug(f"Job {job_name!r}: Starting watch for pod start...")
         last_phase = None
@@ -567,6 +574,7 @@ def _watch_job(self, job_name: str) -> int:
                     print(log.decode().rstrip())
 
         # Wait for job to complete
+        # if `job_watch_timeout_seconds` is None, no timeout will be enforced
         self.logger.debug(f"Job {job_name!r}: Starting watch for job completion")
         watch = kubernetes.watch.Watch()
         with self.get_batch_client() as batch_client:


Test patch:
diff --git a/tests/infrastructure/test_kubernetes_job.py b/tests/infrastructure/test_kubernetes_job.py
--- a/tests/infrastructure/test_kubernetes_job.py
+++ b/tests/infrastructure/test_kubernetes_job.py
@@ -748,20 +748,25 @@ def test_uses_cluster_config_if_not_in_cluster(
     mock_cluster_config.load_kube_config.assert_called_once()
 
 
+@pytest.mark.parametrize("job_timeout", [24, None])
 def test_allows_configurable_timeouts_for_pod_and_job_watches(
     mock_k8s_client,
     mock_watch,
     mock_k8s_batch_client,
+    job_timeout,
 ):
     mock_watch.stream = mock.Mock(
         side_effect=_mock_pods_stream_that_returns_running_pod
     )
-
-    KubernetesJob(
+    k8s_job_args = dict(
         command=["echo", "hello"],
         pod_watch_timeout_seconds=42,
-        job_watch_timeout_seconds=24,
-    ).run(MagicMock())
+    )
+
+    if job_timeout is not None:
+        k8s_job_args["job_watch_timeout_seconds"] = job_timeout
+
+    KubernetesJob(**k8s_job_args).run(MagicMock())
 
     mock_watch.stream.assert_has_calls(
         [
@@ -775,7 +780,7 @@ def test_allows_configurable_timeouts_for_pod_and_job_watches(
                 func=mock_k8s_batch_client.list_namespaced_job,
                 namespace=mock.ANY,
                 field_selector=mock.ANY,
-                timeout_seconds=24,
+                timeout_seconds=job_timeout,
             ),
         ]
     )