Repository: PrefectHQ/prefect

Issue:
Block validations cause a job run to fail but not marked as such
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

I am currently running Prefect on a Kubernetes cluster using `minikube`. Both Orion and the Agent are running *inside* the cluster.

In my code, I declare the infrastructure block programatically (making it anonymous) and then use it inside a deployment.
This deployment is successfully registered in Orion, however, when running the deployment, it gets stuck in the `Pending` state and does *not* have logs.

According to the Agent's stack-trace (see Error), my infrastructure block definition is ill-formed.
This error should be detected when creating the Block or, at least, when registering it or something that includes it. 

Either way, the agent then dies (Kubernetes restarts it) and the job remains pending, not being picked up again...

I believe this happens with other errors too as I've been struggling with this behavior with previous errors but just now noticed this "pattern".

### Reproduction

```python
1. Create a infrastructure block that is lacking required information
2. Register a flow with said infrastructure block
3. Run the flow
4. Watch the agent
```


### Error

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 212, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 141, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/agent.py", line 126, in start
    await critical_service_loop(
  File "/usr/local/lib/python3.8/site-packages/prefect/agent.py", line 286, in __aexit__
    await self.shutdown(*exc_info)
  File "/usr/local/lib/python3.8/site-packages/prefect/agent.py", line 273, in shutdown
    await self.task_group.__aexit__(*exc_info)
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.8/site-packages/prefect/agent.py", line 201, in submit_run
    infrastructure = await self.get_infrastructure(flow_run)
  File "/usr/local/lib/python3.8/site-packages/prefect/agent.py", line 190, in get_infrastructure
    infrastructure_block = Block._from_block_document(infra_document)
  File "/usr/local/lib/python3.8/site-packages/prefect/blocks/core.py", line 514, in _from_block_document
    block = block_cls.parse_obj(block_document.data)
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "/usr/local/lib/python3.8/site-packages/prefect/blocks/core.py", line 171, in __init__
    super().__init__(*args, **kwargs)
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for KubernetesJob
job
  Job is missing required attributes at the following paths: /spec/template/spec/completions, /spec/template/spec/parallelism (type=value_error)
An exception occurred.
```


### Versions

```
root@pdbt-orion-59fc689dc8-jsmqm:/app/proj# prefect version
Version:             2.3.2
API version:         0.8.0
Python version:      3.8.8
Git commit:          6e931ee9
Built:               Tue, Sep 6, 2022 12:36 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         hosted
```


### Additional context

_No response_


Discussion:
Thanks for the well written report! It looks like we need to try/catch on https://github.com/PrefectHQ/prefect/blob/1a3a3adf0bf4d83206f0367b98905a9db15cfec4/src/prefect/agent.py#L202

Perhaps just by pushing that call into `_submit_run_and_capture_errors`.

Resolving patch:
diff --git a/src/prefect/agent.py b/src/prefect/agent.py
--- a/src/prefect/agent.py
+++ b/src/prefect/agent.py
@@ -199,14 +199,20 @@ async def submit_run(self, flow_run: FlowRun) -> None:
         ready_to_submit = await self._propose_pending_state(flow_run)
 
         if ready_to_submit:
-            infrastructure = await self.get_infrastructure(flow_run)
-
-            # Wait for submission to be completed. Note that the submission function
-            # may continue to run in the background after this exits.
-            await self.task_group.start(
-                self._submit_run_and_capture_errors, flow_run, infrastructure
-            )
-            self.logger.info(f"Completed submission of flow run '{flow_run.id}'")
+            try:
+                infrastructure = await self.get_infrastructure(flow_run)
+            except Exception as exc:
+                self.logger.exception(
+                    f"Failed to get infrastructure for flow run '{flow_run.id}'."
+                )
+                await self._propose_failed_state(flow_run, exc)
+            else:
+                # Wait for submission to be completed. Note that the submission function
+                # may continue to run in the background after this exits.
+                await self.task_group.start(
+                    self._submit_run_and_capture_errors, flow_run, infrastructure
+                )
+                self.logger.info(f"Completed submission of flow run '{flow_run.id}'")
 
         self.submitting_flow_run_ids.remove(flow_run.id)
 


Test patch:
diff --git a/tests/test_agent.py b/tests/test_agent.py
--- a/tests/test_agent.py
+++ b/tests/test_agent.py
@@ -442,6 +442,35 @@ async def test_agent_submit_run_aborts_without_raising_if_server_raises_abort(
             "Server sent an abort signal: message"
         )
 
+    async def test_agent_fails_flow_if_get_infrastructure_fails(
+        self, orion_client, deployment, mock_infrastructure_run
+    ):
+        flow_run = await orion_client.create_flow_run_from_deployment(
+            deployment.id,
+            state=Scheduled(scheduled_time=pendulum.now("utc")),
+        )
+
+        async with OrionAgent(
+            work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        ) as agent:
+            agent.submitting_flow_run_ids.add(flow_run.id)
+            agent.logger = MagicMock()
+            agent.get_infrastructure = AsyncMock(side_effect=ValueError("Bad!"))
+
+            await agent.submit_run(flow_run)
+
+        mock_infrastructure_run.assert_not_called()
+        assert flow_run.id not in agent.submitting_flow_run_ids
+        agent.logger.exception.assert_called_once_with(
+            f"Failed to get infrastructure for flow run '{flow_run.id}'."
+        )
+
+        state = (await orion_client.read_flow_run(flow_run.id)).state
+        assert state.is_failed()
+        result = await orion_client.resolve_datadoc(state.data)
+        with pytest.raises(ValueError, match="Bad!"):
+            raise result
+
     async def test_agent_fails_flow_if_infrastructure_submission_fails(
         self, orion_client, deployment, mock_infrastructure_run
     ):