Repository: PrefectHQ/prefect

Issue:
Agent resource manager failing: 'DateTime' is not JSON serializable
I noticed that ~a week ago our agent resource manager seemed to have stopped working and old jobs were piling up in the cluster; tried redeploying off of `0.10.5` but to no avail. Looking at the logs on the resource manager container I see:
```
[prefect-agent-696bdbb897-d6z7l resource-manager] [2020-05-03 23:06:23,823] INFO - resource-manager | Found failed job prefect-job-259f5a9d in namespace default
[prefect-agent-696bdbb897-d6z7l resource-manager] [2020-05-03 23:06:24,023] INFO - resource-manager | Reporting failed pod prefect-job-259f5a9d-4nj4g in namespace default
[prefect-agent-696bdbb897-d6z7l resource-manager] [2020-05-03 23:06:24,023] ERROR - resource-manager | Object of type 'DateTime' is not JSON serializable
[prefect-agent-696bdbb897-d6z7l resource-manager] Traceback (most recent call last):
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/site-packages/prefect/agent/kubernetes/resource_manager.py", line 61, in start
[prefect-agent-696bdbb897-d6z7l resource-manager]     self.clean_resources()
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/site-packages/prefect/agent/kubernetes/resource_manager.py", line 94, in clean_resources
[prefect-agent-696bdbb897-d6z7l resource-manager]     self.report_failed_job(identifier=identifier)
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/site-packages/prefect/agent/kubernetes/resource_manager.py", line 238, in report_failed_job
[prefect-agent-696bdbb897-d6z7l resource-manager]     self.report_failed_pod(pod)
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/site-packages/prefect/agent/kubernetes/resource_manager.py", line 278, in report_failed_pod
[prefect-agent-696bdbb897-d6z7l resource-manager]     info={},
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1241, in write_run_logs
[prefect-agent-696bdbb897-d6z7l resource-manager]     mutation, variables=dict(input=dict(logs=logs))
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 217, in graphql
[prefect-agent-696bdbb897-d6z7l resource-manager]     params=dict(query=parse_graphql(query), variables=json.dumps(variables)),
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/json/__init__.py", line 231, in dumps
[prefect-agent-696bdbb897-d6z7l resource-manager]     return _default_encoder.encode(obj)
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode
[prefect-agent-696bdbb897-d6z7l resource-manager]     chunks = self.iterencode(o, _one_shot=True)
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode
[prefect-agent-696bdbb897-d6z7l resource-manager]     return _iterencode(o, 0)
[prefect-agent-696bdbb897-d6z7l resource-manager]   File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default
[prefect-agent-696bdbb897-d6z7l resource-manager]     o.__class__.__name__)
[prefect-agent-696bdbb897-d6z7l resource-manager] TypeError: Object of type 'DateTime' is not JSON serializable
```
Any recent changes (but not necessarily around a week ago since that's just when I happened to redeploy our agent) that might have caused this? cc @joshmeek @cicdw 


Discussion:
It looks like I broke this in [this commit](https://github.com/PrefectHQ/prefect/commit/8d38788fcafb5dc73fc58f566a2c1e8749b0990f).

Thanks for raising this @bnaul - will fix right now!

Resolving patch:
diff --git a/src/prefect/agent/kubernetes/resource_manager.py b/src/prefect/agent/kubernetes/resource_manager.py
--- a/src/prefect/agent/kubernetes/resource_manager.py
+++ b/src/prefect/agent/kubernetes/resource_manager.py
@@ -271,7 +271,7 @@ def report_failed_pod(self, pod: "kubernetes.client.V1Pod") -> None:
                 [
                     dict(
                         flow_run_id=pod.metadata.labels.get("flow_run_id"),
-                        timestamp=pendulum.now(),
+                        timestamp=pendulum.now("UTC").isoformat(),
                         name="resource-manager",
                         message=logs,
                         level="ERROR",
@@ -299,7 +299,7 @@ def report_unknown_pod(self, pod: "kubernetes.client.V1Pod") -> None:
                 [
                     dict(
                         flow_run_id=pod.metadata.labels.get("flow_run_id"),
-                        timestamp=pendulum.now(),
+                        timestamp=pendulum.now("UTC").isoformat(),
                         name="resource-manager",
                         message="Flow run pod {} entered an unknown state in namespace {}".format(
                             name, self.namespace
@@ -334,7 +334,7 @@ def report_pod_image_pull_error(self, pod: "kubernetes.client.V1Pod") -> None:
                         [
                             dict(
                                 flow_run_id=pod.metadata.labels.get("flow_run_id"),
-                                timestamp=pendulum.now(),
+                                timestamp=pendulum.now("UTC").isoformat(),
                                 name="resource-manager",
                                 message="Flow run image pull error for pod {} in namespace {}".format(
                                     pod.metadata.name, self.namespace


Test patch: