Repository: PrefectHQ/prefect

Issue:
HTTPSConnectionPool time out with k8 agent
## Description

I keeping getting flows failing with `HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None) `. I'm running a k8 agent and the error happens before the flow starts. What is weird is that it is that it is not happening every time but in the same frequency (pictures shows how the same flow only fails every second time.


![image](https://user-images.githubusercontent.com/44194839/92465570-0b664f00-f1cf-11ea-8a3e-17c0d1f90956.png)


## Expected Behavior
No failure as flow works fine if restarted.





## Environment
k8 agent running prefect 0.13.5


Discussion:
after a bit of googling it looks like it has something to do with the `self.batch_client.list_namespaced_job`:

http://mail-archives.apache.org/mod_mbox/airflow-commits/201911.mbox/%3CJIRA.13270065.1574417390000.198297.1574432460155@Atlassian.JIRA%3E

https://issues.apache.org/jira/browse/AIRFLOW-6040?focusedCommentId=16980186&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980186

http://mail-archives.apache.org/mod_mbox/airflow-commits/201911.mbox/%3CJIRA.13270065.1574417390000.198297.1574432460155@Atlassian.JIRA%3E
From a [conversation in the community slack](https://prefect-community.slack.com/archives/CL09KU1K7/p1599549434363200) we received the following agent logs, and confirmed the agent is running 0.13.5.

<details>
<summary>Agent logs</summary>

```
stdout	 	2020-09-07T10:01:59.254Z
stdout	|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_ 	2020-09-07T10:01:59.255Z
stdout	|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__| 	2020-09-07T10:01:59.255Z
stdout	| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __| 	2020-09-07T10:01:59.255Z
stdout	|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_ 	2020-09-07T10:01:59.255Z
stdout	 ____            __           _        _                    _ 	2020-09-07T10:01:59.255Z
stdout	[2020-09-07 10:01:59,255] INFO - agent | Agent connecting to the Prefect API at https://api.prefect.io 	2020-09-07T10:01:59.256Z
stdout	[2020-09-07 10:01:59,254] INFO - agent | Agent documentation can be found at https://docs.prefect.io/orchestration/ 	2020-09-07T10:01:59.256Z
stdout	[2020-09-07 10:01:59,254] INFO - agent | Starting KubernetesAgent with labels [] 	2020-09-07T10:01:59.256Z
stdout	                                           |___/ 	2020-09-07T10:01:59.256Z
stdout	 	2020-09-07T10:01:59.256Z
stdout	[2020-09-07 10:01:59,447] INFO - agent | Waiting for flow runs... 	2020-09-07T10:01:59.448Z
stdout	[2020-09-07 10:02:00,653] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T10:02:00.653Z
stdout	[2020-09-07 10:02:00,870] INFO - agent | Deploying flow run 8ec30ea7-38a7-4016-890f-58b8ae75b496 	2020-09-07T10:02:00.87Z
stdout	[2020-09-07 10:32:02,059] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T10:32:02.06Z
stdout	[2020-09-07 10:32:02,346] INFO - agent | Deploying flow run 4ae6b416-6618-444b-9856-586310731eca 	2020-09-07T10:32:02.347Z
stdout	[2020-09-07 11:02:04,045] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T11:02:04.046Z
stdout	[2020-09-07 11:02:04,275] INFO - agent | Deploying flow run 8dd4aad2-e411-40cf-87b6-dd6e98696d94 	2020-09-07T11:02:04.277Z
stdout	[2020-09-07 11:32:05,353] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T11:32:05.354Z
stdout	[2020-09-07 11:32:05,581] INFO - agent | Deploying flow run 509610bb-612c-47c8-b3c6-a2aeab0eb458 	2020-09-07T11:32:05.581Z
stdout	[2020-09-07 12:02:07,249] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T12:02:07.25Z
stdout	[2020-09-07 12:02:07,469] INFO - agent | Deploying flow run 6741bbc1-4a95-4ea7-ac66-12200c2ea4d9 	2020-09-07T12:02:07.47Z
stdout	[2020-09-07 12:32:10,356] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T12:32:10.356Z
stdout	[2020-09-07 12:32:10,661] INFO - agent | Deploying flow run d2645cd2-2739-486f-a122-05f7d2cb64d9 	2020-09-07T12:32:10.726Z
stdout	[2020-09-07 13:02:01,853] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T13:02:01.857Z
stdout	[2020-09-07 13:02:02,077] INFO - agent | Deploying flow run 0bd1c130-6ca6-43c2-904c-63d927fbc2eb 	2020-09-07T13:02:02.078Z
stdout	[2020-09-07 13:32:02,155] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T13:32:02.156Z
stdout	[2020-09-07 13:32:02,370] INFO - agent | Deploying flow run 3e7ac68d-0523-415b-b405-0d5a87780942 	2020-09-07T13:32:02.371Z
stdout	[2020-09-07 14:02:03,655] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T14:02:03.656Z
stdout	[2020-09-07 14:02:03,975] INFO - agent | Deploying flow run 7869cae6-03fc-4aa6-b55a-8f59d36744a4 	2020-09-07T14:02:03.976Z
stdout	[2020-09-07 14:32:07,270] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T14:32:07.271Z
stdout	[2020-09-07 14:32:07,485] INFO - agent | Deploying flow run 18d54254-a2a9-4f48-a7e6-b3ef073ffdcd 	2020-09-07T14:32:07.486Z
stdout	[2020-09-07 15:02:09,256] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T15:02:09.257Z
stdout	[2020-09-07 15:02:09,468] INFO - agent | Deploying flow run 04b3b454-02fe-4d15-a52c-d8c7f904a073 	2020-09-07T15:02:09.469Z
stdout	[2020-09-07 15:17:49,010] ERROR - agent | Logging platform error for flow run 04b3b454-02fe-4d15-a52c-d8c7f904a073 	2020-09-07T15:17:49.013Z
stdout	[2020-09-07 15:17:49,357] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-07T15:17:49.358Z
stdout	[2020-09-07 15:32:00,847] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T15:32:00.848Z
stdout	[2020-09-07 15:32:01,070] INFO - agent | Deploying flow run 18815776-ec7f-4eb0-9559-d2a42aa13ed5 	2020-09-07T15:32:01.071Z
stdout	[2020-09-07 16:02:02,550] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T16:02:02.551Z
stdout	[2020-09-07 16:02:02,774] INFO - agent | Deploying flow run b1430aa3-118d-4248-8e03-fffa27946d5d 	2020-09-07T16:02:02.774Z
stdout	[2020-09-07 16:32:04,181] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T16:32:04.181Z
stdout	[2020-09-07 16:32:04,379] INFO - agent | Deploying flow run 30edfda9-f3ad-4522-9f2c-83492f46bb70 	2020-09-07T16:32:04.379Z
stdout	[2020-09-07 17:02:07,948] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T17:02:07.949Z
stdout	[2020-09-07 17:02:08,166] INFO - agent | Deploying flow run 940a07d7-3002-4a65-9c7d-c75dbc2875ef 	2020-09-07T17:02:08.166Z
stdout	[2020-09-07 17:32:09,550] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T17:32:09.55Z
stdout	[2020-09-07 17:32:09,775] INFO - agent | Deploying flow run 25bf3070-619f-40cb-be1c-753d75269eda 	2020-09-07T17:32:09.775Z
stdout	[2020-09-07 18:02:01,452] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T18:02:01.453Z
stdout	[2020-09-07 18:02:01,680] INFO - agent | Deploying flow run ae200e42-2faf-410d-ae28-b67aa8cb122e 	2020-09-07T18:02:01.68Z
stdout	[2020-09-07 18:32:03,051] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T18:32:03.052Z
stdout	[2020-09-07 18:32:03,272] INFO - agent | Deploying flow run 2dc9c468-bf9a-4e56-ab89-8abd1cb2c981 	2020-09-07T18:32:03.272Z
stdout	[2020-09-07 18:47:42,158] ERROR - agent | Logging platform error for flow run 2dc9c468-bf9a-4e56-ab89-8abd1cb2c981 	2020-09-07T18:47:42.161Z
stdout	[2020-09-07 18:47:42,563] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-07T18:47:42.563Z
stdout	[2020-09-07 19:02:04,452] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T19:02:04.457Z
stdout	[2020-09-07 19:02:04,761] INFO - agent | Deploying flow run 84120ce6-f9e7-4fb4-b35f-fa7ae3e22342 	2020-09-07T19:02:04.762Z
stdout	[2020-09-07 19:32:07,278] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T19:32:07.278Z
stdout	[2020-09-07 19:32:07,471] INFO - agent | Deploying flow run a02d308f-81f7-40de-84ef-5e3274be6a4d 	2020-09-07T19:32:07.472Z
stdout	[2020-09-07 19:47:46,638] ERROR - agent | Logging platform error for flow run a02d308f-81f7-40de-84ef-5e3274be6a4d 	2020-09-07T19:47:46.638Z
stdout	[2020-09-07 19:47:46,965] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-07T19:47:46.966Z
stdout	[2020-09-07 20:02:00,654] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T20:02:00.655Z
stdout	[2020-09-07 20:02:00,963] INFO - agent | Deploying flow run e1dad01b-b5dd-4cab-ae6a-79dccfe041a1 	2020-09-07T20:02:00.963Z
stdout	[2020-09-07 20:32:02,103] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T20:32:02.104Z
stdout	[2020-09-07 20:32:02,362] INFO - agent | Deploying flow run aa0c20fb-fe60-4a2f-81be-080c68ed7ad1 	2020-09-07T20:32:02Z
stdout	[2020-09-07 20:47:40,878] ERROR - agent | Logging platform error for flow run aa0c20fb-fe60-4a2f-81be-080c68ed7ad1 	2020-09-07T20:47:40.878Z
stdout	[2020-09-07 20:47:41,253] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-07T20:47:41.253Z
stdout	[2020-09-07 21:02:03,356] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T21:02:03.357Z
stdout	[2020-09-07 21:02:03,588] INFO - agent | Deploying flow run c85a9b06-3419-40af-be6b-181cbcbdda5b 	2020-09-07T21:02:03.589Z
stdout	[2020-09-07 21:32:04,552] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T21:32:04.553Z
stdout	[2020-09-07 21:32:04,784] INFO - agent | Deploying flow run b6afe4e7-f7f8-4982-bf1a-cd1f9291aa86 	2020-09-07T21:32:04.785Z
stdout	[2020-09-07 21:47:43,310] ERROR - agent | Logging platform error for flow run b6afe4e7-f7f8-4982-bf1a-cd1f9291aa86 	2020-09-07T21:47:43.31Z
stdout	[2020-09-07 21:47:43,765] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-07T21:47:43.765Z
stdout	[2020-09-07 22:02:06,356] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T22:02:06.357Z
stdout	[2020-09-07 22:02:06,674] INFO - agent | Deploying flow run cdd35419-608c-44ad-a66c-537e3ef2b68f 	2020-09-07T22:02:06.674Z
stdout	[2020-09-07 22:32:07,356] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T22:32:07.356Z
stdout	[2020-09-07 22:32:07,577] INFO - agent | Deploying flow run 796adefc-9cc7-49ae-a557-5f3a7e8436e8 	2020-09-07T22:32:07.578Z
stdout	[2020-09-07 22:47:45,742] ERROR - agent | Logging platform error for flow run 796adefc-9cc7-49ae-a557-5f3a7e8436e8 	2020-09-07T22:47:45.742Z
stdout	[2020-09-07 22:47:46,074] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-07T22:47:46.075Z
stdout	[2020-09-07 23:02:09,747] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T23:02:09.747Z
stdout	[2020-09-07 23:02:09,964] INFO - agent | Deploying flow run 9ca1bb36-395a-499f-945b-3c57cdbbdadf 	2020-09-07T23:02:09.964Z
stdout	[2020-09-07 23:32:01,050] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-07T23:32:01.05Z
stdout	[2020-09-07 23:32:01,270] INFO - agent | Deploying flow run 6b21458f-9e8e-4e80-a1ee-4f0213d5e344 	2020-09-07T23:32:01.271Z
stdout	[2020-09-07 23:47:39,982] ERROR - agent | Logging platform error for flow run 6b21458f-9e8e-4e80-a1ee-4f0213d5e344 	2020-09-07T23:47:39.982Z
stdout	[2020-09-07 23:47:40,370] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-07T23:47:40.371Z
stdout	[2020-09-08 00:02:02,360] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T00:02:02.36Z
stdout	[2020-09-08 00:02:02,579] INFO - agent | Deploying flow run c3f8ebd4-dc79-4af1-9800-cfc80ca05558 	2020-09-08T00:02:02.579Z
stdout	[2020-09-08 00:32:03,788] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T00:32:03.788Z
stdout	[2020-09-08 00:32:03,970] INFO - agent | Deploying flow run 160b8c9f-6760-419d-838b-ca56ad831cbb 	2020-09-08T00:32:03.97Z
stdout	[2020-09-08 00:47:42,414] ERROR - agent | Logging platform error for flow run 160b8c9f-6760-419d-838b-ca56ad831cbb 	2020-09-08T00:47:42.414Z
stdout	[2020-09-08 00:47:42,763] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-08T00:47:42.763Z
stdout	[2020-09-08 01:02:05,367] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T01:02:05.368Z
stdout	[2020-09-08 01:02:05,574] INFO - agent | Deploying flow run 5c87fc19-5c6a-4486-b608-8401fa03acc3 	2020-09-08T01:02:05Z
stdout	[2020-09-08 01:32:01,949] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T01:32:01.949Z
stdout	[2020-09-08 01:32:02,173] INFO - agent | Deploying flow run c82d49a9-068c-46e6-943d-6396fb41a123 	2020-09-08T01:32:02.173Z
stdout	[2020-09-08 02:02:03,457] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T02:02:03.458Z
stdout	[2020-09-08 02:02:03,764] INFO - agent | Deploying flow run 08781efa-ebd3-4026-97fe-a7d4bd62a5a2 	2020-09-08T02:02:03.764Z
stdout	[2020-09-08 02:32:04,451] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T02:32:04.451Z
stdout	[2020-09-08 02:32:04,683] INFO - agent | Deploying flow run 6a36263a-863e-496a-b10f-d908d025d31e 	2020-09-08T02:32:04.684Z
stdout	[2020-09-08 03:02:06,449] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T03:02:06.455Z
stdout	[2020-09-08 03:02:06,680] INFO - agent | Deploying flow run 6973ee3f-8fef-41a8-94de-f0a18d567308 	2020-09-08T03:02:06.68Z
stdout	[2020-09-08 03:32:08,154] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T03:32:08.155Z
stdout	[2020-09-08 03:32:08,528] INFO - agent | Deploying flow run 3a67b138-8345-4f91-afc6-2f88842e492a 	2020-09-08T03:32:08.529Z
stdout	[2020-09-08 04:02:09,953] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T04:02:09.955Z
stdout	[2020-09-08 04:02:10,261] INFO - agent | Deploying flow run 3a32ab8a-c67d-495e-98d4-accae492969c 	2020-09-08T04:02:10.262Z
stdout	[2020-09-08 04:32:01,063] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T04:32:01.063Z
stdout	[2020-09-08 04:32:01,275] INFO - agent | Deploying flow run 81012274-6b0d-4580-8ef3-8dc57c348ae3 	2020-09-08T04:32:01.275Z
stdout	[2020-09-08 05:02:02,147] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T05:02:02.148Z
stdout	[2020-09-08 05:02:02,372] INFO - agent | Deploying flow run bfb51d5c-a42d-4ae0-8e13-222a4105d02c 	2020-09-08T05:02:02.373Z
stdout	[2020-09-08 05:32:03,659] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T05:32:03.66Z
stdout	[2020-09-08 05:32:03,880] INFO - agent | Deploying flow run 1e94a5ed-cd2c-49c3-b6e3-588867cb9c92 	2020-09-08T05:32:03.881Z
stdout	[2020-09-08 06:02:05,402] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T06:02:05.402Z
stdout	[2020-09-08 06:02:05,568] INFO - agent | Deploying flow run 3e34bd91-e87d-4c8c-ac39-d54e3bf07c91 	2020-09-08T06:02:05.569Z
stdout	[2020-09-08 06:32:07,250] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T06:32:07.251Z
stdout	[2020-09-08 06:32:07,477] INFO - agent | Deploying flow run 45be8ae3-0ce8-409b-8a0a-b09443cb968e 	2020-09-08T06:32:07.478Z
stdout	[2020-09-08 07:00:07,860] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T07:00:07.861Z
stdout	[2020-09-08 07:00:08,081] INFO - agent | Deploying flow run ab223b56-e938-4e9b-b494-24211ce15237 	2020-09-08T07:00:08.082Z
stdout	[2020-09-08 07:02:07,052] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T07:02:07.053Z
stdout	[2020-09-08 07:02:07,291] INFO - agent | Deploying flow run ecb3b8f3-2b3f-4de7-b28d-94519e1070a9 	2020-09-08T07:02:07.291Z
stdout	[2020-09-08 07:32:08,956] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T07:32:08.957Z
stdout	[2020-09-08 07:32:09,194] INFO - agent | Deploying flow run 763eecaa-bbc1-447b-818a-ded0f0546def 	2020-09-08T07:32:09.195Z
stdout	[2020-09-08 08:02:01,093] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T08:02:01.093Z
stdout	[2020-09-08 08:02:01,428] INFO - agent | Deploying flow run 3ba16abf-3fa2-49c0-8ac4-2f7a8784d69f 	2020-09-08T08:02:01.429Z
stdout	[2020-09-08 08:32:03,044] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T08:32:03.044Z
stdout	[2020-09-08 08:32:03,264] INFO - agent | Deploying flow run 79e0b65a-4a0e-4bac-b8d2-39904e52d560 	2020-09-08T08:32:03.265Z
stdout	[2020-09-08 08:47:41,390] ERROR - agent | Logging platform error for flow run 79e0b65a-4a0e-4bac-b8d2-39904e52d560 	2020-09-08T08:47:41.39Z
stdout	[2020-09-08 08:47:41,768] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-08T08:47:41.769Z
stdout	[2020-09-08 09:02:05,149] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T09:02:05.15Z
stdout	[2020-09-08 09:02:05,363] INFO - agent | Deploying flow run 26792ef6-04c4-4865-b693-4e5272f20dae 	2020-09-08T09:02:05.364Z
stdout	[2020-09-08 09:32:06,757] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T09:32:06.758Z
stdout	[2020-09-08 09:32:06,969] INFO - agent | Deploying flow run 54598dc5-8d14-487a-85ea-0a72bd9aaa74 	2020-09-08T09:32:06.969Z
stdout	[2020-09-08 09:47:25,607] ERROR - agent | Logging platform error for flow run 54598dc5-8d14-487a-85ea-0a72bd9aaa74 	2020-09-08T09:47:25.607Z
stdout	[2020-09-08 09:47:26,044] ERROR - agent | Error while deploying flow: ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) 	2020-09-08T09:47:26.045Z
stdout	[2020-09-08 10:02:08,457] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:02:08.458Z
stdout	[2020-09-08 10:02:08,665] INFO - agent | Deploying flow run 41441ad2-de0c-4102-9fc2-4ec3954d82b2 	2020-09-08T10:02:08.665Z
stdout	[2020-09-08 10:27:14,651] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:27:14.652Z
stdout	[2020-09-08 10:27:14,876] INFO - agent | Deploying flow run 79e0b65a-4a0e-4bac-b8d2-39904e52d560 	2020-09-08T10:27:14.877Z
stdout	[2020-09-08 10:28:22,948] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:28:22.949Z
stdout	[2020-09-08 10:28:23,196] INFO - agent | Deploying flow run 54598dc5-8d14-487a-85ea-0a72bd9aaa74 	2020-09-08T10:28:23.196Z
stdout	[2020-09-08 10:32:03,851] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:32:03.851Z
stdout	[2020-09-08 10:32:04,068] INFO - agent | Deploying flow run d9c96914-7b3c-46bf-9c55-02c785cedd99 	2020-09-08T10:32:04.069Z
stdout	[2020-09-08 10:35:04,153] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:35:04.154Z
stdout	[2020-09-08 10:35:04,377] INFO - agent | Deploying flow run 160b8c9f-6760-419d-838b-ca56ad831cbb 	2020-09-08T10:35:04.378Z
stdout	[2020-09-08 10:35:31,565] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:35:31.566Z
stdout	[2020-09-08 10:35:31,776] INFO - agent | Deploying flow run 6b21458f-9e8e-4e80-a1ee-4f0213d5e344 	2020-09-08T10:35:31.776Z
stdout	[2020-09-08 10:35:58,846] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:35:58.847Z
stdout	[2020-09-08 10:35:59,074] INFO - agent | Deploying flow run 796adefc-9cc7-49ae-a557-5f3a7e8436e8 	2020-09-08T10:35:59.074Z
stdout	[2020-09-08 10:36:26,145] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:36:26.146Z
stdout	[2020-09-08 10:36:26,376] INFO - agent | Deploying flow run b6afe4e7-f7f8-4982-bf1a-cd1f9291aa86 	2020-09-08T10:36:26.377Z
stdout	[2020-09-08 10:37:03,572] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:37:03.573Z
stdout	[2020-09-08 10:37:03,771] INFO - agent | Deploying flow run aa0c20fb-fe60-4a2f-81be-080c68ed7ad1 	2020-09-08T10:37:03.771Z
stdout	[2020-09-08 10:37:30,754] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:37:30.754Z
stdout	[2020-09-08 10:37:30,977] INFO - agent | Deploying flow run a02d308f-81f7-40de-84ef-5e3274be6a4d 	2020-09-08T10:37:30.977Z
stdout	[2020-09-08 10:37:58,153] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:37:58.154Z
stdout	[2020-09-08 10:37:58,372] INFO - agent | Deploying flow run 2dc9c468-bf9a-4e56-ab89-8abd1cb2c981 	2020-09-08T10:37:58.381Z
stdout	[2020-09-08 10:38:25,555] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T10:38:25.556Z
stdout	[2020-09-08 10:38:25,783] INFO - agent | Deploying flow run 04b3b454-02fe-4d15-a52c-d8c7f904a073 	2020-09-08T10:38:25.784Z
stdout	[2020-09-08 11:02:10,350] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T11:02:10.351Z
stdout	[2020-09-08 11:02:10,581] INFO - agent | Deploying flow run b7e86856-41b4-4f28-a37f-63efc115e02d 	2020-09-08T11:02:10.582Z
stdout	[2020-09-08 11:32:01,763] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T11:32:01.765Z
stdout	[2020-09-08 11:32:01,974] INFO - agent | Deploying flow run b935cfbe-6e32-46a3-84be-27839b53b472 	2020-09-08T11:32:01.974Z
stdout	[2020-09-08 12:02:03,449] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T12:02:03.45Z
stdout	[2020-09-08 12:02:03,759] INFO - agent | Deploying flow run ea6e8b75-941a-43eb-a1d8-fd3eb52e9c0b 	2020-09-08T12:02:03.76Z
stdout	[2020-09-08 12:32:05,559] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T12:32:05.56Z
stdout	[2020-09-08 12:32:05,782] INFO - agent | Deploying flow run 6b3435de-d7e6-4205-ab7b-6579fb8283e3 	2020-09-08T12:32:05.783Z
stdout	[2020-09-08 13:02:07,050] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T13:02:07.05Z
stdout	[2020-09-08 13:02:07,274] INFO - agent | Deploying flow run 75e25b77-e1a5-4714-a766-93bcc0d74c9c 	2020-09-08T13:02:07.275Z
stdout	[2020-09-08 13:32:08,179] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T13:32:08.179Z
stdout	[2020-09-08 13:32:08,473] INFO - agent | Deploying flow run 42ee2ee5-e5b1-422f-ad28-f310242e636b 	2020-09-08T13:32:08.474Z
stdout	[2020-09-08 13:47:47,406] ERROR - agent | Logging platform error for flow run 42ee2ee5-e5b1-422f-ad28-f310242e636b 	2020-09-08T13:47:47.406Z
stdout	[2020-09-08 13:47:47,854] ERROR - agent | Error while deploying flow: ReadTimeoutError("HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)") 	2020-09-08T13:47:47.854Z
stdout	[2020-09-08 14:02:10,454] INFO - agent | Found 1 flow run(s) to submit for execution. 	2020-09-08T14:02:10.454Z
stdout	[2020-09-08 14:02:10,690] INFO - agent | Deploying flow run 24c4e172-b5af-472c-add5-c7625b79910c 	2020-09-08T14:02:10.691Z
```

</details>

From the logs and tracing the code, it appears the timeout happens on receiving the result from `create_namespaced_job`. We'll need to refactor the deploy code here to support retries on timeouts. Note that this is a bit complicated - if the job spec contains the job name, we can debounce submitted flow runs (the k8s api will return a 409 conflict if the name already exists). If using `generateName`, we won't get an error, and may submit multiple flow runs. The prefect cloud api will handle this, but having extra k8s jobs is annoying on an infrastructure end. Needs some care in the implementation.
![image](https://user-images.githubusercontent.com/44194839/92569450-f0541780-f280-11ea-90ec-dd072ddd4b7d.png)

@jcrist is this something that is actively worked to solve? I have quite a few flows crashing due to this. :-( 
@jcrist I think I'm getting the error as I was hitting max numbers of pods on my nodes. Just upgraded the cluster and well get back to you later. However it is still a difficult error for users to debug. :-)

UPDATE: That was not the case. So I don't know why it happens then. But it seems to be happening for every second run?? 😕 
@jcrist I have tried deploying the agent such that it runs on my local machine (and then submitting jobs to the k8 cluster) instead of a sibling container and I'm not getting any errors now. Is this something you have experienced elsewhere?

Resolving patch:
diff --git a/src/prefect/agent/kubernetes/agent.py b/src/prefect/agent/kubernetes/agent.py
--- a/src/prefect/agent/kubernetes/agent.py
+++ b/src/prefect/agent/kubernetes/agent.py
@@ -1,5 +1,5 @@
 import os
-from os import path
+import time
 import uuid
 from typing import Iterable, List
 
@@ -13,8 +13,6 @@
 from prefect.utilities.agent import get_flow_image, get_flow_run_command
 from prefect.utilities.graphql import GraphQLResult
 
-AGENT_DIRECTORY = path.expanduser("~/.prefect/agent")
-
 
 class KubernetesAgent(Agent):
     """
@@ -209,22 +207,41 @@ def deploy_flow(self, flow_run: GraphQLResult) -> str:
         Returns:
             - str: Information about the deployment
         """
+        import urllib3.exceptions
+
         self.logger.info("Deploying flow run {}".format(flow_run.id))  # type: ignore
 
         image = get_flow_image(flow_run=flow_run)
 
         job_spec = self.replace_job_spec_yaml(flow_run=flow_run, image=image)
+        job_name = job_spec["metadata"]["name"]
 
-        self.logger.debug(
-            "Creating namespaced job {}".format(job_spec["metadata"]["name"])
-        )
-        job = self.batch_client.create_namespaced_job(
-            namespace=self.namespace, body=job_spec
-        )
+        self.logger.debug("Creating namespaced job {}".format(job_name))
+        attempts = 3
+        while True:
+            try:
+                self.batch_client.create_namespaced_job(
+                    namespace=self.namespace, body=job_spec
+                )
+                break
+            except self.k8s_client.rest.ApiException as exc:
+                if exc.status == 409:
+                    # object already exists, previous submission was successful
+                    # even though it errored
+                    break
+                raise
+            except urllib3.exceptions.HTTPError:
+                attempts -= 1
+                if attempts == 0:
+                    raise
+                self.logger.warning(
+                    "Error submitting job %s, retrying...", job_name, exc_info=True
+                )
+                time.sleep(1)
 
-        self.logger.debug("Job {} created".format(job.metadata.name))
+        self.logger.debug("Job {} created".format(job_name))
 
-        return "Job {}".format(job.metadata.name)
+        return "Job {}".format(job_name)
 
     def replace_job_spec_yaml(
         self, flow_run: GraphQLResult, image: str, identifier: str = None
@@ -264,7 +281,7 @@ def replace_job_spec_yaml(
         """
         identifier = identifier or str(uuid.uuid4())[:8]
         yaml_path = os.getenv(
-            "YAML_TEMPLATE", path.join(path.dirname(__file__), "job_spec.yaml")
+            "YAML_TEMPLATE", os.path.join(os.path.dirname(__file__), "job_spec.yaml")
         )
         with open(yaml_path, "r") as job_file:
             job = yaml.safe_load(job_file)
@@ -421,7 +438,7 @@ def generate_deployment_yaml(
         )
 
         with open(
-            path.join(path.dirname(__file__), "deployment.yaml"), "r"
+            os.path.join(os.path.dirname(__file__), "deployment.yaml"), "r"
         ) as deployment_file:
             deployment = yaml.safe_load(deployment_file)
 
@@ -484,7 +501,9 @@ def generate_deployment_yaml(
         # Load RBAC if specified
         rbac_yaml = []
         if rbac:
-            with open(path.join(path.dirname(__file__), "rbac.yaml"), "r") as rbac_file:
+            with open(
+                os.path.join(os.path.dirname(__file__), "rbac.yaml"), "r"
+            ) as rbac_file:
                 rbac_generator = yaml.safe_load_all(rbac_file)
 
                 for document in rbac_generator:


Test patch: