Repository: PrefectHQ/prefect

Issue:
"Task run 'xxx' received abort during orchestration" errors for large flows with DaskTaskRunner
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

Our flow is encountering the following error on a daily basis:
`Task run 'name_of_task-1234' - Task run 'task_run_id' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.`

It is not entirely clear why this is happening, but feels like it's a cloud-side issue with very large flows. We had a [similar issue previously](https://github.com/PrefectHQ/prefect/issues/8435) with DB timeouts, but that error at least a specific error message indicating the cause. In this case, it appears prefect is trying to put an already-running task into a running state (but no indication as to why it might be trying to do that).

It's happening for at least one task per production run (sometimes 2-3 tasks). It does not appear to be **directly** tied to any other error / warning log that I can see or find - I've been monitoring the k8s pods directly, with debug logs enabled, but this error just "happens" without any other related issue / error that I can see.


### Some possible indicators

**GIL Locks**
We usually get a couple of GIL warnings on the Dask workers per run. Sometimes they happen near in time to this task abort error, sometimes they happen 30mins apart (i.e. it might be unrelated). Note - our tasks don't handle any data or results at all (they're all run SQL tasks) but many of them are long-running (5-20mins). We've had these even in Prefect 1 for years and they've not been an issue, but you never know...

```shell
19:11:44.337 | INFO    | distributed.core - Event loop was unresponsive in Worker for 30.01s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
---
2023-02-18 19:05:19,995 - distributed.core - INFO - Event loop was unresponsive in Worker for 4.97s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
```

**Retyable Error Encountered**
We have also been encountering the following Retryable Error multiple times per run - this seems to be new in the last week or two, and might be related. In some runs we've seen this happen close to the Abort error, in others it's happened up to 30mins apart, so again, may or may not be related. Note - we have HTTP2 disabled as it was causing the Agent to throw errors if it was alive for >1hr.

```shell
16:11:39.099 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/6.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 191, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
```


### Setup / Flow / Infrastructure
The Flow is 3700 tasks, up to 64 tasks running in paralle at once. Flow is executed with DaskTaskRunner, on 4x ephemeral Dask Worker pods (running on EKS k8s), 16 threads each. All of the k8s pods (agent, job, dask scheduler and 4x dask workers) are alive and healthy for the duration of the flow run, i.e. there are no infrastructure issues, no pod crashes or restarts etc.


### Recent occurences
Some examples from when it's happened in the last couple of days. Timestamps are in UTC (to cross-ref with cloud-side logs)

**2023-02-19**
https://app.prefect.cloud/account/1235831b-7019-4987-b759-4341850b4c5a/workspace/85fb93b5-ea3a-4da7-8be7-aa4e13c6397d/flow-runs/flow-run/126d302e-b300-42f0-9751-3ff3f9a2b0cd
```shell
2023-02-19 19:11:46.948 | WARNING | Task run 'task-name-redacted-362' - Task run '65d18ec7-dcc1-4a8f-aa3d-bfd9ef7e989f' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.
```

**2023-02-18**
https://app.prefect.cloud/account/1235831b-7019-4987-b759-4341850b4c5a/workspace/85fb93b5-ea3a-4da7-8be7-aa4e13c6397d/flow-runs/flow-run/3691c214-485b-4b85-a863-42d1a889ef75
```shell
2023-02-18 20:05:54.511 | WARNING | Task run 'task-name-redacted-2775' - Task run 'b8f93202-bb00-423b-b091-f0e64e50f29c' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.
```


### Impact

This error leaves the affected task_run(s) in a RUNNING state and they never Complete in the backend. Any tasks downstream of these tasks enter a NotReady state, meaning a large proportion of the tasks are left not-ran, causing a production incident for us. We've had to revert back to Prefect 1 for our production runs whilst this is ongoing.



### Reproduction

```python3
"""
Tricky to reproduce this given it appears to be a transient cloud-side error. 
An MRE of our flow can be found in this issue: https://github.com/PrefectHQ/prefect/issues/8579
Specifically the flow `python sync_dask_submission.py` - though the `time.sleep()` for each task will need randomising to between 0 and 20 minutes to behave like real life.

I'm hoping the above recent occurrences will match up with some cloud-side logs which can help to cross reference what the issue is.
"""
```


### Error

```python3
Included in-line above for ease of reading
```


### Versions

```Text
Version:             2.7.11
API version:         0.8.4
Python version:      3.8.7
Git commit:          6b27b476
Built:               Thu, Feb 2, 2023 7:22 PM
OS/Arch:             darwin/x86_64
Profile:             bi-p
Server type:         cloud
```


### Additional context

Some potentially similar / related posts about a similar error message
- https://prefect-community.slack.com/archives/CL09KU1K7/p1676529027206019
- https://prefect-community.slack.com/archives/CL09KU1K7/p1676643478546479


I also saw [this issue](https://github.com/PrefectHQ/prefect/issues/7116) and [this issue](https://github.com/PrefectHQ/prefect/issues/13013) however I am 100% positive this isn't an infrastructure error, because I've watched the last 5 flow runs end to end - no pod crashes / recycles, pods are wayyy below their memory limits and remain available and happy throughout. We have dedicated EKS clusters for running prefect. So, I figure it must be related to either a transient cloud thing, or perhaps the GIL lock in the Dask workers, at a stretch..? 


Discussion:
We'll be looking into a workaround for this. Thanks for the issue!
Progress is probably best tracked in https://github.com/PrefectHQ/prefect/issues/8602
Thanks!

Quick followup - I've enabled DEBUG Dask / Distributed logs (super verbose) and have simplified our Dask infra (1 worker not 4, so worker stealing won't be a thing). I managed to observe the above reported error (abort during orchestration of task run) and to my surprise, there's no Dask funny business going on, no matter how much I scour the scheduler and worker logs - everything dask-wise is normal. No worker stealing, no worker retries, no error / warnings in them.

However, a few seconds before the abort, we got the http retry error (below) from cloud again. 

So, I don't believe a low-level Dask issue is causing this, but I do believe it's related to some kind of cloud latency / need for cloud retries e.g because the task state couldn't be updated, for example. 

Note - we were not getting these abort errors (nor the http retry errors) prior to the [SQLTimeout issue](https://github.com/PrefectHQ/prefect/issues/8435) being fixed. I'm wondering if in fixing that error, some unexpected behaviour is now causing these retry and abort errors to occur?

HTTP Retry error:
```shell
17:23:21.273 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/6.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 191, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
17:23:21.275 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/6.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 191, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
17:23:21.280 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/6.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 191, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
17:23:21.282 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/6.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 191, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
17:23:21.284 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/6.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 191, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
```

then seconds later 
```shell
17:23:28.481 | WARNING | Task run '<task_name_here>-3245' - Task run '65e88bab-a914-46f6-a7bb-41cb9cf02e78' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.
```
Hm this makes me think the order of operations is:

- Propose a transition from PENDING -> RUNNING
- Cloud API accepts the transition
- Client encounters a `ReadTimeout` and retries
- Since the first transition was accepted, this is actually a proposal of RUNNING -> RUNNING
- Cloud API denies the transition

This makes me think we should attach idempotency keys to state change requests so Cloud can just return the RUNNING state that was achieved in the previous request on client-side timeouts. It'd also be great to determine why the timeout is occurring in the first place. Maybe we could also bump the read timeout?
Ahh @madkinsz now you say that, yeah that pattern fits with my logs - there is no `Beginning Execution...` log for this one task!!

Usually for every single task we'd see 
```shell
 DEBUG   | Task run 'task_name-3306' - Beginning execution...
```
This flow run had 3790 tasks, and in my single Dask worker there are 3789 `Beginning Execution...` logs, and there's not one for the task that encountered the abort. So that fits with the task never actually starting ie it [never hit this](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/engine.py#L640), but most likely errored in the process of the state update from `Pending` -> `Running` like you say 👌 
We are also seeing this error happens many times a day:

![CleanShot 2023-02-26 at 11 25 45@2x](https://user-images.githubusercontent.com/5398977/221402533-3bdc994c-d7bb-494b-8bd3-968fbcf5a442.jpg)

Would happily provide any information that will help you debug the issue. Please contact me if you need any help.

* We previously used the Dask runner, but even after switching to the Ray runner, nothing changed.

It it is of any help, the errors occur in a flow that runs continusly inside a while loop. That is, it's not scheduled by a deployment:

![CleanShot 2023-02-26 at 11 28 46@2x](https://user-images.githubusercontent.com/5398977/221402702-930ed14c-9659-4a42-a1da-412b8baac614.jpg)



Resolving patch:
diff --git a/src/prefect/server/orchestration/core_policy.py b/src/prefect/server/orchestration/core_policy.py
--- a/src/prefect/server/orchestration/core_policy.py
+++ b/src/prefect/server/orchestration/core_policy.py
@@ -61,7 +61,6 @@ def priority():
             CacheRetrieval,
             HandleTaskTerminalStateTransitions,
             PreventRunningTasksFromStoppedFlows,
-            PreventRedundantTransitions,
             SecureTaskConcurrencySlots,  # retrieve cached states even if slots are full
             CopyScheduledTime,
             WaitForScheduledTime,


Test patch:
diff --git a/tests/server/api/test_task_runs.py b/tests/server/api/test_task_runs.py
--- a/tests/server/api/test_task_runs.py
+++ b/tests/server/api/test_task_runs.py
@@ -346,14 +346,14 @@ async def test_set_task_run_state(self, task_run, client, session):
         assert run.run_count == 1
 
     @pytest.mark.parametrize("proposed_state", ["PENDING", "RUNNING"])
-    async def test_setting_task_run_state_twice_aborts(
+    async def test_setting_task_run_state_twice_works(
         self, task_run, client, session, proposed_state
     ):
-        # A multi-agent environment may attempt to orchestrate a run more than once,
-        # this test ensures that a 2nd agent cannot re-propose a state that's already
-        # been set
+        # Task runner infrastructure may kill and restart tasks without informing
+        # Prefect. This test checks that a task can re-transition  its current state
+        # to ensure restarts occur without error.
 
-        # first ensure the parent flow run is in a running state
+        # first, ensure the parent flow run is in a running state
         await client.post(
             f"/flow_runs/{task_run.flow_run_id}/set_state",
             json=dict(state=dict(type="RUNNING")),
@@ -372,10 +372,10 @@ async def test_setting_task_run_state_twice_aborts(
             f"/task_runs/{task_run.id}/set_state",
             json=dict(state=dict(type=proposed_state, name="Test State")),
         )
-        assert response.status_code == 200
+        assert response.status_code == 201
 
         api_response = OrchestrationResult.parse_obj(response.json())
-        assert api_response.status == responses.SetStateStatus.ABORT
+        assert api_response.status == responses.SetStateStatus.ACCEPT
 
     async def test_set_task_run_ignores_client_provided_timestamp(
         self, flow_run, client
diff --git a/tests/server/orchestration/test_core_policy.py b/tests/server/orchestration/test_core_policy.py
--- a/tests/server/orchestration/test_core_policy.py
+++ b/tests/server/orchestration/test_core_policy.py
@@ -1530,6 +1530,56 @@ async def test_one_run_wont_consume_multiple_slots(
         assert duplicate_ctx.response_status == SetStateStatus.ACCEPT
         assert (await self.count_concurrency_slots(session, "a generous limit")) == 1
 
+    async def test_task_restart_does_not_consume_multiple_slots(
+        self,
+        session,
+        run_type,
+        initialize_orchestration,
+    ):
+        await self.create_concurrency_limit(session, "a generous limit", 10)
+
+        concurrency_policy = [
+            SecureTaskConcurrencySlots,
+        ]
+        # we should have no consumed slots yet
+        assert (await self.count_concurrency_slots(session, "a generous limit")) == 0
+
+        start_transition = (states.StateType.PENDING, states.StateType.RUNNING)
+
+        ctx = await initialize_orchestration(
+            session, "task", *start_transition, run_tags=["a generous limit"]
+        )
+
+        async with contextlib.AsyncExitStack() as stack:
+            for rule in concurrency_policy:
+                ctx = await stack.enter_async_context(rule(ctx, *start_transition))
+            await ctx.validate_proposed_state()
+
+        assert ctx.response_status == SetStateStatus.ACCEPT
+
+        # PENDING -> RUNNING should consume a slot
+        assert (await self.count_concurrency_slots(session, "a generous limit")) == 1
+
+        running_transition = (states.StateType.RUNNING, states.StateType.RUNNING)
+        restart_ctx = await initialize_orchestration(
+            session,
+            "task",
+            *running_transition,
+            run_override=ctx.run,
+            run_tags=["a generous limit"],
+        )
+
+        async with contextlib.AsyncExitStack() as stack:
+            for rule in concurrency_policy:
+                restart_ctx = await stack.enter_async_context(
+                    rule(restart_ctx, *running_transition)
+                )
+            await restart_ctx.validate_proposed_state()
+
+        # RUNNING -> RUNNING should not consume another slot
+        assert restart_ctx.response_status == SetStateStatus.ACCEPT
+        assert (await self.count_concurrency_slots(session, "a generous limit")) == 1
+
     async def test_concurrency_race_condition_new_tags_arent_double_counted(
         self,
         session,