Repository: PrefectHQ/prefect

Issue:
Agents attempt to cancel flow runs in other work pools
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

When you run a deployment with a pool name other than `default-agent-pool`, the agent(s) that fetch from that pool and queue raise a warning and do not execute the flow run. The assigned queue of that work pool is then marked as unhealthy.

### Reproduction

Create a work pool using the REST API or the UI because CLI raises an error during creation of the work pool

```bash
prefect_api_url=$PREFECT_API_URL
work_pool_name=test-pool
curl -s -o /dev/null \
  -w "%{http_code}" \
  -X POST \
  "$prefect_api_url/work_pools/" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "'$work_pool_name'",
  "description": "",
  "type": "prefect-agent",
  "base_job_template": {},
  "is_paused": false,
  "concurrency_limit": 0
}
```

<details>
  <summary>CLI Error</summary>
  
```bash
prefect work-pool create test-pool
```
returns:
```text
Unknown work pool type 'prefect-agent'. Please choose from process.
```
  
</details>


Launch an agent to work on that pool

```bash
prefect agent start -p test-pool
```

Deploy a flow

```python
import time
from prefect import flow, get_run_logger
from prefect.filesystems import RemoteFileSystem
from prefect.deployments import Deployment

@flow
def my_test_flow():
    logger = get_run_logger()
    logger.info('Hi!')
    time.sleep(5)
    logger.info('Bye!')


if __name__ == "__main__":
    my_test_flow_deployment = Deployment.build_from_flow(
        name='my-test-flow',
        flow=my_test_flow,
        storage=RemoteFileSystem.load('my-block'),
        work_pool_name='test-pool',
        work_queue_name='default',
    )
```

Run the flow using Python or CLI or UI

```bash
prefect deployment run my-test-flow/my-test-flow
```


### Error

```python3
Agent started! Looking for work from work pool 'test-pool'...
 16:56:18.476 | INFO    | prefect.agent - Found 1 flow runs awaiting cancellation.
 16:56:18.520 | INFO    | prefect.agent - Killing process 45cfa547f306:205 for flow run '41206089-c95d-409b-ac72-55999a6bcb2d'...
 16:56:18.521 | WARNING | prefect.agent - Unable to kill process 205: The process is running on a different host '45cfa547f306'. Flow run cannot be cancelled by this agent.
```


### Versions

```Text
Version:             2.10.9
API version:         0.8.4
Python version:      3.9.13
Git commit:          1655c1fa
Built:               Thu, May 11, 2023 2:29 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server
```


### Additional context

My end goal is to deploy several agents in a Docker Swarm to fetch work from this pool in order to distribute the workload.
Flow Cancelling not transitioning to Cancelled
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

When I cancel flows, they transition into cancelling. They do not transition into cancelled.

Heres a screesnhot of my flow runs filtered by cancelling.

<img width="2256" alt="image" src="https://user-images.githubusercontent.com/6748505/228398540-f307b233-206a-4b3c-9d1e-a00de2a3787c.png">

If I take the oldest flow cancelled and click through, you can see that all the task runs are cancelled.

<img width="1313" alt="image" src="https://user-images.githubusercontent.com/6748505/228398640-035e6b2c-4abd-4214-a157-a60e4ea1cf1b.png">

Im not sure what exactly transitions from cancelling to cancelled, but I would expect that all task runs being cancelled would have been the kicker.


### Reproduction

```python3
I cancel any flow.
```


### Error

_No response_

### Versions

```Text
Version:             2.8.7
API version:         0.8.4
Python version:      3.9.12
Git commit:          a6d6c6fc
Built:               Thu, Mar 23, 2023 3:27 PM
```


### Additional context

_No response_


Discussion:
Hey @batuhanfaik, the error you're getting when attempting to create a work pool with a `prefect-agent` type is a bug, and I'd recommend submitting a separate issue. 

For the cancellation error, I'm having trouble parsing what the issue is. Are you saying that agents pick up flow runs from different work pools when looking for flow runs to cancel? The warning that is shown is expected behavior if the agent attempts to cancel a flow run running in a sub-process on another agent. It would not be expected if that flow run came from another work pool.
Hey @desertaxle, thanks for the quick response! Sure I'll open another issue.

No, agents pick up from their own pools. However as soon as I run a flow deployed to be run in that new pool, the flow run goes through the "Scheduled" stage and then gets stuck at "Late". When I check the logs of the agent, the aforementioned warning is printed at the same time as the flow run is set to "Late". I am not hitting cancel or disrupting the process in any way.

There's only one agent in play and it's printing this warning. There are no other agents fetching from the same work pool.
Here are the logs of the agent when I retry the flow run:

```text
 09:24:52.272 | INFO    | prefect.agent - Found 1 flow runs awaiting cancellation.
 09:24:52.274 | ERROR   | prefect.agent - Flow run '710dcd79-03fa-40e2-b823-2e96adc296eb' does not have an infrastructure pid attached. Cancellation cannot be guaranteed.
 ```
 
 ### Additional context

When the aforementioned flow is run, it's set to "Scheduled" and then "Late". When its status is set to "Late", the first warning given in the issue is printed. Then I cancel the flow run from the UI. After it's set to "Cancelled", I hit retry and the status of the flow run is set to "AwaitingRetry". Then the error log presented in this comment is printed by the agent.
Agents are attempting to cancel flow runs from other work pools, which we need to fix. I'm going to rename this issue because that's the primary issue at play here.

I'm not sure why the flow run in your example is being marked as late. Do you have any concurrency limits set up?
Yes I do, but it shouldn't affect these flows as they are neither tagged nor submitted to the same pool.

To be concise, I have flow run concurrency limit set to 24 on default-agent-pool and task concurrency set to 64 for tasks with a `limited` tag.

Edit: there were no other flow runs present during my tests.
Thanks for the issue @Samreay and for taking the time to report this.  

Can you add a bit more info about how you're canceling your runs - for example:

Are you using the cancel button in the UI? 

Are the runs ones kicked off from a deployment or a locally run flow? 




I'm using the cancel button in the UI, and the flows originate from a scheduled RRule deployment
Thanks for the extra details! 

Is the console output of the agent(s) running these flows easily accessible? If so, it would be helpful to check if the agent reports any errors after you click the cancel button. 
I'll see if I can replicate and inspect the agents logs when Im back at work tomorrow :)
@Samreay sorry for all the questions - can you also add what state those runs were in before they were cancelled?

Possibly related - https://github.com/PrefectHQ/prefect/issues/8194
We have an automation set up that is designed to cancel flows that have been running for too long, and it rarely works because the flows simply transition from being stuck in "Running" to being stuck in "Cancelling".
@cumulus-nrussell Can you please include some agent logs for the given flow run especially around the time when it gets cancelled?
> @Samreay sorry for all the questions - can you also add what state those runs were in before they were cancelled?
> 
> Possibly related - #8194

Sorry didn't have a chance to go through the logs in the agent yet, but can confirm they were Running, not scheduled.
Okay, I've just tested this one with the AWS logging up, and literally nothing looks wrong. I cancel the flow "mu2-cuello-h" (autogenerated name), and a moment later I see:

`05:22:01.508 \| INFO \| prefect.infrastructure.process - Process 'mu2-cuello-h' exited cleanly.`

in the logs. 

![image](https://user-images.githubusercontent.com/6748505/229418760-5c0abe9f-7249-4a00-a5cd-510dd1567535.png)

And the tasks cancel properly, but the flow itself stays in "cancelling", both in the flow run page and the flow page itself:

![image](https://user-images.githubusercontent.com/6748505/229418853-4677bf8d-39d1-416d-8992-0018fe1dfd59.png)



This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

Resolving patch:
diff --git a/src/prefect/agent.py b/src/prefect/agent.py
--- a/src/prefect/agent.py
+++ b/src/prefect/agent.py
@@ -21,7 +21,8 @@
     FlowRunFilterState,
     FlowRunFilterStateName,
     FlowRunFilterStateType,
-    FlowRunFilterWorkQueueName,
+    WorkPoolFilter,
+    WorkPoolFilterName,
     WorkQueueFilter,
     WorkQueueFilterName,
 )
@@ -256,20 +257,28 @@ async def check_for_cancelled_flow_runs(self):
 
         self.logger.debug("Checking for cancelled flow runs...")
 
-        work_queue_names = set()
-        async for work_queue in self.get_work_queues():
-            work_queue_names.add(work_queue.name)
+        work_queue_filter = (
+            WorkQueueFilter(name=WorkQueueFilterName(any_=list(self.work_queues)))
+            if self.work_queues
+            else None
+        )
 
+        work_pool_filter = (
+            WorkPoolFilter(name=WorkPoolFilterName(any_=[self.work_pool_name]))
+            if self.work_pool_name
+            else WorkPoolFilter(name=WorkPoolFilterName(any_=["default-agent-pool"]))
+        )
         named_cancelling_flow_runs = await self.client.read_flow_runs(
             flow_run_filter=FlowRunFilter(
                 state=FlowRunFilterState(
                     type=FlowRunFilterStateType(any_=[StateType.CANCELLED]),
                     name=FlowRunFilterStateName(any_=["Cancelling"]),
                 ),
-                work_queue_name=FlowRunFilterWorkQueueName(any_=list(work_queue_names)),
                 # Avoid duplicate cancellation calls
                 id=FlowRunFilterId(not_any_=list(self.cancelling_flow_run_ids)),
             ),
+            work_pool_filter=work_pool_filter,
+            work_queue_filter=work_queue_filter,
         )
 
         typed_cancelling_flow_runs = await self.client.read_flow_runs(
@@ -277,10 +286,11 @@ async def check_for_cancelled_flow_runs(self):
                 state=FlowRunFilterState(
                     type=FlowRunFilterStateType(any_=[StateType.CANCELLING]),
                 ),
-                work_queue_name=FlowRunFilterWorkQueueName(any_=list(work_queue_names)),
                 # Avoid duplicate cancellation calls
                 id=FlowRunFilterId(not_any_=list(self.cancelling_flow_run_ids)),
             ),
+            work_pool_filter=work_pool_filter,
+            work_queue_filter=work_queue_filter,
         )
 
         cancelling_flow_runs = named_cancelling_flow_runs + typed_cancelling_flow_runs


Test patch:
diff --git a/tests/agent/test_agent_run_cancellation.py b/tests/agent/test_agent_run_cancellation.py
--- a/tests/agent/test_agent_run_cancellation.py
+++ b/tests/agent/test_agent_run_cancellation.py
@@ -58,7 +58,9 @@ async def _create_test_deployment_from_orm(
     "cancelling_constructor", [legacy_named_cancelling_state, Cancelling]
 )
 async def test_agent_cancel_run_called_for_cancelling_run(
-    prefect_client: PrefectClient, deployment: ORMDeployment, cancelling_constructor
+    prefect_client: PrefectClient,
+    deployment: ORMDeployment,
+    cancelling_constructor,
 ):
     flow_run = await prefect_client.create_flow_run_from_deployment(
         deployment.id,
@@ -66,7 +68,9 @@ async def test_agent_cancel_run_called_for_cancelling_run(
     )
 
     async with PrefectAgent(
-        work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        work_queues=[deployment.work_queue_name],
+        work_pool_name=flow_run.work_pool_name,
+        prefetch_seconds=10,
     ) as agent:
         agent.cancel_run = AsyncMock()
         await agent.check_for_cancelled_flow_runs()
@@ -249,7 +253,9 @@ async def test_agent_cancel_run_kills_run_with_infrastructure_pid(
     await prefect_client.update_flow_run(flow_run.id, infrastructure_pid="test")
 
     async with PrefectAgent(
-        work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        work_queues=[deployment.work_queue_name],
+        work_pool_name=flow_run.work_pool_name,
+        prefetch_seconds=10,
     ) as agent:
         await agent.check_for_cancelled_flow_runs()
 
@@ -272,7 +278,9 @@ async def test_agent_cancel_run_with_missing_infrastructure_pid(
     )
 
     async with PrefectAgent(
-        work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        work_queues=[deployment.work_queue_name],
+        work_pool_name=flow_run.work_pool_name,
+        prefetch_seconds=10,
     ) as agent:
         await agent.check_for_cancelled_flow_runs()
 
@@ -308,7 +316,9 @@ async def test_agent_cancel_run_updates_state_type(
     await prefect_client.update_flow_run(flow_run.id, infrastructure_pid="test")
 
     async with PrefectAgent(
-        work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        work_queues=[deployment.work_queue_name],
+        work_pool_name=flow_run.work_pool_name,
+        prefetch_seconds=10,
     ) as agent:
         await agent.check_for_cancelled_flow_runs()
 
@@ -364,7 +374,9 @@ async def test_agent_cancel_run_with_infrastructure_not_available_during_kill(
     mock_infrastructure_kill.side_effect = InfrastructureNotAvailable("Test!")
 
     async with PrefectAgent(
-        work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        work_queues=[deployment.work_queue_name],
+        work_pool_name=flow_run.work_pool_name,
+        prefetch_seconds=10,
     ) as agent:
         await agent.check_for_cancelled_flow_runs()
         # Perform a second call to check that it is tracked locally that this agent
@@ -404,7 +416,9 @@ async def test_agent_cancel_run_with_infrastructure_not_found_during_kill(
     mock_infrastructure_kill.side_effect = InfrastructureNotFound("Test!")
 
     async with PrefectAgent(
-        work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        work_queues=[deployment.work_queue_name],
+        work_pool_name=flow_run.work_pool_name,
+        prefetch_seconds=10,
     ) as agent:
         await agent.check_for_cancelled_flow_runs()
         # Perform a second call to check that another cancellation attempt is not made
@@ -442,7 +456,9 @@ async def test_agent_cancel_run_with_unknown_error_during_kill(
     mock_infrastructure_kill.side_effect = ValueError("Oh no!")
 
     async with PrefectAgent(
-        work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        work_queues=[deployment.work_queue_name],
+        work_pool_name=flow_run.work_pool_name,
+        prefetch_seconds=10,
     ) as agent:
         await agent.check_for_cancelled_flow_runs()
         await anyio.sleep(0.5)
@@ -486,7 +502,9 @@ async def test_agent_cancel_run_without_infrastructure_support_for_kill(
     await prefect_client.update_flow_run(flow_run.id, infrastructure_pid="test")
 
     async with PrefectAgent(
-        work_queues=[deployment.work_queue_name], prefetch_seconds=10
+        work_queues=[deployment.work_queue_name],
+        work_pool_name=flow_run.work_pool_name,
+        prefetch_seconds=10,
     ) as agent:
         await agent.check_for_cancelled_flow_runs()
 
@@ -500,3 +518,191 @@ async def test_agent_cancel_run_without_infrastructure_support_for_kill(
         in caplog.text
     )
     assert "Cancellation cannot be guaranteed." in caplog.text
+
+
+@pytest.mark.parametrize(
+    "cancelling_constructor", [legacy_named_cancelling_state, Cancelling]
+)
+async def test_agent_started_in_work_pool_without_work_queue_puts_flow_run_into_cancelled_state(
+    prefect_client: PrefectClient,
+    deployment: ORMDeployment,
+    caplog,
+    cancelling_constructor,
+    work_pool,
+):
+    flow_run = await prefect_client.create_flow_run_from_deployment(
+        deployment.id,
+        state=cancelling_constructor(),
+    )
+
+    # agent with work pool but no work queue
+    async with PrefectAgent(
+        work_pool_name=work_pool.name, prefetch_seconds=10
+    ) as agent:
+        await agent.check_for_cancelled_flow_runs()
+    # make sure it is actually cancelled
+    assert "Found 1 flow runs awaiting cancellation" in caplog.text
+    post_flow_run = await prefect_client.read_flow_run(flow_run.id)
+    assert post_flow_run.state.name == "Cancelled"
+
+
+@pytest.mark.parametrize(
+    "cancelling_constructor", [legacy_named_cancelling_state, Cancelling]
+)
+async def test_agent_started_in_different_work_pool_without_work_queue_does_not_cancel_flow_run(
+    prefect_client: PrefectClient,
+    deployment: ORMDeployment,
+    caplog,
+    cancelling_constructor,
+):
+    flow_run = await prefect_client.create_flow_run_from_deployment(
+        deployment.id,
+        state=cancelling_constructor(),
+    )
+
+    # agent with work pool but no work queue
+    async with PrefectAgent(
+        work_pool_name="another-work-pool", prefetch_seconds=10
+    ) as agent:
+        await agent.check_for_cancelled_flow_runs()
+    assert "Found 1 flow runs awaiting cancellation" not in caplog.text
+    post_flow_run = await prefect_client.read_flow_run(flow_run.id)
+    assert post_flow_run.state.name == "Cancelling"
+
+
+@pytest.mark.parametrize(
+    "cancelling_constructor", [legacy_named_cancelling_state, Cancelling]
+)
+async def test_agent_started_in_different_work_pool_with_same_work_queue_name_does_not_cancel_flow_run(
+    prefect_client: PrefectClient,
+    deployment: ORMDeployment,
+    caplog,
+    cancelling_constructor,
+):
+    flow_run = await prefect_client.create_flow_run_from_deployment(
+        deployment.id,
+        state=cancelling_constructor(),
+    )
+    assert flow_run.work_pool_name == "test-work-pool"
+
+    async with PrefectAgent(
+        work_pool_name="another-work-pool",
+        work_queues=[flow_run.work_queue_name],
+        prefetch_seconds=10,
+    ) as agent:
+        await agent.check_for_cancelled_flow_runs()
+
+    assert "Found 1 flow runs awaiting cancellation" not in caplog.text
+    post_flow_run = await prefect_client.read_flow_run(flow_run.id)
+    assert post_flow_run.state.name == "Cancelling"
+
+
+@pytest.mark.parametrize(
+    "cancelling_constructor", [legacy_named_cancelling_state, Cancelling]
+)
+async def test_agent_started_in_same_work_pool_with_same_work_queue_name_cancels_flow_run(
+    prefect_client: PrefectClient,
+    deployment: ORMDeployment,
+    caplog,
+    cancelling_constructor,
+):
+    flow_run = await prefect_client.create_flow_run_from_deployment(
+        deployment.id,
+        state=cancelling_constructor(),
+    )
+    assert flow_run.work_pool_name == "test-work-pool"
+
+    async with PrefectAgent(
+        work_pool_name=flow_run.work_pool_name,
+        work_queues=[flow_run.work_queue_name],
+        prefetch_seconds=10,
+    ) as agent:
+        await agent.check_for_cancelled_flow_runs()
+
+    # make sure it is actually cancelled
+    assert "Found 1 flow runs awaiting cancellation" in caplog.text
+    post_flow_run = await prefect_client.read_flow_run(flow_run.id)
+    assert post_flow_run.state.name == "Cancelled"
+
+
+@pytest.mark.parametrize(
+    "cancelling_constructor", [legacy_named_cancelling_state, Cancelling]
+)
+async def test_agent_started_in_same_work_pool_with_different_work_queue_name_does_not_cancel_flow_run(
+    prefect_client: PrefectClient,
+    deployment: ORMDeployment,
+    caplog,
+    cancelling_constructor,
+):
+    flow_run = await prefect_client.create_flow_run_from_deployment(
+        deployment.id,
+        state=cancelling_constructor(),
+    )
+    assert flow_run.work_queue_name == "wq-1"
+
+    async with PrefectAgent(
+        work_pool_name=flow_run.work_pool_name,
+        work_queues=["wq-2"],
+        prefetch_seconds=10,
+    ) as agent:
+        await agent.check_for_cancelled_flow_runs()
+
+    assert "Found 1 flow runs awaiting cancellation" not in caplog.text
+    post_flow_run = await prefect_client.read_flow_run(flow_run.id)
+    assert post_flow_run.state.name == "Cancelling"
+
+
+@pytest.mark.parametrize(
+    "cancelling_constructor", [legacy_named_cancelling_state, Cancelling]
+)
+async def test_agent_started_without_work_pool_does_not_cancel_flow_run_in_nondefault_work_pool(
+    prefect_client: PrefectClient,
+    deployment: ORMDeployment,
+    caplog,
+    cancelling_constructor,
+):
+    flow_run = await prefect_client.create_flow_run_from_deployment(
+        deployment.id,
+        state=cancelling_constructor(),
+    )
+    assert flow_run.work_queue_name == "wq-1"
+    assert flow_run.work_pool_name == "test-work-pool"
+
+    async with PrefectAgent(
+        work_queues=["wq-1"],
+        prefetch_seconds=10,
+    ) as agent:
+        assert agent.work_pool_name is None
+        await agent.check_for_cancelled_flow_runs()
+
+    assert "Found 1 flow runs awaiting cancellation" not in caplog.text
+    post_flow_run = await prefect_client.read_flow_run(flow_run.id)
+    assert post_flow_run.state.name == "Cancelling"
+
+
+@pytest.mark.parametrize(
+    "cancelling_constructor", [legacy_named_cancelling_state, Cancelling]
+)
+async def test_agent_started_with_nondefault_work_pool_does_not_cancel_flow_run_in_default_work_pool(
+    prefect_client: PrefectClient,
+    deployment_in_default_work_pool: ORMDeployment,
+    caplog,
+    cancelling_constructor,
+):
+    flow_run = await prefect_client.create_flow_run_from_deployment(
+        deployment_in_default_work_pool.id,
+        state=cancelling_constructor(),
+    )
+    assert flow_run.work_queue_name == "wq-1"
+    assert flow_run.work_pool_name == "default-agent-pool"
+
+    async with PrefectAgent(
+        work_pool_name="test-work-pool",
+        work_queues=["wq-1"],
+        prefetch_seconds=10,
+    ) as agent:
+        await agent.check_for_cancelled_flow_runs()
+
+    assert "Found 1 flow runs awaiting cancellation" not in caplog.text
+    post_flow_run = await prefect_client.read_flow_run(flow_run.id)
+    assert post_flow_run.state.name == "Cancelling"
diff --git a/tests/fixtures/database.py b/tests/fixtures/database.py
--- a/tests/fixtures/database.py
+++ b/tests/fixtures/database.py
@@ -334,7 +334,7 @@ async def deployment(
     flow_function,
     infrastructure_document_id,
     storage_document_id,
-    work_queue_1,
+    work_queue_1,  # attached to a work pool called the work_pool fixture named "test-work-pool"
 ):
     def hello(name: str):
         pass
@@ -362,6 +362,43 @@ def hello(name: str):
     return deployment
 
 
+@pytest.fixture
+async def deployment_in_default_work_pool(
+    session,
+    flow,
+    flow_function,
+    infrastructure_document_id,
+    storage_document_id,
+    work_queue,  # not attached to a work pool
+):
+    """This will create a deployment in the default work pool called `default-agent-pool`"""
+
+    def hello(name: str):
+        pass
+
+    deployment = await models.deployments.create_deployment(
+        session=session,
+        deployment=schemas.core.Deployment(
+            name="My Deployment",
+            tags=["test"],
+            flow_id=flow.id,
+            schedule=schemas.schedules.IntervalSchedule(
+                interval=datetime.timedelta(days=1),
+                anchor_date=pendulum.datetime(2020, 1, 1),
+            ),
+            storage_document_id=storage_document_id,
+            path="./subdir",
+            entrypoint="/file.py:flow",
+            infrastructure_document_id=infrastructure_document_id,
+            work_queue_name=work_queue.name,
+            parameter_openapi_schema=parameter_schema(hello),
+            work_queue_id=work_queue.id,
+        ),
+    )
+    await session.commit()
+    return deployment
+
+
 @pytest.fixture
 async def deployment_in_non_default_work_pool(
     session,