Repository: PrefectHQ/prefect

Issue:
Agents Can't Fetch Work From `default` Queue of Work Pools Except `default-agent-pool`
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

Prefect agents are not able to fetch flow runs from the `default` queue of a new work pool (by new I mean any other pool than `default-agent-pool`). The `default` queue of the new pool is displayed as `unhealthy` in the Prefect UI.

### Reproduction

Start the server:

```bash
PREFECT_SERVER_API_HOST=0.0.0.0
PREFECT_SERVER_API_PORT=4200
prefect server start --host $PREFECT_SERVER_API_HOST --port $PREFECT_SERVER_API_PORT
```

Create the work pool using Prefect CLI

```bash
WORK_POOL_NAME=test-pool
WORK_QUEUE_NAME=default
prefect work-pool create $WORK_POOL_NAME
prefect agent start --pool $WORK_POOL_NAME --work-queue $WORK_QUEUE_NAME
```

OR create the work pool using REST API

```bash
WORK_POOL_NAME=test-pool
WORK_QUEUE_NAME=default
curl -s -o /dev/null \
  -w "%{http_code}" \
  -X POST \
  "$PREFECT_API_URL/work_pools/" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "'$WORK_POOL_NAME'",
  "description": "",
  "type": "prefect-agent",
  "base_job_template": {},
  "is_paused": false,
  "concurrency_limit": 0
}'
prefect agent start --pool $WORK_POOL_NAME --work-queue $WORK_QUEUE_NAME
```


### Error

No errors shown in the server/agent logs.


### Versions

```Text
Version:             2.10.5
API version:         0.8.4
Python version:      3.9.13
Git commit:          d48dfbbc
Built:               Thu, Apr 20, 2023 4:06 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server (PostgreSQL 15.2)
```


### Additional context

Prefect server and agents are different containers in a Docker Swarm.


Discussion:
Thanks for the write up @batuhanfaik! I'll work on reproducing this issue and investigating a root cause.
Hey @batuhanfaik, I haven't been able to reproduce this issue with the instructions that you provided. Could you share the steps you used to create a deployment that schedules flow runs in the created work pool?
Hey @desertaxle, thanks for looking into it! Here you go!

Through Python:

```python
import time
from prefect import flow, get_run_logger
from prefect.deployments import run_deployment
from prefect.filesystems import RemoteFileSystem
from prefect.deployments import Deployment

@flow(name='My-Test-Flow')
def my_test_flow():
    logger = get_run_logger()
    logger.info('Hi!')
    time.sleep(2)
    logger.info('Bye!')


if __name__ == "__main__":
    my_test_flow_deployment = Deployment.build_from_flow(
            name='my-test-flow',
            flow=my_test_flow,
            storage=RemoteFileSystem.load('my-block'),
            work_pool_name='test-pool',
            work_queue_name='default',
            parameters={},
        )
    my_test_flow_deployment.apply()

    run_deployment(
        name='My-Test-Flow/my-test-flow',
        timeout=0,
        flow_run_name='my-test-flow-run',
    )
```

or after deployment, through Prefect UI.

Edit: maybe worth mentioning that my deployments are stored in a Minio S3
Maybe related: #9433

Resolving patch:
diff --git a/src/prefect/server/models/work_queues.py b/src/prefect/server/models/work_queues.py
--- a/src/prefect/server/models/work_queues.py
+++ b/src/prefect/server/models/work_queues.py
@@ -392,9 +392,9 @@ async def read_work_queue_status(
         session=session,
         flow_run_filter=schemas.filters.FlowRunFilter(
             state=schemas.filters.FlowRunFilterState(name={"any_": ["Late"]}),
-            work_queue_name=schemas.filters.FlowRunFilterWorkQueueName(
-                any_=[work_queue.name]
-            ),
+        ),
+        work_queue_filter=schemas.filters.WorkQueueFilter(
+            id=schemas.filters.WorkQueueFilterId(any_=[work_queue_id])
         ),
     )
 


Test patch:
diff --git a/tests/server/api/test_work_queues.py b/tests/server/api/test_work_queues.py
--- a/tests/server/api/test_work_queues.py
+++ b/tests/server/api/test_work_queues.py
@@ -438,6 +438,29 @@ async def recently_polled_work_queue(self, session, work_pool):
         await session.commit()
         return work_queue
 
+    @pytest.fixture
+    async def recently_pool_work_queue_in_different_work_pool(self, session):
+        work_pool = await models.workers.create_work_pool(
+            session=session,
+            work_pool=schemas.actions.WorkPoolCreate(
+                name="another-work-pool",
+                description="All about my work pool",
+                type="test",
+            ),
+        )
+        work_queue = await models.work_queues.create_work_queue(
+            session=session,
+            work_queue=schemas.core.WorkQueue(
+                name="wq-1",
+                description="All about my work queue",
+                last_polled=pendulum.now("UTC"),
+                work_pool_id=work_pool.id,
+                priority=1,
+            ),
+        )
+        await session.commit()
+        return work_queue
+
     @pytest.fixture
     async def not_recently_polled_work_queue(self, session, work_pool):
         work_queue = await models.work_queues.create_work_queue(
@@ -472,7 +495,7 @@ async def work_queue_with_late_runs(self, session, flow, work_pool):
                 state=schemas.states.Late(
                     scheduled_time=pendulum.now().subtract(minutes=60)
                 ),
-                work_queue_name=work_queue.name,
+                work_queue_id=work_queue.id,
             ),
         )
         await session.commit()
@@ -524,6 +547,34 @@ async def test_read_work_queue_status_unhealthy_due_to_late_runs(
         assert parsed_response.late_runs_count == 1
         assert parsed_response.last_polled == work_queue_with_late_runs.last_polled
 
+    async def test_read_work_queue_returns_correct_status_when_work_queues_share_name(
+        self,
+        client,
+        work_queue_with_late_runs,
+        recently_pool_work_queue_in_different_work_pool,
+    ):
+        healthy_response = await client.get(
+            f"/work_queues/{recently_pool_work_queue_in_different_work_pool.id}/status"
+        )
+
+        assert healthy_response.status_code == status.HTTP_200_OK
+
+        parsed_healthy_response = pydantic.parse_obj_as(
+            schemas.core.WorkQueueStatusDetail, healthy_response.json()
+        )
+        assert parsed_healthy_response.healthy is True
+
+        unhealthy_response = await client.get(
+            f"/work_queues/{work_queue_with_late_runs.id}/status"
+        )
+
+        assert unhealthy_response.status_code == status.HTTP_200_OK
+
+        parsed_unhealthy_response = pydantic.parse_obj_as(
+            schemas.core.WorkQueueStatusDetail, unhealthy_response.json()
+        )
+        assert parsed_unhealthy_response.healthy is False
+
     async def test_read_work_queue_status_returns_404_if_does_not_exist(self, client):
         response = await client.get(f"/work_queues/{uuid4()}/status")
         assert response.status_code == status.HTTP_404_NOT_FOUND