Repository: PrefectHQ/prefect

Issue:
Workers create multiple client contexts during normal operation
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

First reported by @madkinsz, workers create multiple client contexts during normal operation. This can be observed via the Prefect logs in debug mode:
```
12:46:46.423 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.423 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.424 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.424 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.424 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.425 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.425 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.425 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.426 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.426 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.426 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.427 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.427 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.427 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.428 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.428 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.428 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.428 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.429 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.429 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.429 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////var/folders/q9/1myqgwxn2r3fvkvf6jy3npcm0000gn/T/tmp_hunfdpd/prefect-test.db
12:46:46.463 | DEBUG   | prefect.worker.ecs.ecsworker a0d6d7bb-dc6e-4144-a373-5bb837106f6e - Entering worker context...
```

Creating multiple API clients is a drag on performance

### Reproduction

```python3
- Turn on Prefect debug logs with `prefect config set PREFECT_DEBUG_MODE=true`
- Run a worker and observe the logs
```


### Error

_No response_

### Versions

```Text
Version:             2.10.5
API version:         0.8.4
Python version:      3.10.6
Git commit:          d48dfbbc
Built:               Thu, Apr 20, 2023 4:06 PM
OS/Arch:             darwin/x86_64
Profile:             local
Server type:         server
```


### Additional context

_No response_


Discussion:

Resolving patch:
diff --git a/src/prefect/workers/base.py b/src/prefect/workers/base.py
--- a/src/prefect/workers/base.py
+++ b/src/prefect/workers/base.py
@@ -11,6 +11,7 @@
 import prefect
 from prefect._internal.compatibility.experimental import experimental
 from prefect.client.orchestration import PrefectClient, get_client
+from prefect.client.utilities import inject_client
 from prefect.engine import propose_state
 from prefect.events import Event, emit_event
 from prefect.events.related import object_as_related_resource, tags_as_related_resources
@@ -100,7 +101,10 @@ def _get_base_config_defaults(variables: dict) -> dict:
         return defaults
 
     @classmethod
-    async def from_template_and_values(cls, base_job_template: dict, values: dict):
+    @inject_client
+    async def from_template_and_values(
+        cls, base_job_template: dict, values: dict, client: "PrefectClient" = None
+    ):
         """Creates a valid worker configuration object from the provided base
         configuration and overrides.
 
@@ -113,9 +117,11 @@ async def from_template_and_values(cls, base_job_template: dict, values: dict):
             variables_schema.get("properties", {})
         )
         variables.update(values)
-        variables = await resolve_block_document_references(variables)
+        variables = await resolve_block_document_references(
+            template=variables, client=client
+        )
 
-        populated_configuration = apply_values(job_config, variables)
+        populated_configuration = apply_values(template=job_config, values=variables)
         return cls(**populated_configuration)
 
     @classmethod
@@ -798,6 +804,7 @@ async def _get_configuration(
         configuration = await self.job_configuration.from_template_and_values(
             base_job_template=self._work_pool.base_job_template,
             values=deployment.infra_overrides or {},
+            client=self._client,
         )
         configuration.prepare_for_flow_run(
             flow_run=flow_run, deployment=deployment, flow=flow


Test patch:
diff --git a/tests/workers/test_base_worker.py b/tests/workers/test_base_worker.py
--- a/tests/workers/test_base_worker.py
+++ b/tests/workers/test_base_worker.py
@@ -1,6 +1,6 @@
 import uuid
 from typing import Optional
-from unittest.mock import call
+from unittest.mock import MagicMock, call
 
 import anyio
 import pendulum
@@ -354,6 +354,50 @@ def test_flow():
     assert flow_run.state_name == "Scheduled"
 
 
+async def test_worker_creates_only_one_client_context(
+    orion_client, worker_deployment_wq1, work_pool, monkeypatch, caplog
+):
+    tracking_mock = MagicMock()
+    orig_get_client = get_client
+
+    def get_client_spy(*args, **kwargs):
+        tracking_mock(*args, **kwargs)
+        return orig_get_client(*args, **kwargs)
+
+    monkeypatch.setattr("prefect.workers.base.get_client", get_client_spy)
+
+    run_mock = AsyncMock()
+
+    @flow
+    def test_flow():
+        pass
+
+    create_run_with_deployment = (
+        lambda state: orion_client.create_flow_run_from_deployment(
+            worker_deployment_wq1.id, state=state
+        )
+    )
+    await create_run_with_deployment(
+        Scheduled(scheduled_time=pendulum.now("utc").subtract(days=1))
+    )
+    await create_run_with_deployment(
+        Scheduled(scheduled_time=pendulum.now("utc").add(seconds=5))
+    )
+    await create_run_with_deployment(
+        Scheduled(scheduled_time=pendulum.now("utc").add(seconds=5))
+    )
+
+    async with WorkerTestImpl(work_pool_name=work_pool.name) as worker:
+        worker._work_pool = work_pool
+        worker.run = run_mock  # don't run anything
+        await worker.get_and_submit_flow_runs()
+
+    assert tracking_mock.call_count == 1
+
+    # Multiple hits if worker's client is not being reused
+    assert caplog.text.count("Using ephemeral application") == 1
+
+
 async def test_base_worker_gets_job_configuration_when_syncing_with_backend_with_just_job_config(
     session, client
 ):