Repository: PrefectHQ/prefect

Issue:
Prefect agents fall over (likely due to OOM) and cause flow runs to be stuck in pending
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

At times when several flows are scheduled to start, we see the agent start to submit several runs to GCP Vertex. The agent appears to fall over, and once it restarts the flow runs are not resubmitted and end up stuck in pending. 

### Reproduction

```python3
N/A
```


### Error

```python3
N/A
```


### Versions

```Text
prefecthq/prefect:2.14.10-python3.9
```


### Additional context

We run the prefect agent on GCP Cloud Run, and at the most recent instance of this bug the agent had 2GB of memory. Typical utilization is very low, around 20%.

In the logs, we see several (~10, not a huge number) flow runs are submitted. e.g. 
```
15:27:38.234 - prefect.agent - Submitting flow run '0d68df5a-2a99-444b-b9b3-49169501538e'
```

At that point all logs stop, presumably because the agent fell over. A few minutes later, we see the service has restarted and we see
```
15:31:38.585 - prefect.agent - Aborted submission of flow run '0d68df5a-2a99-444b-b9b3-49169501538e'. Server sent an abort signal: This run is in a PENDING state and cannot transition to a PENDING state.
```

This causes the flow to be permanently stuck in Pending. 



We've been discussing with @kevingrismore. It seems likely that our memory usage reporting is missing a short lived spike as multiple flow runs are submitting, possibly related to instantiating many instances of the GCP client. 

It's also clear that the recover does not work as might be intended after the failure, as the flow run has been placed into a pending state without it having been fully submitted. I'd imagine it would be possible to leave the flow run as Queued until it is guaranteed to have been submitted to GCP Vertex so that this could more reliably recover. 

We've scaled up the agent to 8GB of memory for now to see if that also remediates, but that is probably just a patch as we eventually should be able to start ~100s or 1000s of flow runs scheduled with the same cron settings. 


Discussion:

Resolving patch:
diff --git a/src/integrations/prefect-gcp/prefect_gcp/aiplatform.py b/src/integrations/prefect-gcp/prefect_gcp/aiplatform.py
--- a/src/integrations/prefect-gcp/prefect_gcp/aiplatform.py
+++ b/src/integrations/prefect-gcp/prefect_gcp/aiplatform.py
@@ -54,7 +54,7 @@
     job.preview()
     ```
 """  # noqa
-
+import asyncio
 import datetime
 import re
 import shlex
@@ -68,7 +68,7 @@
 from prefect._internal.compatibility.deprecated import deprecated_class
 from prefect.exceptions import InfrastructureNotFound
 from prefect.infrastructure import Infrastructure, InfrastructureResult
-from prefect.utilities.asyncutils import run_sync_in_worker_thread, sync_compatible
+from prefect.utilities.asyncutils import sync_compatible
 
 if PYDANTIC_VERSION.startswith("2."):
     from pydantic.v1 import Field
@@ -82,7 +82,7 @@
 # if google-cloud-aiplatform is not installed
 try:
     from google.api_core.client_options import ClientOptions
-    from google.cloud.aiplatform.gapic import JobServiceClient
+    from google.cloud.aiplatform.gapic import JobServiceAsyncClient
     from google.cloud.aiplatform_v1.types.custom_job import (
         ContainerSpec,
         CustomJob,
@@ -94,7 +94,7 @@
     from google.cloud.aiplatform_v1.types.job_state import JobState
     from google.cloud.aiplatform_v1.types.machine_resources import DiskSpec, MachineSpec
     from google.protobuf.duration_pb2 import Duration
-    from tenacity import retry, stop_after_attempt, wait_fixed, wait_random
+    from tenacity import AsyncRetrying, stop_after_attempt, wait_fixed, wait_random
 except ModuleNotFoundError:
     pass
 
@@ -384,7 +384,9 @@ def _build_job_spec(self) -> "CustomJobSpec":
         return job_spec
 
     async def _create_and_begin_job(
-        self, job_spec: "CustomJobSpec", job_service_client: "JobServiceClient"
+        self,
+        job_spec: "CustomJobSpec",
+        job_service_async_client: "JobServiceAsyncClient",
     ) -> "CustomJob":
         """
         Builds a custom job and begins running it.
@@ -398,27 +400,26 @@ async def _create_and_begin_job(
 
         # run job
         self.logger.info(
-            f"{self._log_prefix}: Job {self.job_name!r} starting to run "
-            f"the command {' '.join(self.command)!r} in region "
+            f"{self._log_prefix}: Creating job {self.job_name!r} "
+            f"with command {' '.join(self.command)!r} in region "
             f"{self.region!r} using image {self.image!r}"
         )
 
         project = self.gcp_credentials.project
         resource_name = f"projects/{project}/locations/{self.region}"
 
-        retry_policy = retry(
+        async for attempt in AsyncRetrying(
             stop=stop_after_attempt(3), wait=wait_fixed(1) + wait_random(0, 3)
-        )
-
-        custom_job_run = await run_sync_in_worker_thread(
-            retry_policy(job_service_client.create_custom_job),
-            parent=resource_name,
-            custom_job=custom_job,
-        )
+        ):
+            with attempt:
+                custom_job_run = await job_service_async_client.create_custom_job(
+                    parent=resource_name,
+                    custom_job=custom_job,
+                )
 
         self.logger.info(
-            f"{self._log_prefix}: Job {self.job_name!r} has successfully started; "
-            f"the full job name is {custom_job_run.name!r}"
+            f"{self._log_prefix}: Job {self.job_name!r} created. "
+            f"The full job name is {custom_job_run.name!r}"
         )
 
         return custom_job_run
@@ -426,21 +427,26 @@ async def _create_and_begin_job(
     async def _watch_job_run(
         self,
         full_job_name: str,  # different from self.job_name
-        job_service_client: "JobServiceClient",
+        job_service_async_client: "JobServiceAsyncClient",
         current_state: "JobState",
         until_states: Tuple["JobState"],
         timeout: int = None,
     ) -> "CustomJob":
         """
         Polls job run to see if status changed.
+
+        State changes reported by the Vertex AI API may sometimes be inaccurate
+        immediately upon startup, but should eventually report a correct running
+        and then terminal state. The minimum training duration for a custom job is
+        30 seconds, so short-lived jobs may be marked as successful some time
+        after a flow run has completed.
         """
         state = JobState.JOB_STATE_UNSPECIFIED
         last_state = current_state
         t0 = time.time()
 
         while state not in until_states:
-            job_run = await run_sync_in_worker_thread(
-                job_service_client.get_custom_job,
+            job_run = await job_service_async_client.get_custom_job(
                 name=full_job_name,
             )
             state = job_run.state
@@ -451,7 +457,7 @@ async def _watch_job_run(
                     .replace("state", "state is now:")
                 )
                 # results in "New job state is now: succeeded"
-                self.logger.info(
+                self.logger.debug(
                     f"{self._log_prefix}: {self.job_name} has new {state_label}"
                 )
                 last_state = state
@@ -466,7 +472,7 @@ async def _watch_job_run(
                     f"Timed out after {elapsed_time}s while watching job for states "
                     "{until_states!r}"
                 )
-            time.sleep(self.job_watch_poll_interval)
+            await asyncio.sleep(self.job_watch_poll_interval)
 
         return job_run
 
@@ -488,26 +494,29 @@ async def run(
         )
 
         job_spec = self._build_job_spec()
-        with self.gcp_credentials.get_job_service_client(
+        job_service_async_client = self.gcp_credentials.get_job_service_async_client(
             client_options=client_options
-        ) as job_service_client:
-            job_run = await self._create_and_begin_job(job_spec, job_service_client)
-
-            if task_status:
-                task_status.started(self.job_name)
-
-            final_job_run = await self._watch_job_run(
-                full_job_name=job_run.name,
-                job_service_client=job_service_client,
-                current_state=job_run.state,
-                until_states=(
-                    JobState.JOB_STATE_SUCCEEDED,
-                    JobState.JOB_STATE_FAILED,
-                    JobState.JOB_STATE_CANCELLED,
-                    JobState.JOB_STATE_EXPIRED,
-                ),
-                timeout=self.maximum_run_time.total_seconds(),
-            )
+        )
+        job_run = await self._create_and_begin_job(
+            job_spec,
+            job_service_async_client,
+        )
+
+        if task_status:
+            task_status.started(self.job_name)
+
+        final_job_run = await self._watch_job_run(
+            full_job_name=job_run.name,
+            job_service_async_client=job_service_async_client,
+            current_state=job_run.state,
+            until_states=(
+                JobState.JOB_STATE_SUCCEEDED,
+                JobState.JOB_STATE_FAILED,
+                JobState.JOB_STATE_CANCELLED,
+                JobState.JOB_STATE_EXPIRED,
+            ),
+            timeout=self.maximum_run_time.total_seconds(),
+        )
 
         error_msg = final_job_run.error.message
         if error_msg:
@@ -534,18 +543,17 @@ async def kill(self, identifier: str, grace_seconds: int = 30) -> None:
         client_options = ClientOptions(
             api_endpoint=f"{self.region}-aiplatform.googleapis.com"
         )
-        with self.gcp_credentials.get_job_service_client(
+        job_service_async_client = self.gcp_credentials.get_job_service_async_client(
             client_options=client_options
-        ) as job_service_client:
-            await run_sync_in_worker_thread(
-                self._kill_job,
-                job_service_client=job_service_client,
-                full_job_name=identifier,
-            )
-            self.logger.info(f"Requested to cancel {identifier}...")
+        )
+        await self._kill_job(
+            job_service_async_client=job_service_async_client,
+            full_job_name=identifier,
+        )
+        self.logger.info(f"Requested to cancel {identifier}...")
 
-    def _kill_job(
-        self, job_service_client: "JobServiceClient", full_job_name: str
+    async def _kill_job(
+        self, job_service_async_client: "JobServiceAsyncClient", full_job_name: str
     ) -> None:
         """
         Thin wrapper around Job.delete, wrapping a try/except since
@@ -554,7 +562,7 @@ def _kill_job(
         """
         cancel_custom_job_request = CancelCustomJobRequest(name=full_job_name)
         try:
-            job_service_client.cancel_custom_job(
+            await job_service_async_client.cancel_custom_job(
                 request=cancel_custom_job_request,
             )
         except Exception as exc:
diff --git a/src/integrations/prefect-gcp/prefect_gcp/credentials.py b/src/integrations/prefect-gcp/prefect_gcp/credentials.py
--- a/src/integrations/prefect-gcp/prefect_gcp/credentials.py
+++ b/src/integrations/prefect-gcp/prefect_gcp/credentials.py
@@ -4,6 +4,7 @@
 import json
 from enum import Enum
 from pathlib import Path
+from threading import Lock
 from typing import Any, Dict, Optional, Union
 
 import google.auth
@@ -36,10 +37,17 @@
     pass
 
 try:
-    from google.cloud.aiplatform.gapic import JobServiceClient
+    from google.cloud.aiplatform.gapic import JobServiceAsyncClient, JobServiceClient
 except ModuleNotFoundError:
     pass
 
+try:
+    from google.api_core.client_options import ClientOptions, from_dict
+except ModuleNotFoundError:
+    pass
+
+_LOCK = Lock()
+
 
 def _raise_help_msg(key: str):
     """
@@ -79,6 +87,25 @@ class ClientType(Enum):
     AIPLATFORM = "job_service"  # vertex ai
 
 
+@functools.lru_cache(maxsize=8, typed=True)
+def _get_job_service_async_client_cached(
+    ctx, client_options: tuple
+) -> JobServiceAsyncClient:
+    """
+    Gets an authenticated Job Service async client for Vertex AI.
+
+    Returns:
+        An authenticated Job Service async client.
+    """
+    with _LOCK:
+        client_options = dict(client_options)
+        credentials = ctx.get_credentials_from_service_account()
+        job_service_client = JobServiceAsyncClient(
+            credentials=credentials, client_options=client_options
+        )
+    return job_service_client
+
+
 class GcpCredentials(CredentialsBlock):
     """
     Block used to manage authentication with GCP. Google authentication is
@@ -117,6 +144,16 @@ class GcpCredentials(CredentialsBlock):
 
     _service_account_email: Optional[str] = None
 
+    def __hash__(self):
+        return hash(
+            (
+                hash(self.service_account_file),
+                hash(frozenset(self.service_account_info.dict().items())),
+                hash(self.project),
+                hash(self._service_account_email),
+            )
+        )
+
     @root_validator
     def _provide_one_service_account_source(cls, values):
         """
@@ -412,7 +449,7 @@ def example_get_client_flow():
 
     @_raise_help_msg("aiplatform")
     def get_job_service_client(
-        self, client_options: Dict[str, Any] = None
+        self, client_options: Union[Dict[str, Any], ClientOptions] = None
     ) -> "JobServiceClient":
         """
         Gets an authenticated Job Service client for Vertex AI.
@@ -462,8 +499,67 @@ def example_get_client_flow():
             example_get_client_flow()
             ```
         """
+        if isinstance(client_options, dict):
+            client_options = from_dict(client_options)
+
         credentials = self.get_credentials_from_service_account()
-        job_service_client = JobServiceClient(
-            credentials=credentials, client_options=client_options
+        return JobServiceClient(credentials=credentials, client_options=client_options)
+
+    @_raise_help_msg("aiplatform")
+    def get_job_service_async_client(
+        self, client_options: Union[Dict[str, Any], ClientOptions] = None
+    ) -> "JobServiceAsyncClient":
+        """
+        Gets an authenticated Job Service async client for Vertex AI.
+
+        Returns:
+            An authenticated Job Service async client.
+
+        Examples:
+            Gets a GCP Job Service client from a path.
+            ```python
+            from prefect import flow
+            from prefect_gcp.credentials import GcpCredentials
+
+            @flow()
+            def example_get_client_flow():
+                service_account_file = "~/.secrets/prefect-service-account.json"
+                client = GcpCredentials(
+                    service_account_file=service_account_file
+                ).get_job_service_async_client()
+
+            example_get_client_flow()
+            ```
+
+            Gets a GCP Cloud Storage client from a dictionary.
+            ```python
+            from prefect import flow
+            from prefect_gcp.credentials import GcpCredentials
+
+            @flow()
+            def example_get_client_flow():
+                service_account_info = {
+                    "type": "service_account",
+                    "project_id": "project_id",
+                    "private_key_id": "private_key_id",
+                    "private_key": "private_key",
+                    "client_email": "client_email",
+                    "client_id": "client_id",
+                    "auth_uri": "auth_uri",
+                    "token_uri": "token_uri",
+                    "auth_provider_x509_cert_url": "auth_provider_x509_cert_url",
+                    "client_x509_cert_url": "client_x509_cert_url"
+                }
+                client = GcpCredentials(
+                    service_account_info=service_account_info
+                ).get_job_service_async_client()
+
+            example_get_client_flow()
+            ```
+        """
+        if isinstance(client_options, dict):
+            client_options = from_dict(client_options)
+
+        return _get_job_service_async_client_cached(
+            self, tuple(client_options.__dict__.items())
         )
-        return job_service_client
diff --git a/src/integrations/prefect-gcp/prefect_gcp/workers/vertex.py b/src/integrations/prefect-gcp/prefect_gcp/workers/vertex.py
--- a/src/integrations/prefect-gcp/prefect_gcp/workers/vertex.py
+++ b/src/integrations/prefect-gcp/prefect_gcp/workers/vertex.py
@@ -18,6 +18,7 @@
 Read more about configuring work pools
 [here](https://docs.prefect.io/latest/concepts/work-pools/#work-pool-overview).
 """
+import asyncio
 import datetime
 import re
 import shlex
@@ -30,7 +31,6 @@
 
 from prefect.exceptions import InfrastructureNotFound
 from prefect.logging.loggers import PrefectLogAdapter
-from prefect.utilities.asyncutils import run_sync_in_worker_thread
 from prefect.utilities.pydantic import JsonPatch
 from prefect.workers.base import (
     BaseJobConfiguration,
@@ -52,7 +52,7 @@
 # if google-cloud-aiplatform is not installed
 try:
     from google.api_core.client_options import ClientOptions
-    from google.cloud.aiplatform.gapic import JobServiceClient
+    from google.cloud.aiplatform.gapic import JobServiceAsyncClient
     from google.cloud.aiplatform_v1.types.custom_job import (
         ContainerSpec,
         CustomJob,
@@ -64,7 +64,7 @@
     from google.cloud.aiplatform_v1.types.job_state import JobState
     from google.cloud.aiplatform_v1.types.machine_resources import DiskSpec, MachineSpec
     from google.protobuf.duration_pb2 import Duration
-    from tenacity import retry, stop_after_attempt, wait_fixed, wait_random
+    from tenacity import AsyncRetrying, stop_after_attempt, wait_fixed, wait_random
 except ModuleNotFoundError:
     pass
 
@@ -409,35 +409,42 @@ async def run(
         job_name = configuration.job_name
 
         job_spec = self._build_job_spec(configuration)
-        with configuration.credentials.get_job_service_client(
-            client_options=client_options
-        ) as job_service_client:
-            job_run = await self._create_and_begin_job(
-                job_name, job_spec, job_service_client, configuration, logger
+        job_service_async_client = (
+            configuration.credentials.get_job_service_async_client(
+                client_options=client_options
             )
+        )
 
-            if task_status:
-                task_status.started(job_run.name)
-
-            final_job_run = await self._watch_job_run(
-                job_name=job_name,
-                full_job_name=job_run.name,
-                job_service_client=job_service_client,
-                current_state=job_run.state,
-                until_states=(
-                    JobState.JOB_STATE_SUCCEEDED,
-                    JobState.JOB_STATE_FAILED,
-                    JobState.JOB_STATE_CANCELLED,
-                    JobState.JOB_STATE_EXPIRED,
-                ),
-                configuration=configuration,
-                logger=logger,
-                timeout=int(
-                    datetime.timedelta(
-                        hours=configuration.job_spec["maximum_run_time_hours"]
-                    ).total_seconds()
-                ),
-            )
+        job_run = await self._create_and_begin_job(
+            job_name,
+            job_spec,
+            job_service_async_client,
+            configuration,
+            logger,
+        )
+
+        if task_status:
+            task_status.started(job_run.name)
+
+        final_job_run = await self._watch_job_run(
+            job_name=job_name,
+            full_job_name=job_run.name,
+            job_service_async_client=job_service_async_client,
+            current_state=job_run.state,
+            until_states=(
+                JobState.JOB_STATE_SUCCEEDED,
+                JobState.JOB_STATE_FAILED,
+                JobState.JOB_STATE_CANCELLED,
+                JobState.JOB_STATE_EXPIRED,
+            ),
+            configuration=configuration,
+            logger=logger,
+            timeout=int(
+                datetime.timedelta(
+                    hours=configuration.job_spec["maximum_run_time_hours"]
+                ).total_seconds()
+            ),
+        )
 
         error_msg = final_job_run.error.message
 
@@ -489,7 +496,7 @@ async def _create_and_begin_job(
         self,
         job_name: str,
         job_spec: "CustomJobSpec",
-        job_service_client: "JobServiceClient",
+        job_service_async_client: "JobServiceAsyncClient",
         configuration: VertexAIWorkerJobConfiguration,
         logger: PrefectLogAdapter,
     ) -> "CustomJob":
@@ -504,24 +511,23 @@ async def _create_and_begin_job(
         )
 
         # run job
-        logger.info(f"Job {job_name!r} starting to run ")
+        logger.info(f"Creating job {job_name!r}")
 
         project = configuration.project
         resource_name = f"projects/{project}/locations/{configuration.region}"
 
-        retry_policy = retry(
+        async for attempt in AsyncRetrying(
             stop=stop_after_attempt(3), wait=wait_fixed(1) + wait_random(0, 3)
-        )
-
-        custom_job_run = await run_sync_in_worker_thread(
-            retry_policy(job_service_client.create_custom_job),
-            parent=resource_name,
-            custom_job=custom_job,
-        )
+        ):
+            with attempt:
+                custom_job_run = await job_service_async_client.create_custom_job(
+                    parent=resource_name,
+                    custom_job=custom_job,
+                )
 
         logger.info(
-            f"Job {job_name!r} has successfully started; "
-            f"the full job name is {custom_job_run.name!r}"
+            f"Job {job_name!r} created. "
+            f"The full job name is {custom_job_run.name!r}"
         )
 
         return custom_job_run
@@ -530,7 +536,7 @@ async def _watch_job_run(
         self,
         job_name: str,
         full_job_name: str,  # different from job_name
-        job_service_client: "JobServiceClient",
+        job_service_async_client: "JobServiceAsyncClient",
         current_state: "JobState",
         until_states: Tuple["JobState"],
         configuration: VertexAIWorkerJobConfiguration,
@@ -539,14 +545,19 @@ async def _watch_job_run(
     ) -> "CustomJob":
         """
         Polls job run to see if status changed.
+
+        State changes reported by the Vertex AI API may sometimes be inaccurate
+        immediately upon startup, but should eventually report a correct running
+        and then terminal state. The minimum training duration for a custom job is
+        30 seconds, so short-lived jobs may be marked as successful some time
+        after a flow run has completed.
         """
         state = JobState.JOB_STATE_UNSPECIFIED
         last_state = current_state
         t0 = time.time()
 
         while state not in until_states:
-            job_run = await run_sync_in_worker_thread(
-                job_service_client.get_custom_job,
+            job_run = await job_service_async_client.get_custom_job(
                 name=full_job_name,
             )
             state = job_run.state
@@ -557,7 +568,7 @@ async def _watch_job_run(
                     .replace("state", "state is now:")
                 )
                 # results in "New job state is now: succeeded"
-                logger.info(f"{job_name} has new {state_label}")
+                logger.debug(f"{job_name} has new {state_label}")
                 last_state = state
             else:
                 # Intermittently, the job will not be described. We want to respect the
@@ -570,7 +581,7 @@ async def _watch_job_run(
                     f"Timed out after {elapsed_time}s while watching job for states "
                     "{until_states!r}"
                 )
-            time.sleep(configuration.job_watch_poll_interval)
+            await asyncio.sleep(configuration.job_watch_poll_interval)
 
         return job_run
 
@@ -621,22 +632,23 @@ async def kill_infrastructure(
         client_options = ClientOptions(
             api_endpoint=f"{configuration.region}-aiplatform.googleapis.com"
         )
-        with configuration.credentials.get_job_service_client(
-            client_options=client_options
-        ) as job_service_client:
-            await run_sync_in_worker_thread(
-                self._stop_job,
-                client=job_service_client,
-                vertex_job_name=infrastructure_pid,
+        job_service_async_client = (
+            configuration.credentials.get_job_service_async_client(
+                client_options=client_options
             )
+        )
+        await self._stop_job(
+            client=job_service_async_client,
+            vertex_job_name=infrastructure_pid,
+        )
 
-    def _stop_job(self, client: "JobServiceClient", vertex_job_name: str):
+    async def _stop_job(self, client: "JobServiceAsyncClient", vertex_job_name: str):
         """
         Calls the `cancel_custom_job` method on the Vertex AI Job Service Client.
         """
         cancel_custom_job_request = CancelCustomJobRequest(name=vertex_job_name)
         try:
-            client.cancel_custom_job(
+            await client.cancel_custom_job(
                 request=cancel_custom_job_request,
             )
         except Exception as exc:


Test patch:
diff --git a/src/integrations/prefect-gcp/tests/conftest.py b/src/integrations/prefect-gcp/tests/conftest.py
--- a/src/integrations/prefect-gcp/tests/conftest.py
+++ b/src/integrations/prefect-gcp/tests/conftest.py
@@ -1,6 +1,6 @@
 import json
 from pathlib import Path
-from unittest.mock import MagicMock
+from unittest.mock import AsyncMock, MagicMock
 
 import pytest
 from google.api_core.exceptions import NotFound as ApiCoreNotFound
@@ -250,6 +250,10 @@ def mock_credentials(monkeypatch):
         "prefect_gcp.credentials.google.auth",  # noqa
         mock_auth,
     )
+    monkeypatch.setattr(
+        "prefect_gcp.credentials.Credentials.from_service_account_info.universe_domain",  # noqa
+        "fake_universe_domain",
+    )
     return mock_credentials
 
 
@@ -271,7 +275,33 @@ def job_service_client():
 
 
 @pytest.fixture
-def gcp_credentials(monkeypatch, google_auth, mock_credentials, job_service_client):
+def job_service_async_client():
+    job_service_client_async_mock = MagicMock()
+    custom_run = MagicMock(name="mock_name")
+    job_service_client_async_mock.create_custom_job = AsyncMock(return_value=custom_run)
+
+    error = MagicMock(message="")
+    custom_run_final = MagicMock(
+        name="mock_name",
+        state=JobState.JOB_STATE_SUCCEEDED,
+        error=error,
+        display_name="mock_display_name",
+    )
+    job_service_client_async_mock.get_custom_job = AsyncMock(
+        return_value=custom_run_final
+    )
+    job_service_client_async_mock.cancel_custom_job = AsyncMock()
+    return job_service_client_async_mock
+
+
+@pytest.fixture
+def gcp_credentials(
+    monkeypatch,
+    google_auth,
+    mock_credentials,
+    job_service_client,
+    job_service_async_client,
+):
     gcp_credentials_mock = GcpCredentials(project="gcp_credentials_project")
     gcp_credentials_mock._service_account_email = "my_service_account_email"
 
@@ -279,6 +309,10 @@ def gcp_credentials(monkeypatch, google_auth, mock_credentials, job_service_clie
     gcp_credentials_mock.secret_manager_client = SecretManagerClient()
     gcp_credentials_mock.job_service_client = job_service_client
     gcp_credentials_mock.job_service_client.__enter__.return_value = job_service_client
+    gcp_credentials_mock.job_service_async_client = job_service_async_client
+    gcp_credentials_mock.job_service_client.__enter__.return_value = (
+        job_service_async_client
+    )
 
     gcp_credentials_mock.get_cloud_storage_client = (
         lambda *args, **kwargs: gcp_credentials_mock.cloud_storage_client
@@ -290,6 +324,9 @@ def gcp_credentials(monkeypatch, google_auth, mock_credentials, job_service_clie
     gcp_credentials_mock.get_job_service_client = (
         lambda *args, **kwargs: gcp_credentials_mock.job_service_client
     )
+    gcp_credentials_mock.get_job_service_async_client = (
+        lambda *args, **kwargs: gcp_credentials_mock.job_service_async_client
+    )
     return gcp_credentials_mock
 
 
diff --git a/src/integrations/prefect-gcp/tests/test_aiplatform.py b/src/integrations/prefect-gcp/tests/test_aiplatform.py
--- a/src/integrations/prefect-gcp/tests/test_aiplatform.py
+++ b/src/integrations/prefect-gcp/tests/test_aiplatform.py
@@ -113,8 +113,8 @@ def test_kill_infrastructure_not_found(
     ):
         identifier = "projects/1234/locations/us-east1/customJobs/12345"
         gcp_credentials = vertex_ai_custom_training_job.gcp_credentials
-        job_service_client = gcp_credentials.job_service_client
-        job_service_client.cancel_custom_job.side_effect = self.raise_not_found
+        job_service_async_client = gcp_credentials.job_service_async_client
+        job_service_async_client.cancel_custom_job.side_effect = self.raise_not_found
         with pytest.raises(
             InfrastructureNotFound, match="Cannot stop Vertex AI job; the job name"
         ):
@@ -128,8 +128,8 @@ def test_kill_infrastructure_error(
     ):
         identifier = "projects/1234/locations/us-east1/customJobs/12345"
         gcp_credentials = vertex_ai_custom_training_job.gcp_credentials
-        job_service_client = gcp_credentials.job_service_client
-        job_service_client.cancel_custom_job.side_effect = self.raise_random_error
+        job_service_async_client = gcp_credentials.job_service_async_client
+        job_service_async_client.cancel_custom_job.side_effect = self.raise_random_error
         with pytest.raises(RuntimeError, match="Random error"):
             vertex_ai_custom_training_job.kill(identifier)
 
@@ -149,7 +149,7 @@ def test_run_error(self, vertex_ai_custom_training_job: VertexAICustomTrainingJo
             display_name="mock_display_name",
         )
         gcp_credentials = vertex_ai_custom_training_job.gcp_credentials
-        gcp_credentials.job_service_client.get_custom_job.return_value = (
+        gcp_credentials.job_service_async_client.get_custom_job.return_value = (
             failed_run_final
         )
         with pytest.raises(RuntimeError, match="my error msg"):
@@ -159,13 +159,15 @@ def test_run_start_error(
         self, vertex_ai_custom_training_job: VertexAICustomTrainingJob
     ):
         gcp_credentials = vertex_ai_custom_training_job.gcp_credentials
-        gcp_credentials.job_service_client.create_custom_job.side_effect = (
+        gcp_credentials.job_service_async_client.create_custom_job.side_effect = (
             RuntimeError()
         )
 
         with pytest.raises(RetryError):
             vertex_ai_custom_training_job.run()
-        assert gcp_credentials.job_service_client.create_custom_job.call_count == 3
+        assert (
+            gcp_credentials.job_service_async_client.create_custom_job.call_count == 3
+        )
 
     def test_machine_spec(
         self, vertex_ai_custom_training_job: VertexAICustomTrainingJob
diff --git a/src/integrations/prefect-gcp/tests/test_credentials.py b/src/integrations/prefect-gcp/tests/test_credentials.py
--- a/src/integrations/prefect-gcp/tests/test_credentials.py
+++ b/src/integrations/prefect-gcp/tests/test_credentials.py
@@ -3,9 +3,9 @@
 from pathlib import Path
 
 import pytest
-from google.cloud.aiplatform.gapic import JobServiceClient
+from google.cloud.aiplatform.gapic import JobServiceAsyncClient, JobServiceClient
 from prefect_gcp import GcpCredentials
-from prefect_gcp.credentials import ClientType
+from prefect_gcp.credentials import ClientType, _get_job_service_async_client_cached
 
 from prefect import flow, task
 from prefect.blocks.core import Block
@@ -157,6 +157,47 @@ def test_flow():
     test_flow()
 
 
+async def test_get_job_service_async_client(service_account_info, oauth2_credentials):
+    @flow
+    def test_flow():
+        project = "test_project"
+        credentials = GcpCredentials(
+            service_account_info=service_account_info,
+            project=project,
+        )
+        client = credentials.get_job_service_async_client(client_options={})
+        assert isinstance(client, JobServiceAsyncClient)
+
+    test_flow()
+
+
+async def test_get_job_service_async_client_cached(
+    service_account_info, oauth2_credentials
+):
+    """
+    Test to ensure that _get_job_service_async_client_cached function returns the same instance
+    for multiple calls with the same parameters and properly utilizes lru_cache.
+    """
+    _get_job_service_async_client_cached.cache_clear()
+
+    project = "test_project"
+    credentials = GcpCredentials(
+        service_account_info=service_account_info,
+        project=project,
+    )
+
+    assert (
+        _get_job_service_async_client_cached.cache_info().hits == 0
+    ), "Initial call count should be 0"
+
+    credentials.get_job_service_async_client(client_options={})
+    credentials.get_job_service_async_client(client_options={})
+    credentials.get_job_service_async_client(client_options={})
+
+    assert _get_job_service_async_client_cached.cache_info().misses == 1
+    assert _get_job_service_async_client_cached.cache_info().hits == 2
+
+
 class MockTargetConfigs(Block):
     credentials: GcpCredentials
 
diff --git a/src/integrations/prefect-gcp/tests/test_vertex_worker.py b/src/integrations/prefect-gcp/tests/test_vertex_worker.py
--- a/src/integrations/prefect-gcp/tests/test_vertex_worker.py
+++ b/src/integrations/prefect-gcp/tests/test_vertex_worker.py
@@ -157,11 +157,12 @@ async def test_successful_worker_run(self, flow_run, job_config):
             job_config.prepare_for_flow_run(flow_run, None, None)
             result = await worker.run(flow_run=flow_run, configuration=job_config)
             assert (
-                job_config.credentials.job_service_client.create_custom_job.call_count
+                job_config.credentials.job_service_async_client.create_custom_job.call_count
                 == 1
             )
             assert (
-                job_config.credentials.job_service_client.get_custom_job.call_count == 1
+                job_config.credentials.job_service_async_client.get_custom_job.call_count
+                == 1
             )
             assert result == VertexAIWorkerResult(
                 status_code=0, identifier="mock_display_name"
@@ -171,7 +172,7 @@ async def test_failed_worker_run(self, flow_run, job_config):
         job_config.prepare_for_flow_run(flow_run, None, None)
         error_msg = "something went kablooey"
         error_job_display_name = "catastrophization"
-        job_config.credentials.job_service_client.get_custom_job.return_value = (
+        job_config.credentials.job_service_async_client.get_custom_job.return_value = (
             MagicMock(
                 name="error_mock_name",
                 state=JobState.JOB_STATE_FAILED,
@@ -184,17 +185,18 @@ async def test_failed_worker_run(self, flow_run, job_config):
                 await worker.run(flow_run=flow_run, configuration=job_config)
 
             assert (
-                job_config.credentials.job_service_client.create_custom_job.call_count
+                job_config.credentials.job_service_async_client.create_custom_job.call_count
                 == 1
             )
             assert (
-                job_config.credentials.job_service_client.get_custom_job.call_count == 1
+                job_config.credentials.job_service_async_client.get_custom_job.call_count
+                == 1
             )
 
     async def test_cancelled_worker_run(self, flow_run, job_config):
         job_config.prepare_for_flow_run(flow_run, None, None)
         job_display_name = "a-job-well-done"
-        job_config.credentials.job_service_client.get_custom_job.return_value = (
+        job_config.credentials.job_service_async_client.get_custom_job.return_value = (
             MagicMock(
                 name="cancelled_mock_name",
                 state=JobState.JOB_STATE_CANCELLED,
@@ -205,18 +207,19 @@ async def test_cancelled_worker_run(self, flow_run, job_config):
         async with VertexAIWorker("test-pool") as worker:
             result = await worker.run(flow_run=flow_run, configuration=job_config)
             assert (
-                job_config.credentials.job_service_client.create_custom_job.call_count
+                job_config.credentials.job_service_async_client.create_custom_job.call_count
                 == 1
             )
             assert (
-                job_config.credentials.job_service_client.get_custom_job.call_count == 1
+                job_config.credentials.job_service_async_client.get_custom_job.call_count
+                == 1
             )
             assert result == VertexAIWorkerResult(
                 status_code=1, identifier=job_display_name
             )
 
     async def test_kill_infrastructure(self, flow_run, job_config):
-        mock = job_config.credentials.job_service_client.create_custom_job
+        mock = job_config.credentials.job_service_async_client.create_custom_job
         # the CancelCustomJobRequest class seems to reject a MagicMock value
         # so here, we'll use a SimpleNamespace as the mocked return values
         mock.return_value = SimpleNamespace(
@@ -229,14 +232,14 @@ async def test_kill_infrastructure(self, flow_run, job_config):
                     result = await tg.start(worker.run, flow_run, job_config)
                 await worker.kill_infrastructure(result, job_config)
 
-            mock = job_config.credentials.job_service_client.cancel_custom_job
+            mock = job_config.credentials.job_service_async_client.cancel_custom_job
             assert mock.call_count == 1
             mock.assert_called_with(request=CancelCustomJobRequest(name="foobar"))
 
     async def test_kill_infrastructure_no_grace_seconds(
         self, flow_run, job_config, caplog
     ):
-        mock = job_config.credentials.job_service_client.create_custom_job
+        mock = job_config.credentials.job_service_async_client.create_custom_job
         mock.return_value = SimpleNamespace(
             name="bazzbar", state=JobState.JOB_STATE_PENDING
         )
@@ -260,8 +263,8 @@ async def test_kill_infrastructure_no_grace_seconds(
 
     async def test_kill_infrastructure_not_found(self, job_config):
         async with VertexAIWorker("test-pool") as worker:
-            job_config.credentials.job_service_client.cancel_custom_job.side_effect = (
-                Exception("does not exist")
+            job_config.credentials.job_service_async_client.cancel_custom_job.side_effect = Exception(
+                "does not exist"
             )
             with pytest.raises(
                 InfrastructureNotFound, match="Cannot stop Vertex AI job"