Repository: apache/airflow (Python)

Issue:
Serialization problem in trigger in Google provider that raises NotImplementedError in case when non-primitive type is returned from triggerer
### Apache Airflow version

3.0.5

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

In Airflow 3.0.3 with PR https://github.com/apache/airflow/pull/51699 we have introduced new logic to serialize objects, that caused multiple failures in Google provider triggers. 
There are for now 7 triggers that fail because of "too complicated" objects returned from triggerer on the step when it is trying to serialize the object before passing it to worker. The one example I can give is "airflow.providers.google.cloud.triggers.dataplex.DataplexDataProfileJobTrigger" and the error that is returned from the trigger is:
```
triggerer  | TriggerRunner().run()
triggerer  | File "/opt/airflow/airflow-core/src/airflow/jobs/triggerer_job_runner.py", line 845, in run
triggerer  | asyncio.run(self.arun())
triggerer  | File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
triggerer  | return loop.run_until_complete(main)
triggerer  | File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
triggerer  | return future.result()
triggerer  | File "/opt/airflow/airflow-core/src/airflow/jobs/triggerer_job_runner.py", line 869, in arun
triggerer  | await self.sync_state_to_supervisor(finished_ids)
triggerer  | File "/opt/airflow/airflow-core/src/airflow/jobs/triggerer_job_runner.py", line 1055, in sync_state_to_supervisor
triggerer  | resp = await self.comms_decoder.asend(msg)
triggerer  | File "/opt/airflow/airflow-core/src/airflow/jobs/triggerer_job_runner.py", line 790, in asend
triggerer  | bytes = frame.as_bytes()
triggerer  | File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/comms.py", line 148, in as_bytes
triggerer  | self.req_encoder.encode_into(self, buffer, 4)
triggerer  | File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/comms.py", line 125, in _msgpack_enc_hook
triggerer  | raise NotImplementedError(f"Objects of type {type(obj)} are not supported")
triggerer  | NotImplementedError: Objects of type <enum 'State'> are not supported
triggerer  | 2025-08-21 12:39:09 [warning  ] Process exited abnormally      [airflow.jobs.triggerer_job_runner] exit_code=126
triggerer  | 2025-08-21 12:39:09 [error    ] Trigger runner process has died! Exiting. [airflow.jobs.triggerer_job_runner]
triggerer  | [2025-08-21T12:39:09.349+0000] {triggerer_job_runner.py:177} INFO - Waiting for triggers to clean up
triggerer  | [2025-08-21T12:39:09.350+0000] {triggerer_job_runner.py:182} INFO - Exited trigger loop
```

After that the trigger is dying and not doing anything. the task fails with trigger timeout
Not sure if we have issue for that, but the whole list looks like this:
```
- dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
- dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
- dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
- dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
- example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
- example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
- example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
```

For all those system tests we observe the same error and I think this should be covered in here to extend the logic for types (introduced in https://github.com/apache/airflow/pull/51699 ):
```
def _msgpack_enc_hook(obj: Any) -> Any:
    import pendulum

    if isinstance(obj, pendulum.DateTime):
        # convert the pendulm Datetime subclass into a raw datetime so that msgspec can use it's native
        # encoding
        return datetime(
            obj.year, obj.month, obj.day, obj.hour, obj.minute, obj.second, obj.microsecond, tzinfo=obj.tzinfo
        )
    if isinstance(obj, Path):
        return str(obj)
    if isinstance(obj, BaseModel):
        return obj.model_dump(exclude_unset=True)

    # Raise a NotImplementedError for other types
    raise NotImplementedError(f"Objects of type {type(obj)} are not supported")
```



### What you think should happen instead?

_No response_

### How to reproduce

1. Run example_dataplex_dp.py in breeze
2. wait until the task run_data_scan_def will start
3. observe breeze cli to see the actual error in the triggerer. Should look like this:
`triggerer  | NotImplementedError: Objects of type <enum 'State'> are not supported`
4. the task will die after some time with `triggerer  timeout`

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

This is the problem from airflow core, was introduced in version 3.0.3, starting from this PR: https://github.com/apache/airflow/pull/51699

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion: I think we have two ways of solving it:

1) More generic -> possibly we could serialize those classes (they are protobufs really) using generic to/from String and store the "class" of the object (so that we can deserialize it and know what to deserialize it from) 

2) More "extendable"  - add "serialization" extension to providers.

We could implement extendable serialization - and have providers to implement "serialize/deserialize" extention. We could have that Google-provider serialzation "extension" recognize and serialize/deserialize the objects we want.

I am torn between the two. 1) is more generic but 2) is something I would also like to see for other providers possibly, and we could have a mechanism that finally allows to add serialization capabilities (and code) independently from airflow-core.

WDYT. @kaxil @amoghrajesh @bolkedebruin @ashb ?

Could we make do with a generic fallback for `str(value)` in the msg enc hook for anything we don't know about? Would that work here? (It was about to get stored in the DB anyway, so what process did it use to get there anyway?)
but this is communication channel between python processes, so other part have to deserialize it , so from str it can be tricky
2. is a much much longer term goal I see and many provider owners would have to kind of chime in to help with that effort and considering 3.1 priorities, now doesn't seem to be a great time for that.

For now, for any object that isn't / cannot be handled by: https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/comms.py#L110-L129, maybe we could just stringify it?
> but this is communication channel between python processes, so other part have to deserialize it , so from str it can be tricky

Correct, but that should not be too tricky imo. This is where deserialization is defined:
https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/comms.py#L183-L185. Maybe do something using `ast` library to get actual object imo.
Hi,

We are also facing the similar issue in the standard filesystem provider. We are getting this error in FileSensor - NotImplementedError: Objects of type <class 'pydantic_core._pydantic_core.SerializationIterator'> are not supported.

Is there a patch or quick fix for this issue?
> Hi,
> 
> We are also facing the similar issue in the standard filesystem provider. We are getting this error in FileSensor - NotImplementedError: Objects of type <class 'pydantic_core._pydantic_core.SerializationIterator'> are not supported.
> 
> Is there a patch or quick fix for this issue?

This is a different issue. 3.0.5 has been yanked due to this issue - you should downgrade to 3.0.4 or wait for 3.0.6 - you can test  rc1 that is just being released https://github.com/apache/airflow/issues/54835 -> please test and report in the issue if your problem is fixed @stantaneous 
> > Hi,
> > We are also facing the similar issue in the standard filesystem provider. We are getting this error in FileSensor - NotImplementedError: Objects of type <class 'pydantic_core._pydantic_core.SerializationIterator'> are not supported.
> > Is there a patch or quick fix for this issue?
> 
> This is a different issue. 3.0.5 has been yanked due to this issue - you should downgrade to 3.0.4 or wait for 3.0.6 - you can test rc1 that is just being released [#54835](https://github.com/apache/airflow/issues/54835) -> please test and report in the issue if your problem is fixed [@stantaneous](https://github.com/stantaneous)

With 3.0.4, we have the issue with paramiko library as DSSKEY is removed in the paramiko 4.0.0. I was able to fix it by manually adding paramiko<4.0.0 in my requirements.

With 3.0.6rc1, Everything seems to be working fine. Waiting for 3.0.6 :)

@stantaneous 3.0.4 already had fix for paramiko: https://github.com/apache/airflow/pull/54078 though
@amoghrajesh The docker image for 3.0.4 ships with the apache-airflow-provider-ssh==4.1.2 which installs the latest paramiko==4.0.0 when we extend and build the image. The temp fix is applied in the apache-airflow-provider-ssh==4.1.3 -  https://pypi.org/project/apache-airflow-providers-ssh/4.1.3/ 

I quickly installed 3.0.4 back and checked the package version in docker desktop, See screenshot of package version -

<img width="674" height="482" alt="Image" src="https://github.com/user-attachments/assets/6053f8c8-34c9-490a-925d-a7b3a94af074" />



Fix (patch):
diff --git a/providers/google/src/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py b/providers/google/src/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
index 596585f08ccab..73cf777a51b03 100644
--- a/providers/google/src/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
+++ b/providers/google/src/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
@@ -36,13 +36,13 @@
 from datetime import timedelta
 from typing import TYPE_CHECKING, Any
 
-from google.api_core import protobuf_helpers
 from google.cloud.storage_transfer_v1 import (
     ListTransferJobsRequest,
     StorageTransferServiceAsyncClient,
     TransferJob,
     TransferOperation,
 )
+from google.protobuf.json_format import MessageToDict
 from googleapiclient.discovery import Resource, build
 from googleapiclient.errors import HttpError
 
@@ -603,7 +603,7 @@ async def list_transfer_operations(
         self,
         request_filter: dict | None = None,
         **kwargs,
-    ) -> list[TransferOperation]:
+    ) -> list[dict[str, Any]]:
         """
         Get a transfer operation in Google Storage Transfer Service.
 
@@ -660,7 +660,12 @@ async def list_transfer_operations(
             )
 
         transfer_operations = [
-            protobuf_helpers.from_any_pb(TransferOperation, op.metadata) for op in operations
+            MessageToDict(
+                getattr(op, "_pb", op),
+                preserving_proto_field_name=True,
+                use_integers_for_enums=True,
+            )
+            for op in operations
         ]
 
         return transfer_operations
@@ -677,7 +682,7 @@ async def _inject_project_id(self, body: dict, param_name: str, target_key: str)
 
     @staticmethod
     async def operations_contain_expected_statuses(
-        operations: list[TransferOperation], expected_statuses: set[str] | str
+        operations: list[dict[str, Any]], expected_statuses: set[str] | str
     ) -> bool:
         """
         Check whether an operation exists with the expected status.
@@ -696,7 +701,7 @@ async def operations_contain_expected_statuses(
         if not operations:
             return False
 
-        current_statuses = {operation.status.name for operation in operations}
+        current_statuses = {TransferOperation.Status(op["metadata"]["status"]).name for op in operations}
 
         if len(current_statuses - expected_statuses_set) != len(current_statuses):
             return True
diff --git a/providers/google/src/airflow/providers/google/cloud/operators/dataplex.py b/providers/google/src/airflow/providers/google/cloud/operators/dataplex.py
index e6f04e6306233..b72c5f094c3dd 100644
--- a/providers/google/src/airflow/providers/google/cloud/operators/dataplex.py
+++ b/providers/google/src/airflow/providers/google/cloud/operators/dataplex.py
@@ -1082,11 +1082,11 @@ def execute_complete(self, context, event=None) -> None:
         """
         job_state = event["job_state"]
         job_id = event["job_id"]
-        if job_state == DataScanJob.State.FAILED:
+        if job_state == DataScanJob.State.FAILED.name:  # type: ignore
             raise AirflowException(f"Job failed:\n{job_id}")
-        if job_state == DataScanJob.State.CANCELLED:
+        if job_state == DataScanJob.State.CANCELLED.name:  # type: ignore
             raise AirflowException(f"Job was cancelled:\n{job_id}")
-        if job_state == DataScanJob.State.SUCCEEDED:
+        if job_state == DataScanJob.State.SUCCEEDED.name:  # type: ignore
             job = event["job"]
             if not job["data_quality_result"]["passed"]:
                 if self.fail_on_dq_failure:
@@ -1260,11 +1260,11 @@ def execute_complete(self, context, event=None) -> None:
         job_state = event["job_state"]
         job_id = event["job_id"]
         job = event["job"]
-        if job_state == DataScanJob.State.FAILED:
+        if job_state == DataScanJob.State.FAILED.name:  # type: ignore
             raise AirflowException(f"Job failed:\n{job_id}")
-        if job_state == DataScanJob.State.CANCELLED:
+        if job_state == DataScanJob.State.CANCELLED.name:  # type: ignore
             raise AirflowException(f"Job was cancelled:\n{job_id}")
-        if job_state == DataScanJob.State.SUCCEEDED:
+        if job_state == DataScanJob.State.SUCCEEDED.name:  # type: ignore
             if not job["data_quality_result"]["passed"]:
                 if self.fail_on_dq_failure:
                     raise AirflowDataQualityScanException(
@@ -1639,12 +1639,12 @@ def execute(self, context: Context) -> dict:
                 result_timeout=self.result_timeout,
             )
 
-            if job.state == DataScanJob.State.FAILED:
+            if job.state == DataScanJob.State.FAILED.name:  # type: ignore
                 raise AirflowException(f"Data Profile job failed: {job_id}")
-            if job.state == DataScanJob.State.SUCCEEDED:
+            if job.state == DataScanJob.State.SUCCEEDED.name:  # type: ignore
                 self.log.info("Data Profile job executed successfully.")
             else:
-                self.log.info("Data Profile job execution returned status: %s", job.status)
+                self.log.info("Data Profile job execution returned status: %s", job.state)
 
         return job_id
 
@@ -1657,11 +1657,11 @@ def execute_complete(self, context, event=None) -> None:
         """
         job_state = event["job_state"]
         job_id = event["job_id"]
-        if job_state == DataScanJob.State.FAILED:
+        if job_state == DataScanJob.State.FAILED.name:  # type: ignore
             raise AirflowException(f"Job failed:\n{job_id}")
-        if job_state == DataScanJob.State.CANCELLED:
+        if job_state == DataScanJob.State.CANCELLED.name:  # type: ignore
             raise AirflowException(f"Job was cancelled:\n{job_id}")
-        if job_state == DataScanJob.State.SUCCEEDED:
+        if job_state == DataScanJob.State.SUCCEEDED.name:  # type: ignore
             self.log.info("Data Profile job executed successfully.")
         return job_id
 
diff --git a/providers/google/src/airflow/providers/google/cloud/operators/dataproc.py b/providers/google/src/airflow/providers/google/cloud/operators/dataproc.py
index 678259d4207a5..72a5fda7ed866 100644
--- a/providers/google/src/airflow/providers/google/cloud/operators/dataproc.py
+++ b/providers/google/src/airflow/providers/google/cloud/operators/dataproc.py
@@ -1952,9 +1952,9 @@ def execute_complete(self, context, event=None) -> None:
         job_state = event["job_state"]
         job_id = event["job_id"]
         job = event["job"]
-        if job_state == JobStatus.State.ERROR:
+        if job_state == JobStatus.State.ERROR.name:  # type: ignore
             raise AirflowException(f"Job {job_id} failed:\n{job}")
-        if job_state == JobStatus.State.CANCELLED:
+        if job_state == JobStatus.State.CANCELLED.name:  # type: ignore
             raise AirflowException(f"Job {job_id} was cancelled:\n{job}")
         self.log.info("%s completed successfully.", self.task_id)
         return job_id
@@ -2462,7 +2462,7 @@ def execute(self, context: Context):
                 if not self.hook.check_error_for_resource_is_not_ready_msg(batch.state_message):
                     break
 
-        self.handle_batch_status(context, batch.state, batch_id, batch.state_message)
+        self.handle_batch_status(context, batch.state.name, batch_id, batch.state_message)
         return Batch.to_dict(batch)
 
     @cached_property
@@ -2487,19 +2487,19 @@ def on_kill(self):
             self.operation.cancel()
 
     def handle_batch_status(
-        self, context: Context, state: Batch.State, batch_id: str, state_message: str | None = None
+        self, context: Context, state: str, batch_id: str, state_message: str | None = None
     ) -> None:
         # The existing batch may be a number of states other than 'SUCCEEDED'\
         # wait_for_operation doesn't fail if the job is cancelled, so we will check for it here which also
         # finds a cancelling|canceled|unspecified job from wait_for_batch or the deferred trigger
         link = DATAPROC_BATCH_LINK.format(region=self.region, project_id=self.project_id, batch_id=batch_id)
-        if state == Batch.State.FAILED:
+        if state == Batch.State.FAILED.name:  # type: ignore
             raise AirflowException(
                 f"Batch job {batch_id} failed with error: {state_message}.\nDriver logs: {link}"
             )
-        if state in (Batch.State.CANCELLED, Batch.State.CANCELLING):
+        if state in (Batch.State.CANCELLED.name, Batch.State.CANCELLING.name):  # type: ignore
             raise AirflowException(f"Batch job {batch_id} was cancelled.\nDriver logs: {link}")
-        if state == Batch.State.STATE_UNSPECIFIED:
+        if state == Batch.State.STATE_UNSPECIFIED.name:  # type: ignore
             raise AirflowException(f"Batch job {batch_id} unspecified.\nDriver logs: {link}")
         self.log.info("Batch job %s completed.\nDriver logs: %s", batch_id, link)
 
diff --git a/providers/google/src/airflow/providers/google/cloud/triggers/dataplex.py b/providers/google/src/airflow/providers/google/cloud/triggers/dataplex.py
index f9b83e004b571..408bbfebdf803 100644
--- a/providers/google/src/airflow/providers/google/cloud/triggers/dataplex.py
+++ b/providers/google/src/airflow/providers/google/cloud/triggers/dataplex.py
@@ -103,7 +103,13 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
                 self.polling_interval_seconds,
             )
             await asyncio.sleep(self.polling_interval_seconds)
-        yield TriggerEvent({"job_id": self.job_id, "job_state": state, "job": self._convert_to_dict(job)})
+        yield TriggerEvent(
+            {
+                "job_id": self.job_id,
+                "job_state": DataScanJob.State(state).name,
+                "job": self._convert_to_dict(job),
+            }
+        )
 
     def _convert_to_dict(self, job: DataScanJob) -> dict:
         """Return a representation of a DataScanJob instance as a dict."""
@@ -185,7 +191,13 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
                 self.polling_interval_seconds,
             )
             await asyncio.sleep(self.polling_interval_seconds)
-        yield TriggerEvent({"job_id": self.job_id, "job_state": state, "job": self._convert_to_dict(job)})
+        yield TriggerEvent(
+            {
+                "job_id": self.job_id,
+                "job_state": DataScanJob.State(state).name,
+                "job": self._convert_to_dict(job),
+            }
+        )
 
     def _convert_to_dict(self, job: DataScanJob) -> dict:
         """Return a representation of a DataScanJob instance as a dict."""
diff --git a/providers/google/src/airflow/providers/google/cloud/triggers/dataproc.py b/providers/google/src/airflow/providers/google/cloud/triggers/dataproc.py
index d89ce7e477b4f..89a559b2ec1ea 100644
--- a/providers/google/src/airflow/providers/google/cloud/triggers/dataproc.py
+++ b/providers/google/src/airflow/providers/google/cloud/triggers/dataproc.py
@@ -27,7 +27,7 @@
 
 from asgiref.sync import sync_to_async
 from google.api_core.exceptions import NotFound
-from google.cloud.dataproc_v1 import Batch, Cluster, ClusterStatus, JobStatus
+from google.cloud.dataproc_v1 import Batch, Cluster, ClusterStatus, Job, JobStatus
 
 from airflow.exceptions import AirflowException
 from airflow.providers.google.cloud.hooks.dataproc import DataprocAsyncHook, DataprocHook
@@ -194,7 +194,9 @@ async def run(self):
                 if state in (JobStatus.State.DONE, JobStatus.State.CANCELLED, JobStatus.State.ERROR):
                     break
                 await asyncio.sleep(self.polling_interval_seconds)
-            yield TriggerEvent({"job_id": self.job_id, "job_state": state, "job": job})
+            yield TriggerEvent(
+                {"job_id": self.job_id, "job_state": JobStatus.State(state).name, "job": Job.to_dict(job)}
+            )
         except asyncio.CancelledError:
             self.log.info("Task got cancelled.")
             try:
@@ -212,7 +214,12 @@ async def run(self):
                         job_id=self.job_id, project_id=self.project_id, region=self.region
                     )
                     self.log.info("Job: %s is cancelled", self.job_id)
-                    yield TriggerEvent({"job_id": self.job_id, "job_state": ClusterStatus.State.DELETING})
+                    yield TriggerEvent(
+                        {
+                            "job_id": self.job_id,
+                            "job_state": ClusterStatus.State.DELETING.name,
+                        }
+                    )
             except Exception as e:
                 self.log.error("Failed to cancel the job: %s with error : %s", self.job_id, str(e))
                 raise e
@@ -322,7 +329,7 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
                     yield TriggerEvent(
                         {
                             "cluster_name": self.cluster_name,
-                            "cluster_state": ClusterStatus.State(ClusterStatus.State.DELETING).name,
+                            "cluster_state": ClusterStatus.State.DELETING.name,  # type: ignore
                             "cluster": Cluster.to_dict(cluster),
                         }
                     )
@@ -428,12 +435,16 @@ async def run(self):
 
             if state in (Batch.State.FAILED, Batch.State.SUCCEEDED, Batch.State.CANCELLED):
                 break
-            self.log.info("Current state is %s", state)
+            self.log.info("Current state is %s", Batch.State(state).name)
             self.log.info("Sleeping for %s seconds.", self.polling_interval_seconds)
             await asyncio.sleep(self.polling_interval_seconds)
 
         yield TriggerEvent(
-            {"batch_id": self.batch_id, "batch_state": state, "batch_state_message": batch.state_message}
+            {
+                "batch_id": self.batch_id,
+                "batch_state": Batch.State(state).name,
+                "batch_state_message": batch.state_message,
+            }
         )