Repository: apache/airflow

Issue:
KubernetesExecutor leaves failed pods due to deepcopy issue with Google providers
### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

With Airflow 2.3 and 2.4 there appears to be a bug in the KubernetesExecutor when used in conjunction with the Google airflow providers. This bug does not affect Airflow 2.2 due to the pip version requirements.

The bug specifically presents itself when using nearly any Google provider operator. During the pod lifecycle, all is well until the executor in the pod starts to clean up following a successful run. Airflow itself still see's the task marked as a success, but in Kubernetes, while the task is finishing up after reporting status, it actually crashes and puts the pod into a Failed state silently:
```
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/__main__.py", line 39, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 52, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 103, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 382, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 189, in _run_task_by_selected_method
    _run_task_by_local_task_job(args, ti)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 247, in _run_task_by_local_task_job
    run_job.run()
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/base_job.py", line 247, in run
    self._execute()
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/local_task_job.py", line 137, in _execute
    self.handle_task_exit(return_code)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/local_task_job.py", line 168, in handle_task_exit
    self._run_mini_scheduler_on_child_tasks()
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/local_task_job.py", line 253, in _run_mini_scheduler_on_child_tasks
    partial_dag = task.dag.partial_subset(
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dag.py", line 2188, in partial_subset
    dag.task_dict = {
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dag.py", line 2189, in <dictcomp>
    t.task_id: _deepcopy_task(t)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dag.py", line 2186, in _deepcopy_task
    return copy.deepcopy(t, memo)
  File "/usr/local/lib/python3.9/copy.py", line 153, in deepcopy
    y = copier(memo)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 1163, in __deepcopy__
    setattr(result, k, copy.deepcopy(v, memo))
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 264, in _reconstruct
    y = func(*args)
  File "/usr/local/lib/python3.9/enum.py", line 384, in __call__
    return cls.__new__(cls, value)
  File "/usr/local/lib/python3.9/enum.py", line 702, in __new__
    raise ve_exc
ValueError: <object object at 0x7f570181a3c0> is not a valid _MethodDefault
```

Based on a quick look, it appears to be related to the default argument that Google is using in its operators which happens to be an Enum, and fails during a deepcopy at the end of the task.

Example operator that is affected: https://github.com/apache/airflow/blob/403ed7163f3431deb7fc21108e1743385e139907/airflow/providers/google/cloud/hooks/dataproc.py#L753
Reference to the Google Python API core which has the Enum causing the problem: https://github.com/googleapis/python-api-core/blob/main/google/api_core/gapic_v1/method.py#L31

### What you think should happen instead

Kubernetes pods should succeed, be marked as `Completed`, and then be gracefully terminated.

### How to reproduce

Use any `apache-airflow-providers-google` >= 7.0.0 which includes `google-api-core` >= 2.2.2. Run a DAG with a task which uses any of the Google operators which have `_MethodDefault` as a default argument.

### Operating System

Debian GNU/Linux 11 (bullseye)

### Versions of Apache Airflow Providers

apache-airflow-providers-amazon==6.0.0
apache-airflow-providers-apache-hive==5.0.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.4.0
apache-airflow-providers-common-sql==1.3.1
apache-airflow-providers-docker==3.2.0
apache-airflow-providers-elasticsearch==4.2.1
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.4.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.3.0
apache-airflow-providers-mysql==3.2.1
apache-airflow-providers-odbc==3.1.2
apache-airflow-providers-postgres==5.2.2
apache-airflow-providers-presto==4.2.0
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.1.0
apache-airflow-providers-slack==6.0.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.2.0

### Deployment

Other 3rd-party Helm chart

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

This seems to be a limitation in Python until 3.10 (not sure when exactly this was fixed). I’m not quite sure if we want to (or even _can_) fix this, to be honest.
Ah, maybe this could be possible if we implement custom copying logic on operators in the Google provider. Can you provide a quick example DAG that replicates this?
Hi @uranusjr - I don't have a public example DAG to provide unfortunately (very sorry!), but nearly any of the google providers with the `_MethodDefault` with the default parameter for the `retry` argument should exhibit this. We've seen it most with dataproc operators (where we found this bug):
- https://airflow.apache.org/docs/apache-airflow-providers-google/7.0.0/_api/airflow/providers/google/cloud/operators/dataproc/index.html#airflow.providers.google.cloud.operators.dataproc.DataprocSubmitJobOperator
I'm facing a similar issue with Airflow 2.3.2 deployed in EKS

```Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 38, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 51, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 99, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 376, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 182, in _run_task_by_selected_method
    _run_task_by_local_task_job(args, ti)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 240, in _run_task_by_local_task_job
    run_job.run()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 244, in run
    self._execute()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 133, in _execute
    self.handle_task_exit(return_code)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 171, in handle_task_exit
    self._run_mini_scheduler_on_child_tasks()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 253, in _run_mini_scheduler_on_child_tasks
    partial_dag = task.dag.partial_subset(
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2013, in partial_subset
    dag.task_dict = {
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2014, in <dictcomp>
    t.task_id: _deepcopy_task(t)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2011, in _deepcopy_task
    return copy.deepcopy(t, memo)
  File "/usr/local/lib/python3.8/copy.py", line 153, in deepcopy
    y = copier(memo)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1156, in __deepcopy__
    setattr(result, k, copy.deepcopy(v, memo))
  File "/usr/local/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'module' object

```
@uranusjr Also experiencing this with `BigQueryExecuteQueryOperator`.

### Environment
Airflow 2.4.3
Python 3.8.15
KubernetesExecutor
GKE Version 1.23.13-gke.900

### Error Encountered
Interestingly, there is no indication of the failed task from Airflow UI. It seems to succeed...

![image](https://user-images.githubusercontent.com/22802852/212174908-13f43702-2aab-43cb-8e8b-c03e8959524c.png)
 
However there are failed task pods in GKE.
![image](https://user-images.githubusercontent.com/22802852/212174699-792083a6-52fe-4ea6-a196-5ae99feaddae.png)

With the following errors:

Traceback (most recent call last): File "/home/airflow/.local/bin/airflow", line 8, in <module> sys.exit(main()) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 39, in main args.func(args) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 52, in command return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 103, in wrapper return f(*args, **kwargs) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 382, in task_run _run_task_by_selected_method(args, dag, ti) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 189, in _run_task_by_selected_method _run_task_by_local_task_job(args, ti) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 247, in _run_task_by_local_task_job run_job.run() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 247, in run self._execute() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 132, in _execute self.handle_task_exit(return_code) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 163, in handle_task_exit self.task_instance.schedule_downstream_tasks() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 75, in wrapper return func(*args, session=session, **kwargs) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2603, in schedule_downstream_tasks partial_dag = task.dag.partial_subset( File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2188, in partial_subset dag.task_dict = { File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2189, in <dictcomp> t.task_id: _deepcopy_task(t) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2186, in _deepcopy_task return copy.deepcopy(t, memo) File "/usr/local/lib/python3.8/copy.py", line 153, in deepcopy y = copier(memo) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1163, in __deepcopy__ setattr(result, k, copy.deepcopy(v, memo)) File "/usr/local/lib/python3.8/copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "/usr/local/lib/python3.8/copy.py", line 270, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python3.8/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/local/lib/python3.8/copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python3.8/copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "/usr/local/lib/python3.8/copy.py", line 264, in _reconstruct y = func(*args) File "/usr/local/lib/python3.8/copy.py", line 263, in <genexpr> args = (deepcopy(arg, memo) for arg in args) File "/usr/local/lib/python3.8/copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "/usr/local/lib/python3.8/copy.py", line 264, in _reconstruct y = func(*args) File "/usr/local/lib/python3.8/enum.py", line 339, in __call__ return cls.__new__(cls, value) File "/usr/local/lib/python3.8/enum.py", line 663, in __new__ raise ve_exc ValueError: <object object at 0x7f80e0221810> is not a valid _MethodDefault


### Providers

```
apache-airflow                           2.4.3
apache-airflow-providers-amazon          6.0.0
apache-airflow-providers-celery          3.0.0
apache-airflow-providers-cncf-kubernetes 4.4.0
apache-airflow-providers-common-sql      1.3.1
apache-airflow-providers-docker          3.2.0
apache-airflow-providers-elasticsearch   4.2.1
apache-airflow-providers-ftp             3.2.0
apache-airflow-providers-google          8.4.0
apache-airflow-providers-grpc            3.0.0
apache-airflow-providers-hashicorp       3.1.0
apache-airflow-providers-http            4.1.0
apache-airflow-providers-imap            3.1.0
apache-airflow-providers-microsoft-azure 4.3.0
apache-airflow-providers-mysql           3.2.1
apache-airflow-providers-odbc            3.1.2
apache-airflow-providers-pagerduty       3.1.0
apache-airflow-providers-postgres        5.2.2
apache-airflow-providers-redis           3.0.0
apache-airflow-providers-salesforce      5.2.0
apache-airflow-providers-sendgrid        3.0.0
apache-airflow-providers-sftp            4.1.0
apache-airflow-providers-slack           6.0.0
apache-airflow-providers-sqlite          3.3.1
apache-airflow-providers-ssh             3.2.0
```
I believe this a broader issue with google client that persists objects with `google.api_core.gapic_v1.method.DEFAULT` as a default parameter - those do not handle the deepcopy properly.

On one hand this can be easily fixed by memoing this object which can be safely assumed as a singleton, on the other hand it highlights a bigger problem - the assumption of operators having a proper deepcopy support is implicit, applies to all operators and isn't enforced in any of the unittests (and would be hard to enforce properly as some operators have a complicated state). 

Should we consider tackling the second part globally for all providers as a seprate bug?
This also impacts airflow 2.5.1; it mostly occurs when a operation fails; otherwise I have not seen it occur. 

Resolving patch:
diff --git a/airflow/providers/google/cloud/operators/cloud_base.py b/airflow/providers/google/cloud/operators/cloud_base.py
--- a/airflow/providers/google/cloud/operators/cloud_base.py
+++ b/airflow/providers/google/cloud/operators/cloud_base.py
@@ -18,6 +18,8 @@
 """This module contains a Google API base operator."""
 from __future__ import annotations
 
+from google.api_core.gapic_v1.method import DEFAULT
+
 from airflow.models import BaseOperator
 
 
@@ -27,4 +29,11 @@ class GoogleCloudBaseOperator(BaseOperator):
     on top of Google API client libraries.
     """
 
-    pass
+    def __deepcopy__(self, memo):
+        """
+        Updating the memo to fix the non-copyable global constant.
+        This constant can be specified in operator parameters as a retry configuration to indicate a default.
+        See https://github.com/apache/airflow/issues/28751 for details.
+        """
+        memo[id(DEFAULT)] = DEFAULT
+        return super().__deepcopy__(memo)


Test patch:
diff --git a/tests/providers/google/cloud/operators/test_cloud_base.py b/tests/providers/google/cloud/operators/test_cloud_base.py
new file mode 100644
--- /dev/null
+++ b/tests/providers/google/cloud/operators/test_cloud_base.py
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import copy
+import unittest
+
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
+from google.api_core.retry import Retry
+
+from airflow.providers.google.cloud.operators.cloud_base import GoogleCloudBaseOperator
+
+TASK_ID = "task-id"
+
+
+class GoogleSampleOperator(GoogleCloudBaseOperator):
+    def __init__(
+        self,
+        retry: Retry | _MethodDefault = DEFAULT,
+        config: dict | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.retry = retry
+        self.config = config
+
+
+class TestGoogleCloudBaseOperator(unittest.TestCase):
+    def test_handles_deepcopy_with_method_default(self):
+        op = GoogleSampleOperator(task_id=TASK_ID)
+        copied_op = copy.deepcopy(op)
+
+        self.assertEqual(copied_op.retry, DEFAULT)
+        self.assertEqual(copied_op.config, None)
+
+    def test_handles_deepcopy_with_non_default_retry(self):
+        op = GoogleSampleOperator(task_id=TASK_ID, retry=Retry(deadline=30), config={"config": "value"})
+        copied_op = copy.deepcopy(op)
+
+        self.assertEqual(copied_op.retry.deadline, 30)
+        self.assertEqual(copied_op.config, {"config": "value"})