Repository: apache/airflow

Issue:
Kubernetes Invalid executor_config, pod_override filled with Encoding.VAR
### Apache Airflow version

2.3.4

### What happened

Trying to start Kubernetes tasks using a `pod_override` results in pods not starting after upgrading from 2.3.2 to 2.3.4

The pod_override look very odd, filled with many Encoding.VAR objects, see following scheduler log:
```
{kubernetes_executor.py:550} INFO - Add task TaskInstanceKey(dag_id='commit_check', task_id='sync_and_build', run_id='5776-2-1662037155', try_number=1, map_index=-1) with command ['airflow', 'tasks', 'run', 'commit_check', 'sync_and_build', '5776-2-1662037155', '--local', '--subdir', 'DAGS_FOLDER/dag_on_commit.py'] with executor_config {'pod_override': {'Encoding.VAR': {'Encoding.VAR': {'Encoding.VAR': {'metadata': {'Encoding.VAR': {'annotations': {'Encoding.VAR': {}, 'Encoding.TYPE': 'dict'}}, 'Encoding.TYPE': 'dict'}, 'spec': {'Encoding.VAR': {'containers': REDACTED 'Encoding.TYPE': 'k8s.V1Pod'}, 'Encoding.TYPE': 'dict'}}
{kubernetes_executor.py:554} ERROR - Invalid executor_config for TaskInstanceKey(dag_id='commit_check', task_id='sync_and_build', run_id='5776-2-1662037155', try_number=1, map_index=-1)
```

Looking in the UI, the task get stuck in scheduled state forever. By clicking instance details, it shows similar state of the pod_override with many Encoding.VAR. 


This appears like a recent addition, in 2.3.4 via https://github.com/apache/airflow/pull/24356. 
@dstandish  do you understand if this is connected?


### What you think should happen instead

_No response_

### How to reproduce

_No response_

### Operating System

Debian GNU/Linux 11 (bullseye)

### Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-common-sql==1.1.0
apache-airflow-providers-docker==3.1.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.2.0
apache-airflow-providers-sqlite==3.2.0
kubernetes==23.6.0

### Deployment

Other Docker-based deployment

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
+1 seeing the same thing on 2.3.4, KubernetesExector.  Wanted to provide more info.

We run 40 DAGs with several thousand total daily tasks, and every day we have 5-10 tasks that get stuck in Queued state.

We see the following sequence in the logs:
First this error the first time the task attempts to run
```
Invalid executor_config for ["tenant_extraction_acme_recruiting_id_72","discovery_finder_on_29735_agg_month","scheduled__2022-08-31T11:00:00+00:00",1,-1]

```

The in a repeated loop we see logs like this:
```
{base_executor.py:211} INFO - task TaskInstanceKey(dag_id='tenant_extraction_acme_recruiting_id_72', task_id='discovery_finder_on_29735_agg_month', run_id='scheduled__2022-08-31T11:00:00+00:00', try_number=1, map_index=-1) is still running
 ```
for each stuck task, followed shortly by:
```
{base_executor.py:215} ERROR - could not queue task TaskInstanceKey(dag_id='tenant_extraction_acme_recruiting_id_72', task_id='discovery_finder_on_29735_agg_month', run_id='scheduled__2022-08-31T11:00:00+00:00', try_number=1, map_index=-1) (still running after 4 attempts)
```

This continues until the task is marked as failed in the airflow UI.


The executor_config on the invalid tasks looks like this:
```
{'pod_override': {<Encoding.VAR: '__var'>: {'spec': {'containers': [{'name': 'base', 'resources': {'limits': {}, 'requests': {'memory': '16Gi', 'cpu': '1'}}}]}}, <Encoding.TYPE: '__type'>: <DagAttributeTypes.POD: 'k8s.V1Pod'>}}
```


Which is odd, because for the thousands of unaffected tasks it looks like this:
```
{'pod_override': {'api_version': None, 'kind': None, 'metadata': None, 'spec': {'active_deadline_seconds': None, 'affinity': None, 'automount_service_account_token': None, 'containers': [{'args': None, 'command': None, 'env': None, 'env_from': None, 'image': None, 'image_pull_policy': None, 'lifecycle': None, 'liveness_probe': None, 'name': 'base', 'ports': None, 'readiness_probe': None, 'resources': {'limits': {}, 'requests': {'cpu': '1', 'memory': '16Gi'}}, 'security_context': None, 'startup_probe': None, 'stdin': None, 'stdin_once': None, 'termination_message_path': None, 'termination_message_policy': None, 'tty': None, 'volume_devices': None, 'volume_mounts': None, 'working_dir': None}], 'dns_config': None, 'dns_policy': None, 'enable_service_links': None, 'ephemeral_containers': None, 'host_aliases': None, 'host_ipc': None, 'host_network': None, 'host_pid': None, 'hostname': None, 'image_pull_secrets': None, 'init_containers': None, 'node_name': None, 'node_selector': None, 'os': None, 'overhead': None, 'preemption_policy': None, 'priority': None, 'priority_class_name': None, 'readiness_gates': None, 'restart_policy': None, 'runtime_class_name': None, 'scheduler_name': None, 'security_context': None, 'service_account': None, 'service_account_name': None, 'set_hostname_as_fqdn': None, 'share_process_namespace': None, 'subdomain': None, 'termination_grace_period_seconds': None, 'tolerations': None, 'topology_spread_constraints': None, 'volumes': None}, 'status': None}}
```
@dstandish this seems to introduce non-deterministic behavior for any pod that uses KubernetesExecuter and specifies a pod override. Only pods that specify an override seem to be impacted. And the issue seems to be that the deserialization of the executer config is broken in some way in the new system.

This can be easily repro'd in the airflow web UI by hitting reload repeatedly on the task instance details pane for a task that uses KubernetesExecuter and specifies an executor config. I was even able to cause the web ui to crash as a result.


[airflow_pickle_bug.txt](https://github.com/apache/airflow/files/9481532/airflow_pickle_bug.txt)



Resolving patch:
diff --git a/airflow/utils/sqlalchemy.py b/airflow/utils/sqlalchemy.py
--- a/airflow/utils/sqlalchemy.py
+++ b/airflow/utils/sqlalchemy.py
@@ -15,7 +15,7 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-
+import copy
 import datetime
 import json
 import logging
@@ -166,9 +166,10 @@ def bind_processor(self, dialect):
         super_process = super().bind_processor(dialect)
 
         def process(value):
-            if isinstance(value, dict) and 'pod_override' in value:
-                value['pod_override'] = BaseSerialization.serialize(value['pod_override'])
-            return super_process(value)
+            val_copy = copy.copy(value)
+            if isinstance(val_copy, dict) and 'pod_override' in val_copy:
+                val_copy['pod_override'] = BaseSerialization.serialize(val_copy['pod_override'])
+            return super_process(val_copy)
 
         return process
 


Test patch:
diff --git a/tests/utils/test_sqlalchemy.py b/tests/utils/test_sqlalchemy.py
--- a/tests/utils/test_sqlalchemy.py
+++ b/tests/utils/test_sqlalchemy.py
@@ -19,7 +19,6 @@
 import datetime
 import pickle
 import unittest
-from copy import copy
 from unittest import mock
 from unittest.mock import MagicMock
 
@@ -31,7 +30,7 @@
 
 from airflow import settings
 from airflow.models import DAG
-from airflow.serialization.enums import Encoding
+from airflow.serialization.enums import DagAttributeTypes, Encoding
 from airflow.serialization.serialized_objects import BaseSerialization
 from airflow.settings import Session
 from airflow.utils.sqlalchemy import ExecutorConfigType, nowait, prohibit_commit, skip_locked, with_row_locks
@@ -239,10 +238,21 @@ def tearDown(self):
 
 class TestExecutorConfigType:
     @pytest.mark.parametrize(
-        'input',
-        ['anything', {'pod_override': TEST_POD}],
+        'input, expected',
+        [
+            ('anything', 'anything'),
+            (
+                {'pod_override': TEST_POD},
+                {
+                    "pod_override": {
+                        "__var": {"spec": {"containers": [{"name": "base"}]}},
+                        "__type": DagAttributeTypes.POD,
+                    }
+                },
+            ),
+        ],
     )
-    def test_bind_processor(self, input):
+    def test_bind_processor(self, input, expected):
         """
         The returned bind processor should pickle the object as is, unless it is a dictionary with
         a pod_override node, in which case it should run it through BaseSerialization.
@@ -251,10 +261,8 @@ def test_bind_processor(self, input):
         mock_dialect = MagicMock()
         mock_dialect.dbapi = None
         process = config_type.bind_processor(mock_dialect)
-        expected = copy(input)
-        if 'pod_override' in input:
-            expected['pod_override'] = BaseSerialization.serialize(input['pod_override'])
         assert pickle.loads(process(input)) == expected
+        assert pickle.loads(process(input)) == expected, "should should not mutate variable"
 
     @pytest.mark.parametrize(
         'input',