Repository: apache/airflow

Issue:
Unable to configure Google Secrets Manager in 2.3.4
### Apache Airflow version

2.3.4

### What happened

I am attempting to configure a Google Secrets Manager secrets backend using the `gcp_keyfile_dict` param in a `.env` file with the following ENV Vars:

```
AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": <json-keyfile>}'
```

In previous versions including 2.3.3 this worked without issue

After upgrading to Astro Runtime 5.0.8 I get the following error taken from the scheduler container logs. The scheduler, webserver, and triggerer are continually restarting

```
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.9/site-packages/airflow/__init__.py", line 35, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.9/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1618, in <module>
    secrets_backend_list = initialize_secrets_backends()
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1540, in initialize_secrets_backends
    custom_secret_backend = get_custom_secret_backend()
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1523, in get_custom_secret_backend
    return _custom_secrets_backend(secrets_backend_cls, **alternative_secrets_config_dict)
TypeError: unhashable type: 'dict'
```





### What you think should happen instead

Containers should remain healthy and the secrets backend should successfully be added

### How to reproduce

`astro dev init` a fresh project

Dockerfile:
`FROM quay.io/astronomer/astro-runtime:5.0.8`


`.env` file: 
```
AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": <service-acct-json-keyfile>}'
```

`astro dev start`


### Operating System

macOS 11.6.8

### Versions of Apache Airflow Providers

[apache-airflow-providers-google](https://airflow.apache.org/docs/apache-airflow-providers-google/8.1.0/) 8.1.0

### Deployment

Astronomer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
@pdebelak - I think this is caused by the LRU cache introduced in https://github.com/apache/airflow/pull/25556 - is it possible you take a look and see if it can be fixed/workarounded ?
I believe the problem is that dict-indeed is not hashable, and you can pass the dict as parameter of the secret backend configuration. 

For now, I don't see an easy workaround other than using `gcp_key_path` and putting the key in the same path in your workers  - would that be a feasible workaround for now @aspain ? 
With an Astronomer project I don't have access to the workers (other than locally) and would have to include the keyfile in my repository the project deploys from, ideally the keyfile would not need to be in the repository

In my local environment I am using a .env file but not pushing it to the repo, and in the Astro UI I am able to add environment variables 
Yes, this is related to the new `lru_cache` in 2.3.4, I didn't realize this would break in this way. There isn't an easy workaround. We might need to revert that change in this case and add a test to make sure we don't break it in the same way again.
I see a fix for this that I will PR, but I don't see a workaround for version 2.3.4 if you have a `AIRFLOW__SECRETS__BACKEND_KWARGS` containing a nested dictionary.
yeah. There is no easy workaround I could see for that one. I will raise it to the release mgmt team (we have one more bug that might make us do 2.3.5 before we reelase 2.4.0. In the meantime @pdebelak - looking forward to a fix :D

Resolving patch:
diff --git a/airflow/configuration.py b/airflow/configuration.py
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -1545,19 +1545,18 @@ def get_custom_secret_backend() -> Optional[BaseSecretsBackend]:
     """Get Secret Backend if defined in airflow.cfg"""
     secrets_backend_cls = conf.getimport(section='secrets', key='backend')
     if secrets_backend_cls:
-        try:
-            backends: Any = conf.get(section='secrets', key='backend_kwargs', fallback='{}')
-            alternative_secrets_config_dict = json.loads(backends)
-        except JSONDecodeError:
-            alternative_secrets_config_dict = {}
-
-        return _custom_secrets_backend(secrets_backend_cls, **alternative_secrets_config_dict)
+        backends: Any = conf.get(section='secrets', key='backend_kwargs', fallback='{}')
+        return _custom_secrets_backend(secrets_backend_cls, backends)
     return None
 
 
 @functools.lru_cache(maxsize=2)
-def _custom_secrets_backend(secrets_backend_cls, **alternative_secrets_config_dict):
+def _custom_secrets_backend(secrets_backend_cls, backend_kwargs):
     """Separate function to create secrets backend instance to allow caching"""
+    try:
+        alternative_secrets_config_dict = json.loads(backend_kwargs)
+    except JSONDecodeError:
+        alternative_secrets_config_dict = {}
     return secrets_backend_cls(**alternative_secrets_config_dict)
 
 


Test patch:
diff --git a/tests/core/test_configuration.py b/tests/core/test_configuration.py
--- a/tests/core/test_configuration.py
+++ b/tests/core/test_configuration.py
@@ -1378,3 +1378,45 @@ def fake_read_secret(path, mount_point, version):
         assert 'fake_key' == test_conf.get('test', 'secret_key')
         mock_hvac.Client.assert_called_once()
         _custom_secrets_backend.cache_clear()
+
+    @mock.patch('airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend')
+    @conf_vars(
+        {
+            (
+                "secrets",
+                "backend",
+            ): "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend",
+            (
+                "secrets",
+                "backend_kwargs",
+            ): '{"connections_prefix": "airflow-connections", '
+            '"gcp_keyfile_dict": {"keyfilearg1": "keyfileval1", "keyfilearg2": "keyfileval2"}}',
+        }
+    )
+    def test_config_from_secret_backend_allows_dict_arguments(self, mock_backend):
+        """Get Config Value from a Secret Backend with dict arguments"""
+        _custom_secrets_backend.cache_clear()
+
+        test_config = '''[test]
+sql_alchemy_conn_secret = sql_alchemy_conn
+'''
+        test_config_default = '''[test]
+sql_alchemy_conn = airflow
+'''
+
+        test_conf = AirflowConfigParser(default_config=parameterized_config(test_config_default))
+        test_conf.read_string(test_config)
+        test_conf.sensitive_config_values = test_conf.sensitive_config_values | {
+            ('test', 'sql_alchemy_conn'),
+        }
+
+        mock_backend.return_value.get_config.return_value = 'google_conf'
+
+        assert 'google_conf' == test_conf.get('test', 'sql_alchemy_conn')
+
+        mock_backend.assert_called_with(
+            connections_prefix='airflow-connections',
+            gcp_keyfile_dict={'keyfilearg1': 'keyfileval1', 'keyfilearg2': 'keyfileval2'},
+        )
+
+        _custom_secrets_backend.cache_clear()