Repository: apache/airflow

Issue:
Webserver fails to pull secrets from Hashicorp Vault on start up
### Apache Airflow version

2.3.4

### What happened

Since upgrading to Airflow 2.3.4 our webserver fails on start up to pull secrets from our Vault instance.  Setting AIRFLOW__WEBSERVER_WORKERS = 1 allowed the webserver to start up successfully, but reverting the change added here [https://github.com/apache/airflow/pull/25556](url) was the only way we found to fix the issue without adjusting the webserver's worker count.

### What you think should happen instead

The airflow webserver should be able to successfully read from Vault with AIRFLOW__WEBSERVERS__WORKERS > 1.

### How to reproduce

Star a Webserver instance set to authenticate with Vault using the approle method and AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_SECRET and AIRFLOW__WEBSERVER__SECRET_KEY_SECRET set.  The webserver should fail to initialize all of the gunicorn workers and exit.

### Operating System

Fedora 29

### Versions of Apache Airflow Providers

apache-airflow-providers-hashicorp==3.1.0

### Deployment

Docker-Compose

### Deployment details

Python 3.9.13
Vault 1.9.4

### Anything else

None

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

@pdebelak @potiuk  is there anything extra we can provide here? 

It was a bit of a surprise to us that we needed to revert https://github.com/apache/airflow/pull/25556 to run the webserver on 2.3.4, it was one of the PRs we were looking forward to. But it's not a 100% clear to us why this PR broke the webserver/vault interaction.
@notatallshaw-work @Wilder60 I don't use approle, but this is mysterious. Are there any other clues you can provide? All other configuration is at default values? Do you get a stack trace when the workers crash?
Yes we do get a stack trace it's a `http.client.BadStatusLine`, after staring at it a while _I think_ that the vault clients in the different processes are all reading the same HTTP responses (in the order that came from the vault server), rather than reading their own individual responses. 

Here is the full exception log (I've had to sanitize some of the values): https://gist.github.com/notatallshaw-work/6c1a77b4341caa653a231190bd64cdaa

Running just 1 worker or reverting the PR both fixes this exception.
Yeah, I don't really understand how/why that might be happening, but that's clearly a problem.. It's possible that this change is too fundamental to be safe for all configurations and we should just get rid of this caching and deal with the multiple logins.

@potiuk  - what do you think? Should I just revert this caching change altogether?
I think so. It has proven to be more problematic than we both thought and the benefit is limited, I think that using external caching mechanism from the respective secrets providers is a much more robust way.

We should treat it as a failed experiment I think. 

Resolving patch:
diff --git a/airflow/configuration.py b/airflow/configuration.py
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -1544,20 +1544,16 @@ def ensure_secrets_loaded() -> List[BaseSecretsBackend]:
 def get_custom_secret_backend() -> Optional[BaseSecretsBackend]:
     """Get Secret Backend if defined in airflow.cfg"""
     secrets_backend_cls = conf.getimport(section='secrets', key='backend')
-    if secrets_backend_cls:
-        backends: Any = conf.get(section='secrets', key='backend_kwargs', fallback='{}')
-        return _custom_secrets_backend(secrets_backend_cls, backends)
-    return None
 
+    if secrets_backend_cls:
+        try:
+            backends: Any = conf.get(section='secrets', key='backend_kwargs', fallback='{}')
+            alternative_secrets_config_dict = json.loads(backends)
+        except JSONDecodeError:
+            alternative_secrets_config_dict = {}
 
-@functools.lru_cache(maxsize=2)
-def _custom_secrets_backend(secrets_backend_cls, backend_kwargs):
-    """Separate function to create secrets backend instance to allow caching"""
-    try:
-        alternative_secrets_config_dict = json.loads(backend_kwargs)
-    except JSONDecodeError:
-        alternative_secrets_config_dict = {}
-    return secrets_backend_cls(**alternative_secrets_config_dict)
+        return secrets_backend_cls(**alternative_secrets_config_dict)
+    return None
 
 
 def initialize_secrets_backends() -> List[BaseSecretsBackend]:


Test patch:
diff --git a/tests/core/test_configuration.py b/tests/core/test_configuration.py
--- a/tests/core/test_configuration.py
+++ b/tests/core/test_configuration.py
@@ -33,7 +33,6 @@
 from airflow.configuration import (
     AirflowConfigException,
     AirflowConfigParser,
-    _custom_secrets_backend,
     conf,
     expand_env_var,
     get_airflow_config,
@@ -297,7 +296,6 @@ def test_hidding_of_sensitive_config_values(self):
     def test_config_raise_exception_from_secret_backend_connection_error(self, mock_hvac):
         """Get Config Value from a Secret Backend"""
 
-        _custom_secrets_backend.cache_clear()
         mock_client = mock.MagicMock()
         # mock_client.side_effect = AirflowConfigException
         mock_hvac.Client.return_value = mock_client
@@ -324,7 +322,6 @@ def test_config_raise_exception_from_secret_backend_connection_error(self, mock_
             ),
         ):
             test_conf.get('test', 'sql_alchemy_conn')
-        _custom_secrets_backend.cache_clear()
 
     def test_getboolean(self):
         """Test AirflowConfigParser.getboolean"""
@@ -1300,123 +1297,3 @@ def test_conf_as_dict_when_deprecated_value_in_secrets_disabled_config(
                 conf.read_dict(dictionary=cfg_dict)
                 os.environ.clear()
                 assert conf.get('database', 'sql_alchemy_conn') == f'sqlite:///{HOME_DIR}/airflow/airflow.db'
-
-    @mock.patch("airflow.providers.hashicorp._internal_client.vault_client.hvac")
-    @conf_vars(
-        {
-            ("secrets", "backend"): "airflow.providers.hashicorp.secrets.vault.VaultBackend",
-            ("secrets", "backend_kwargs"): '{"url": "http://127.0.0.1:8200", "token": "token"}',
-        }
-    )
-    def test_config_from_secret_backend_caches_instance(self, mock_hvac):
-        """Get Config Value from a Secret Backend"""
-        _custom_secrets_backend.cache_clear()
-
-        test_config = '''[test]
-sql_alchemy_conn_secret = sql_alchemy_conn
-secret_key_secret = secret_key
-'''
-        test_config_default = '''[test]
-sql_alchemy_conn = airflow
-secret_key = airflow
-'''
-
-        mock_client = mock.MagicMock()
-        mock_hvac.Client.return_value = mock_client
-
-        def fake_read_secret(path, mount_point, version):
-            if path.endswith('sql_alchemy_conn'):
-                return {
-                    'request_id': '2d48a2ad-6bcb-e5b6-429d-da35fdf31f56',
-                    'lease_id': '',
-                    'renewable': False,
-                    'lease_duration': 0,
-                    'data': {
-                        'data': {'value': 'fake_conn'},
-                        'metadata': {
-                            'created_time': '2020-03-28T02:10:54.301784Z',
-                            'deletion_time': '',
-                            'destroyed': False,
-                            'version': 1,
-                        },
-                    },
-                    'wrap_info': None,
-                    'warnings': None,
-                    'auth': None,
-                }
-            if path.endswith('secret_key'):
-                return {
-                    'request_id': '2d48a2ad-6bcb-e5b6-429d-da35fdf31f56',
-                    'lease_id': '',
-                    'renewable': False,
-                    'lease_duration': 0,
-                    'data': {
-                        'data': {'value': 'fake_key'},
-                        'metadata': {
-                            'created_time': '2020-03-28T02:10:54.301784Z',
-                            'deletion_time': '',
-                            'destroyed': False,
-                            'version': 1,
-                        },
-                    },
-                    'wrap_info': None,
-                    'warnings': None,
-                    'auth': None,
-                }
-
-        mock_client.secrets.kv.v2.read_secret_version.side_effect = fake_read_secret
-
-        test_conf = AirflowConfigParser(default_config=parameterized_config(test_config_default))
-        test_conf.read_string(test_config)
-        test_conf.sensitive_config_values = test_conf.sensitive_config_values | {
-            ('test', 'sql_alchemy_conn'),
-            ('test', 'secret_key'),
-        }
-
-        assert 'fake_conn' == test_conf.get('test', 'sql_alchemy_conn')
-        mock_hvac.Client.assert_called_once()
-        assert 'fake_key' == test_conf.get('test', 'secret_key')
-        mock_hvac.Client.assert_called_once()
-        _custom_secrets_backend.cache_clear()
-
-    @mock.patch('airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend')
-    @conf_vars(
-        {
-            (
-                "secrets",
-                "backend",
-            ): "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend",
-            (
-                "secrets",
-                "backend_kwargs",
-            ): '{"connections_prefix": "airflow-connections", '
-            '"gcp_keyfile_dict": {"keyfilearg1": "keyfileval1", "keyfilearg2": "keyfileval2"}}',
-        }
-    )
-    def test_config_from_secret_backend_allows_dict_arguments(self, mock_backend):
-        """Get Config Value from a Secret Backend with dict arguments"""
-        _custom_secrets_backend.cache_clear()
-
-        test_config = '''[test]
-sql_alchemy_conn_secret = sql_alchemy_conn
-'''
-        test_config_default = '''[test]
-sql_alchemy_conn = airflow
-'''
-
-        test_conf = AirflowConfigParser(default_config=parameterized_config(test_config_default))
-        test_conf.read_string(test_config)
-        test_conf.sensitive_config_values = test_conf.sensitive_config_values | {
-            ('test', 'sql_alchemy_conn'),
-        }
-
-        mock_backend.return_value.get_config.return_value = 'google_conf'
-
-        assert 'google_conf' == test_conf.get('test', 'sql_alchemy_conn')
-
-        mock_backend.assert_called_with(
-            connections_prefix='airflow-connections',
-            gcp_keyfile_dict={'keyfilearg1': 'keyfileval1', 'keyfilearg2': 'keyfileval2'},
-        )
-
-        _custom_secrets_backend.cache_clear()