Repository: great-expectations/great_expectations

Issue:
GE is very slow when aws secret store is integrated
**Describe the bug**
When I have integrated aws secret store for my db crendentials GE has become very slow. I have three checkpoints (50+ expectations) orchestrated via github actions in my current project. Before secret store integration it took around 5 mins to finish, after secret store integration it is almost 24 mins.

Also opening the notebook to update expectation suite takes around 5 mins now. Before it was immediate action. I guess because GE is substituting the variables for so long.

I have also tried to check how the `great_expectations.yml` file looks after variables substitution, so I tried this:

```import great_expectations as ge
context = ge.data_context.DataContext()
c = context.get_config_with_variables_substituted()
print(c.to_yaml_str())
```

Just this few lines of code took almost two minutes. Then I got substituted `great_expectations.yml` file which looks correct. I just do not know why the variable substitution takes so long. I tried to access the secrets directly via boto3 and I get all of them immediately.

I also tried fresh GE installation and simple expectation suite + checkpoint but it is also slow in this scenario. Without secret store the checkpoint ran for 49 secs. With secret store integration and getting the db credentials from there it ran for 3mins and 45 seconds.

**To Reproduce**
Steps to reproduce the behavior:
1. install great expectations with aws secret store extension `pip install great_expectations[aws_secrets]`
2. create new secret in aws secret store and store there DB credentials
3. Define some DB source ( I used Snowflake)
4. Create simple expectation suite
5. Create checkpoint for it
6. run the checkpoint to see how long it takes without secret store
7. replace credentials for DB access in `great_expectations.yml` to get them from aws secret store. How it looks at my end

```
datasources:
  sf_database:
    module_name: great_expectations.datasource
    data_asset_type:
      class_name: SqlAlchemyDataset
      module_name: great_expectations.dataset
    class_name: SqlAlchemyDatasource
    credentials:
      drivername: snowflake
      host: secret|arn:aws:secretsmanager:eu-west-1:account_id:secret:/my_secret|ge_sf_host
      database: secret|arn:aws:secretsmanager:eu-west-1:account_id:secret:/my_secret|ge_sf_db_${GE_ENV}
      username: secret|arn:aws:secretsmanager:eu-west-1:account_id:secret:/my_secret|ge_sf_username_${GE_ENV}
      password: secret|arn:aws:secretsmanager:eu-west-1:account_id:secret:/my_secret|ge_sf_password_${GE_ENV}
      query:
        schema: secret|arn:aws:secretsmanager:eu-west-1:account_id:secret:/my_secret|ge_sf_schema_${GE_ENV}
        warehouse: secret|arn:aws:secretsmanager:eu-west-1:account_id:secret:/my_secret|ge_sf_warehouse_${GE_ENV}
        role: secret|arn:aws:secretsmanager:eu-west-1:account_id:secret:/my_secret|ge_sf_role_${GE_ENV}
```
8. Run the checkpoint again and compare running times. It should be much slower with secret store

**Expected behavior**
I would expect similar performance no matter where the credentials are taken from.

**Environment (please complete the following information):**
 - Operating System: MacOS
 - Great Expectations Version: 0.13.19 with aws secret store extension

**Additional context**
Add any other context about the problem here.



Discussion:
I have experienced exactly the same issue when using Azure Key Vault and when switching to environment variables the waiting times are gone.
@tomas-sobotik Thank you for such a detailed report! Adding this to the core team's backlog. This might take a while, so any help in the meantime will be very much appreciated.

Resolving patch:
diff --git a/great_expectations/data_context/util.py b/great_expectations/data_context/util.py
--- a/great_expectations/data_context/util.py
+++ b/great_expectations/data_context/util.py
@@ -7,6 +7,7 @@
 import re
 import warnings
 from collections import OrderedDict
+from functools import lru_cache
 from typing import Optional
 from urllib.parse import urlparse
 
@@ -233,6 +234,7 @@ def substitute_config_variable(
     return template_str
 
 
+@lru_cache(maxsize=None)
 def substitute_value_from_secret_store(value):
     """
     This method takes a value, tries to parse the value to fetch a secret from a secret manager


Test patch: