Repository: great-expectations/great_expectations

Issue:
password skipping config substitution
**Describe the bug**
When using environment variables to provide datasource credentials in great_expectations.yml, example:
```
    credentials:
      drivername: postgresql+psycopg2
      host: ${GE_REDSHIFT_HOST}
      port: ${GE_REDSHIFT_PORT}
      username: ${GE_REDSHIFT_USER}
      password: ${GE_REDSHIFT_PASSWORD}
      database: ${GE_REDSHIFT_DB}

```
the password is reading "${GE_REDSHIFT_PASSWORD}" as the `password` value due to [this bugfix](https://github.com/great-expectations/great_expectations/commit/9d85b7e0543d170e73df99dcc985c649a0f395bb).

**To Reproduce**
Steps to reproduce the behavior:
1. Use an environment variable in your datasource credentials' password
2. Run anything that needs to connect to the datasource. For example, you can create a new suite or edit an old one and run the first Jupyter Notebook cell. You'll get an authentication error like this:
<img width="1198" alt="Captura de pantalla 2020-10-21 a las 18 11 02" src="https://user-images.githubusercontent.com/40449560/96747904-28df2900-13c9-11eb-901b-c1c01ffcc1bd.png">

<img width="1396" alt="Captura de pantalla 2020-10-21 a las 18 11 55" src="https://user-images.githubusercontent.com/40449560/96747889-2381de80-13c9-11eb-8771-933e218c435f.png">

**Expected behavior**
Not to crash and also read the value of the environment variable (${GE_REDSHIFT_PASSWORD} in this case) as the `password` value for the datasource credentials.

**Environment (please complete the following information):**
 - OS: All
 - GE Version: 0.12.6

**Additional context**
@anthonyburdi 


Discussion:
+1 on this, am currently working around this by generating the yaml file and interpolating in the password from the environment manually.
+1 
@EliPino @whatsrupp @martinheld This is indeed a case of a fix breaking something else. We will fix this in a more robust way and cut a release.

Resolving patch:
diff --git a/great_expectations/data_context/data_context.py b/great_expectations/data_context/data_context.py
--- a/great_expectations/data_context/data_context.py
+++ b/great_expectations/data_context/data_context.py
@@ -11,6 +11,7 @@
 import uuid
 import warnings
 import webbrowser
+from collections import OrderedDict
 from typing import Dict, List, Optional, Union
 
 from dateutil.parser import parse
@@ -209,6 +210,7 @@ class BaseDataContext:
         os.path.expanduser("~/.great_expectations/great_expectations.conf"),
         "/etc/great_expectations.conf",
     ]
+    DOLLAR_SIGN_ESCAPE_STRING = r"\$"
 
     @classmethod
     def validate_config(cls, project_config):
@@ -716,7 +718,9 @@ def get_config_with_variables_substituted(self, config=None):
             config = self._project_config
 
         substituted_config_variables = substitute_all_config_variables(
-            dict(self._load_config_variables_file()), dict(os.environ)
+            dict(self._load_config_variables_file()),
+            dict(os.environ),
+            self.DOLLAR_SIGN_ESCAPE_STRING,
         )
 
         substitutions = {
@@ -726,9 +730,40 @@ def get_config_with_variables_substituted(self, config=None):
         }
 
         return DataContextConfig(
-            **substitute_all_config_variables(config, substitutions)
+            **substitute_all_config_variables(
+                config, substitutions, self.DOLLAR_SIGN_ESCAPE_STRING
+            )
         )
 
+    def escape_all_config_variables(
+        self,
+        value: Union[str, dict, list],
+        dollar_sign_escape_string: str = DOLLAR_SIGN_ESCAPE_STRING,
+    ) -> Union[str, dict, list]:
+        """
+        Replace all `$` characters with the DOLLAR_SIGN_ESCAPE_STRING
+
+        Args:
+            value: config variable value
+            dollar_sign_escape_string: replaces instances of `$`
+
+        Returns:
+            input value with all `$` characters replaced with the escape string
+        """
+
+        if isinstance(value, dict) or isinstance(value, OrderedDict):
+            return {
+                k: self.escape_all_config_variables(v, dollar_sign_escape_string)
+                for k, v in value.items()
+            }
+
+        elif isinstance(value, list):
+            return [
+                self.escape_all_config_variables(v, dollar_sign_escape_string)
+                for v in value
+            ]
+        return value.replace("$", dollar_sign_escape_string)
+
     def save_config_variable(self, config_variable_name, value):
         """Save config variable value
 
@@ -740,6 +775,7 @@ def save_config_variable(self, config_variable_name, value):
             None
         """
         config_variables = self._load_config_variables_file()
+        value = self.escape_all_config_variables(value, self.DOLLAR_SIGN_ESCAPE_STRING)
         config_variables[config_variable_name] = value
         config_variables_filepath = self.get_config().config_variables_file_path
         if not config_variables_filepath:
diff --git a/great_expectations/data_context/util.py b/great_expectations/data_context/util.py
--- a/great_expectations/data_context/util.py
+++ b/great_expectations/data_context/util.py
@@ -99,56 +99,70 @@ def format_dict_for_error_message(dict_):
     return "\n\t".join("\t\t".join((str(key), str(dict_[key]))) for key in dict_)
 
 
-def substitute_config_variable(template_str, config_variables_dict):
+def substitute_config_variable(
+    template_str, config_variables_dict, dollar_sign_escape_string: str = r"\$"
+):
     """
     This method takes a string, and if it contains a pattern ${SOME_VARIABLE} or $SOME_VARIABLE,
     returns a string where the pattern is replaced with the value of SOME_VARIABLE,
-    otherwise returns the string unchanged.
+    otherwise returns the string unchanged. These patterns are case sensitive. There can be multiple
+    patterns in a string, e.g. all 3 will be substituted in the following:
+    $SOME_VARIABLE${some_OTHER_variable}$another_variable
 
     If the environment variable SOME_VARIABLE is set, the method uses its value for substitution.
     If it is not set, the value of SOME_VARIABLE is looked up in the config variables store (file).
     If it is not found there, the input string is returned as is.
 
+    If the value to substitute is not a string, it is returned as-is.
+
+    If the value to substitute begins with dollar_sign_escape_string it is not substituted.
+
     :param template_str: a string that might or might not be of the form ${SOME_VARIABLE}
             or $SOME_VARIABLE
     :param config_variables_dict: a dictionary of config variables. It is loaded from the
             config variables store (by default, "uncommitted/config_variables.yml file)
-    :return:
+    :param dollar_sign_escape_string: a string that will be used in place of a `$` when substitution
+            is not desired.
+
+    :return: a string with values substituted, or the same object if template_str is not a string.
     """
+
     if template_str is None:
         return template_str
 
+    # 1. Make substitutions for non-escaped patterns
     try:
-        match = re.search(r"\$\{(.*?)\}", template_str) or re.search(
-            r"\$([_a-z][_a-z0-9]*)", template_str
+        match = re.finditer(
+            r"(?<!\\)\$\{(.*?)\}|(?<!\\)\$([_a-zA-Z][_a-zA-Z0-9]*)", template_str
         )
     except TypeError:
         # If the value is not a string (e.g., a boolean), we should return it as is
         return template_str
 
-    if match:
-        config_variable_value = config_variables_dict.get(match.group(1))
+    for m in match:
+        # Match either the first group e.g. ${Variable} or the second e.g. $Variable
+        config_variable_name = m.group(1) or m.group(2)
+        config_variable_value = config_variables_dict.get(config_variable_name)
+
         if config_variable_value is not None:
-            if match.start() == 0 and match.end() == len(template_str):
+            if not isinstance(config_variable_value, str):
                 return config_variable_value
-            else:
-                return (
-                    template_str[: match.start()]
-                    + config_variable_value
-                    + template_str[match.end() :]
-                )
-
-        raise MissingConfigVariableError(
-            f"""\n\nUnable to find a match for config substitution variable: `{match.group(1)}`.
+            template_str = template_str.replace(m.group(), config_variable_value)
+        else:
+            raise MissingConfigVariableError(
+                f"""\n\nUnable to find a match for config substitution variable: `{config_variable_name}`.
 Please add this missing variable to your `uncommitted/config_variables.yml` file or your environment variables.
 See https://great-expectations.readthedocs.io/en/latest/reference/data_context_reference.html#managing-environment-and-secrets""",
-            missing_config_variable=match.group(1),
-        )
+                missing_config_variable=config_variable_name,
+            )
 
-    return template_str
+    # 2. Replace the "$"'s that had been escaped
+    return template_str.replace(dollar_sign_escape_string, "$")
 
 
-def substitute_all_config_variables(data, replace_variables_dict):
+def substitute_all_config_variables(
+    data, replace_variables_dict, dollar_sign_escape_string: str = r"\$"
+):
     """
     Substitute all config variables of the form ${SOME_VARIABLE} in a dictionary-like
     config object for their values.
@@ -165,15 +179,15 @@ def substitute_all_config_variables(data, replace_variables_dict):
     if isinstance(data, dict) or isinstance(data, OrderedDict):
         return {
             k: substitute_all_config_variables(v, replace_variables_dict)
-            if k.lower() not in {"password"}
-            else v
             for k, v in data.items()
         }
     elif isinstance(data, list):
         return [
             substitute_all_config_variables(v, replace_variables_dict) for v in data
         ]
-    return substitute_config_variable(data, replace_variables_dict)
+    return substitute_config_variable(
+        data, replace_variables_dict, dollar_sign_escape_string
+    )
 
 
 def file_relative_path(dunderfile, relative_path):


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1958,6 +1958,7 @@ def empty_data_context(tmp_path_factory):
 @pytest.fixture
 def empty_data_context_with_config_variables(monkeypatch, empty_data_context):
     monkeypatch.setenv("FOO", "BAR")
+    monkeypatch.setenv("REPLACE_ME_ESCAPED_ENV", "ive_been_$--replaced")
     root_dir = empty_data_context.root_directory
     ge_config_path = file_relative_path(
         __file__, "./test_fixtures/great_expectations_basic_with_variables.yml",
diff --git a/tests/data_context/conftest.py b/tests/data_context/conftest.py
--- a/tests/data_context/conftest.py
+++ b/tests/data_context/conftest.py
@@ -27,6 +27,7 @@ def data_context_without_config_variables_filepath_configured(tmp_path_factory):
 @pytest.fixture()
 def data_context_with_variables_in_config(tmp_path_factory, monkeypatch):
     monkeypatch.setenv("FOO", "BAR")
+    monkeypatch.setenv("REPLACE_ME_ESCAPED_ENV", "ive_been_$--replaced")
     # This data_context is *manually* created to have the config we want, vs created with DataContext.create
     project_path = str(tmp_path_factory.mktemp("data_context"))
     context_path = os.path.join(project_path, "great_expectations")
diff --git a/tests/data_context/test_data_context_config_variables.py b/tests/data_context/test_data_context_config_variables.py
--- a/tests/data_context/test_data_context_config_variables.py
+++ b/tests/data_context/test_data_context_config_variables.py
@@ -1,4 +1,5 @@
 import os
+from collections import OrderedDict
 
 import pytest
 from ruamel.yaml import YAML, YAMLError
@@ -99,6 +100,80 @@ def test_setting_config_variables_is_visible_immediately(
         "batch_kwargs_generators"
     ]["mygenerator"]["reader_options"]["test_variable_sub2"] == {"n1": "v1"}
 
+    # verify the same for escaped variables
+    context.save_config_variable(
+        "escaped_password", "this_is_$mypassword_escape_the_$signs"
+    )
+    dict_to_escape = {
+        "drivername": "po$tgresql",
+        "host": "localhost",
+        "port": "5432",
+        "username": "postgres",
+        "password": "$pas$wor$d1$",
+        "database": "postgres",
+    }
+    context.save_config_variable(
+        "escaped_password_dict", dict_to_escape,
+    )
+
+    context._project_config["datasources"]["mydatasource"]["batch_kwargs_generators"][
+        "mygenerator"
+    ]["reader_options"]["test_variable_sub_escaped"] = "${escaped_password}"
+    context._project_config["datasources"]["mydatasource"]["batch_kwargs_generators"][
+        "mygenerator"
+    ]["reader_options"]["test_variable_sub_escaped_dict"] = "${escaped_password_dict}"
+
+    assert (
+        context.get_config().datasources["mydatasource"]["batch_kwargs_generators"][
+            "mygenerator"
+        ]["reader_options"]["test_variable_sub_escaped"]
+        == "${escaped_password}"
+    )
+    assert (
+        context.get_config().datasources["mydatasource"]["batch_kwargs_generators"][
+            "mygenerator"
+        ]["reader_options"]["test_variable_sub_escaped_dict"]
+        == "${escaped_password_dict}"
+    )
+
+    # Ensure that the value saved in config variables has escaped the $
+    config_variables_with_escaped_vars = context._load_config_variables_file()
+    assert (
+        config_variables_with_escaped_vars["escaped_password"]
+        == r"this_is_\$mypassword_escape_the_\$signs"
+    )
+    assert config_variables_with_escaped_vars["escaped_password_dict"] == {
+        "drivername": r"po\$tgresql",
+        "host": "localhost",
+        "port": "5432",
+        "username": "postgres",
+        "password": r"\$pas\$wor\$d1\$",
+        "database": "postgres",
+    }
+
+    # Ensure that when reading the escaped config variable, the escaping should be removed
+    assert (
+        context.get_config_with_variables_substituted().datasources["mydatasource"][
+            "batch_kwargs_generators"
+        ]["mygenerator"]["reader_options"]["test_variable_sub_escaped"]
+        == "this_is_$mypassword_escape_the_$signs"
+    )
+    assert (
+        context.get_config_with_variables_substituted().datasources["mydatasource"][
+            "batch_kwargs_generators"
+        ]["mygenerator"]["reader_options"]["test_variable_sub_escaped_dict"]
+        == dict_to_escape
+    )
+
+    assert (
+        context.get_config_with_variables_substituted().datasources["mydatasource"][
+            "batch_kwargs_generators"
+        ]["mygenerator"]["reader_options"][
+            "test_escaped_manually_entered_value_from_config"
+        ]
+        == "correct_hor$e_battery_$taple"
+    )
+
     try:
         # verify that the value of the env var takes precedence over the one from the config variables file
         os.environ["replace_me_2"] = "value_from_env_var"
@@ -153,6 +228,7 @@ def test_substituted_config_variables_not_written_to_file(tmp_path_factory):
 
 def test_runtime_environment_are_used_preferentially(tmp_path_factory, monkeypatch):
     monkeypatch.setenv("FOO", "BAR")
+    monkeypatch.setenv("REPLACE_ME_ESCAPED_ENV", r"ive_been_\$replaced")
     value_from_environment = "from_environment"
     os.environ["replace_me"] = value_from_environment
 
@@ -194,7 +270,12 @@ def test_runtime_environment_are_used_preferentially(tmp_path_factory, monkeypat
 
 
 def test_substitute_config_variable():
-    config_variables_dict = {"arg0": "val_of_arg_0", "arg2": {"v1": 2}}
+    config_variables_dict = {
+        "arg0": "val_of_arg_0",
+        "arg2": {"v1": 2},
+        "aRg3": "val_of_aRg_3",
+        "ARG4": "val_of_ARG_4",
+    }
     assert (
         substitute_config_variable("abc${arg0}", config_variables_dict)
         == "abcval_of_arg_0"
@@ -223,33 +304,185 @@ def test_substitute_config_variable():
     )
     assert exc.value.missing_config_variable == "arg1"
 
+    # Null cases
+    assert substitute_config_variable("", config_variables_dict) == ""
+    assert substitute_config_variable(None, config_variables_dict) == None
+
+    # Test with mixed case
+    assert (
+        substitute_config_variable("prefix_${aRg3}_suffix", config_variables_dict)
+        == "prefix_val_of_aRg_3_suffix"
+    )
+    assert (
+        substitute_config_variable("${aRg3}", config_variables_dict) == "val_of_aRg_3"
+    )
+    # Test with upper case
+    assert (
+        substitute_config_variable("prefix_$ARG4/suffix", config_variables_dict)
+        == "prefix_val_of_ARG_4/suffix"
+    )
+    assert substitute_config_variable("$ARG4", config_variables_dict) == "val_of_ARG_4"
+
+    # Test with multiple substitutions
+    assert (
+        substitute_config_variable("prefix${arg0}$aRg3", config_variables_dict)
+        == "prefixval_of_arg_0val_of_aRg_3"
+    )
+
+    # Escaped `$` (don't substitute, but return un-escaped string)
+    assert (
+        substitute_config_variable(r"abc\${arg0}\$aRg3", config_variables_dict)
+        == "abc${arg0}$aRg3"
+    )
+
+    # Multiple configurations together
+    assert (
+        substitute_config_variable(
+            r"prefix$ARG4.$arg0/$aRg3:${ARG4}/\$dontsub${arg0}:${aRg3}.suffix",
+            config_variables_dict,
+        )
+        == "prefixval_of_ARG_4.val_of_arg_0/val_of_aRg_3:val_of_ARG_4/$dontsubval_of_arg_0:val_of_aRg_3.suffix"
+    )
+
 
 def test_substitute_env_var_in_config_variable_file(
     monkeypatch, empty_data_context_with_config_variables
 ):
     monkeypatch.setenv("FOO", "correct_val_of_replace_me")
+    monkeypatch.setenv("REPLACE_ME_ESCAPED_ENV", r"ive_been_\$replaced")
     context = empty_data_context_with_config_variables
     context_config = context.get_config_with_variables_substituted()
+    my_generator = context_config["datasources"]["mydatasource"][
+        "batch_kwargs_generators"
+    ]["mygenerator"]
+    reader_options = my_generator["reader_options"]
+
+    assert reader_options["test_variable_sub3"] == "correct_val_of_replace_me"
+    assert reader_options["test_variable_sub4"] == {
+        "inner_env_sub": "correct_val_of_replace_me"
+    }
+    assert reader_options["password"] == "dont$replaceme"
+
+    # Escaped variables (variables containing `$` that have been escaped)
     assert (
-        context_config["datasources"]["mydatasource"]["batch_kwargs_generators"][
-            "mygenerator"
-        ]["reader_options"]["test_variable_sub3"]
-        == "correct_val_of_replace_me"
+        reader_options["test_escaped_env_var_from_config"]
+        == "prefixive_been_$replaced/suffix"
     )
-    assert context_config["datasources"]["mydatasource"]["batch_kwargs_generators"][
-        "mygenerator"
-    ]["reader_options"]["test_variable_sub4"] == {
-        "inner_env_sub": "correct_val_of_replace_me"
+    assert (
+        my_generator["test_variable_escaped"]
+        == "dont$replace$me$please$$$$thanksive_been_$replaced"
+    )
+
+
+def test_escape_all_config_variables(empty_data_context_with_config_variables):
+    """
+    Make sure that all types of input to escape_all_config_variables are escaped properly: str, dict, OrderedDict, list
+    Make sure that changing the escape string works as expected.
+    """
+    context = empty_data_context_with_config_variables
+
+    # str
+    value_str = "pas$word1"
+    escaped_value_str = r"pas\$word1"
+    assert context.escape_all_config_variables(value=value_str) == escaped_value_str
+
+    value_str2 = "$pas$wor$d1$"
+    escaped_value_str2 = r"\$pas\$wor\$d1\$"
+    assert context.escape_all_config_variables(value=value_str2) == escaped_value_str2
+
+    # dict
+    value_dict = {
+        "drivername": "postgresql",
+        "host": "localhost",
+        "port": "5432",
+        "username": "postgres",
+        "password": "pass$word1",
+        "database": "postgres",
     }
+    escaped_value_dict = {
+        "drivername": "postgresql",
+        "host": "localhost",
+        "port": "5432",
+        "username": "postgres",
+        "password": r"pass\$word1",
+        "database": "postgres",
+    }
+    assert context.escape_all_config_variables(value=value_dict) == escaped_value_dict
+
+    # OrderedDict
+    value_ordered_dict = OrderedDict(
+        [
+            ("UNCOMMITTED", "uncommitted"),
+            ("docs_test_folder", "$test$folder"),
+            (
+                "test_db",
+                {
+                    "drivername": "postgresql",
+                    "host": "some_host",
+                    "port": "5432",
+                    "username": "postgres",
+                    "password": "pa$sword1",
+                    "database": "postgres",
+                },
+            ),
+        ]
+    )
+    escaped_value_ordered_dict = OrderedDict(
+        [
+            ("UNCOMMITTED", "uncommitted"),
+            ("docs_test_folder", r"\$test\$folder"),
+            (
+                "test_db",
+                {
+                    "drivername": "postgresql",
+                    "host": "some_host",
+                    "port": "5432",
+                    "username": "postgres",
+                    "password": r"pa\$sword1",
+                    "database": "postgres",
+                },
+            ),
+        ]
+    )
     assert (
-        context_config["datasources"]["mydatasource"]["batch_kwargs_generators"][
-            "mygenerator"
-        ]["reader_options"]["password"]
-        == "dont$replaceme"
+        context.escape_all_config_variables(value=value_ordered_dict)
+        == escaped_value_ordered_dict
     )
+
+    # list
+    value_list = [
+        "postgresql",
+        "localhost",
+        "5432",
+        "postgres",
+        "pass$word1",
+        "postgres",
+    ]
+    escaped_value_list = [
+        "postgresql",
+        "localhost",
+        "5432",
+        "postgres",
+        r"pass\$word1",
+        "postgres",
+    ]
+    assert context.escape_all_config_variables(value=value_list) == escaped_value_list
+
+    # Custom escape string
+    value_str_custom_escape_string = "pas$word1"
+    escaped_value_str_custom_escape_string = "pas@*&$word1"
     assert (
-        context_config["datasources"]["mydatasource"]["batch_kwargs_generators"][
-            "mygenerator"
-        ]["password"]
-        == "dont$replace$me$please$$$$thanks"
+        context.escape_all_config_variables(
+            value=value_str_custom_escape_string, dollar_sign_escape_string="@*&$"
+        )
+        == escaped_value_str_custom_escape_string
+    )
+
+    value_str_custom_escape_string2 = "$pas$wor$d1$"
+    escaped_value_str_custom_escape_string2 = "@*&$pas@*&$wor@*&$d1@*&$"
+    assert (
+        context.escape_all_config_variables(
+            value=value_str_custom_escape_string2, dollar_sign_escape_string="@*&$"
+        )
+        == escaped_value_str_custom_escape_string2
     )
diff --git a/tests/test_fixtures/config_variables.yml b/tests/test_fixtures/config_variables.yml
--- a/tests/test_fixtures/config_variables.yml
+++ b/tests/test_fixtures/config_variables.yml
@@ -5,3 +5,5 @@ replace_me_2:
   inner_env_sub: ${FOO}
 arg1:
   v1: 2
+REPLACE_ME_FROM_ESCAPED_ENV: ${REPLACE_ME_ESCAPED_ENV}
+escaped_password_already_in_config_yml: correct_hor\$e_battery_\$taple
diff --git a/tests/test_fixtures/great_expectations_basic_with_variables.yml b/tests/test_fixtures/great_expectations_basic_with_variables.yml
--- a/tests/test_fixtures/great_expectations_basic_with_variables.yml
+++ b/tests/test_fixtures/great_expectations_basic_with_variables.yml
@@ -13,16 +13,17 @@ datasources:
       mygenerator:
         class_name: SubdirReaderBatchKwargsGenerator
         base_directory: ../data
-        password: dont$replace$me$please$$$$thanks
+        test_variable_escaped: dont\$replace\$me\$please\$\$\$\$thanks${REPLACE_ME_FROM_ESCAPED_ENV}
         reader_options:
           sep:
           engine: python
-          password: dont$replaceme
+          password: dont\$replaceme
           test_variable_sub1: ${replace_me}
           test_variable_sub2: $replace_me
           test_variable_sub3: ${replace_me_1}
           test_variable_sub4: ${replace_me_2}
-
+          test_escaped_env_var_from_config: prefix$REPLACE_ME_FROM_ESCAPED_ENV/suffix
+          test_escaped_manually_entered_value_from_config: $escaped_password_already_in_config_yml
 
 config_variables_file_path: uncommitted/config_variables.yml
 expectations_store_name: expectations_store