Repository: great-expectations/great_expectations

Issue:
Custom inherited expectation class produce an InvalidExpectationConfigurationError
**Describe the bug**
Trying to create a custom expectation using custom parameters based on [this doc](https://docs.greatexpectations.io/docs/guides/expectations/creating_custom_expectations/how_to_create_custom_parameterized_expectations).
In this one I replace the value_set by my own, but when running the suite, I get an error great_expectations.exceptions.exceptions.InvalidExpectationConfigurationError: value_set is required from the inherited class ExpectColumnValuesToBeInSet

**To Reproduce**
My class code :
```
class ExpectColumnValuesToBeTwoLetterCountryCode(ExpectColumnValuesToBeInSet):
    default_kwarg_values = {
        "value_set": ["FR", "DE", "CH", "ES", "IT", "BE", "NL", "PL"],
        "parse_strings_as_datetimes": False
    }

    def validate_configuration(self, configuration: Optional[ExpectationConfiguration]):
        super().validate_configuration(configuration)
        assert "value_set" not in configuration.kwargs, "value_set cannot be altered"
        assert "parse_strings_as_datetimes" not in configuration.kwargs, "parse_strings_as_datetimes cannot be altered"
```

**Expected behavior**
Based on the documentation I expect this class to work.

**Environment (please complete the following information):**
 - Operating System: windows
 - Great Expectations Version: 0.13.42

**Additional context**

```datasource_config = {
    "name": "DS_pandas_article",
    "class_name": "Datasource",
    "module_name": "great_expectations.datasource",
    "execution_engine": {
        "module_name": "great_expectations.execution_engine",
        "class_name": "PandasExecutionEngine",
    },
    "data_connectors": {
        "default_runtime_data_connector_name": {
            "class_name": "RuntimeDataConnector",
            "module_name": "great_expectations.datasource.data_connector",
            "batch_identifiers": ["default_identifier_name"],
        },
    },
}
```


```from custom_expectations import ExpectColumnValuesToBeTwoLetterCountryCode

batch_request = RuntimeBatchRequest(
    datasource_name="DS_pandas_article",
    data_connector_name="default_runtime_data_connector_name",
    data_asset_name="DA_article",
    runtime_parameters={"batch_data": dataframe},
    batch_identifiers={"default_identifier_name": "default_identifier"},


validator = context.get_validator(
    batch_request=batch_request, 
    expectation_suite_name="article.custom_suite"
)
my_validation = validator.validate()
```
and my suite:
```
{
  "data_asset_type": null,
  "expectation_suite_name": "article.custom_suite",
  "expectations": [
    {
      "expectation_type": "expect_column_values_to_be_two_letter_country_code",
      "ge_cloud_id": null,
      "kwargs": {
        "column": "Country",
		"result_format": "COMPLETE"
      }
    }
  ],
  "ge_cloud_id": null,
  "meta": {
    "great_expectations_version": "0.13.42"
  }
}
```


Discussion:
Thanks for opening this issue, @pmariac! We will review and be in touch.
When creating a **custom parameterized Expectation** from `ExpectColumnValuesToBeInSet`, the problem is (I think) that:

On line 112 of `expect_column_values_to_be_in_set.py` the `value_set` is set to `None`:
`default_kwarg_values = {"value_set": None, "parse_strings_as_datetimes": False}`

I want to set default values (a simple country list), so, just like in the documentation I use this:
```
    default_kwarg_values = {
        "value_set": ["FR", "DE", "CH", "ES", "IT", "BE", "NL", "PL"],
        "parse_strings_as_datetimes": False
    }
```

When GE parses this, it uses the `nested_update` function in util.py, which crashes when trying to append my default values to the base ones: `TypeError: unsupported operand type(s) for +: 'NoneType' and 'list'`

My hint: `default_kwarg_values.value_set` should be an empty array rather than a None

Full stacktrace:

```
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_18691/445229333.py in <module>
     11 import json
     12 import os
---> 13 from plugins import expect_column_values_to_be_two_letters_country_code
     14 
     15 

~/dev/demo/great_expectations/uncommitted/plugins/expect_column_values_to_be_two_letters_country_code.py in <module>
      3 from typing import Optional
      4 
----> 5 class ExpectColumnValuesToBeTwoLetterCountryCode(ExpectColumnValuesToBeInSet):
      6     default_kwarg_values = {
      7         "value_set": ["FR", "DE", "CH", "ES", "IT", "BE", "NL", "PL"],

~/.local/lib/python3.8/site-packages/great_expectations/expectations/expectation.py in __new__(cls, clsname, bases, attrs)
     88             default_kwarg_values = nested_update(default_kwarg_values, default_kwargs)
     89 
---> 90         newclass.default_kwarg_values = nested_update(
     91             default_kwarg_values, attrs.get("default_kwarg_values", {})
     92         )

~/.local/lib/python3.8/site-packages/great_expectations/core/util.py in nested_update(d, u, dedup, concat_lists)
    100                     d[k] = list(set(l1 + l2))
    101                 else:
--> 102                     d[k] = l1 + l2
    103             else:
    104                 d[k] = l2

TypeError: unsupported operand type(s) for +: 'NoneType' and 'list'
```

Resolving patch:
diff --git a/great_expectations/expectations/core/expect_column_values_to_be_in_set.py b/great_expectations/expectations/core/expect_column_values_to_be_in_set.py
--- a/great_expectations/expectations/core/expect_column_values_to_be_in_set.py
+++ b/great_expectations/expectations/core/expect_column_values_to_be_in_set.py
@@ -161,7 +161,7 @@ class ExpectColumnValuesToBeInSet(ColumnMapExpectation):
     )
 
     default_kwarg_values = {
-        "value_set": None,
+        "value_set": [],
         "parse_strings_as_datetimes": False,
         "auto": False,
         "profiler_config": default_profiler_config,
@@ -411,15 +411,20 @@ def validate_configuration(
         self, configuration: Optional[ExpectationConfiguration]
     ) -> bool:
         super().validate_configuration(configuration)
+        # supports extensibility by allowing value_set to not be provided in config but captured via child-class default_kwarg_values, e.g. parameterized expectations
+        value_set = configuration.kwargs.get(
+            "value_set"
+        ) or self.default_kwarg_values.get("value_set")
         try:
-            assert "value_set" in configuration.kwargs, "value_set is required"
             assert (
-                isinstance(configuration.kwargs["value_set"], (list, set, dict))
-                or configuration.kwargs["value_set"] is None
-            ), "value_set must be a list, set, or None"
-            if isinstance(configuration.kwargs["value_set"], dict):
+                "value_set" in configuration.kwargs or value_set
+            ), "value_set is required"
+            assert isinstance(
+                value_set, (list, set, dict)
+            ), "value_set must be a list, set, or dict"
+            if isinstance(value_set, dict):
                 assert (
-                    "$PARAMETER" in configuration.kwargs["value_set"]
+                    "$PARAMETER" in value_set
                 ), 'Evaluation Parameter dict for value_set kwarg must have "$PARAMETER" key.'
         except AssertionError as e:
             raise InvalidExpectationConfigurationError(str(e))


Test patch:
diff --git a/tests/expectations/core/test_expect_column_mean_to_be_positive.py b/tests/expectations/core/test_expect_column_mean_to_be_positive.py
new file mode 100644
--- /dev/null
+++ b/tests/expectations/core/test_expect_column_mean_to_be_positive.py
@@ -0,0 +1,51 @@
+import pandas as pd
+import pytest
+
+import great_expectations.exceptions.exceptions
+from great_expectations.core.batch import RuntimeBatchRequest
+from great_expectations.data_context import DataContext
+from great_expectations.expectations.core.expect_column_mean_to_be_between import (
+    ExpectColumnMeanToBeBetween,
+)
+
+
+# <snippet>
+class ExpectColumnMeanToBePositive(ExpectColumnMeanToBeBetween):
+    """Expects the mean of values in this column to be positive"""
+
+    default_kwarg_values = {
+        "min_value": 0,
+        "strict_min": True,
+    }
+
+    def validate_configuration(self, configuration):
+        super().validate_configuration(configuration)
+        assert "min_value" not in configuration.kwargs, "min_value cannot be altered"
+        assert "max_value" not in configuration.kwargs, "max_value cannot be altered"
+        assert "strict_min" not in configuration.kwargs, "strict_min cannot be altered"
+        assert "strict_max" not in configuration.kwargs, "strict_max cannot be altered"
+
+    library_metadata = {"tags": ["basic stats"], "contributors": ["@joegargery"]}
+
+
+# </snippet>
+def test_expect_column_mean_to_be_positive(data_context_with_datasource_pandas_engine):
+    context: DataContext = data_context_with_datasource_pandas_engine
+
+    df = pd.DataFrame({"a": [0, 1, 3, 4, 5]})
+
+    batch_request = RuntimeBatchRequest(
+        datasource_name="my_datasource",
+        data_connector_name="default_runtime_data_connector_name",
+        data_asset_name="my_data_asset",
+        runtime_parameters={"batch_data": df},
+        batch_identifiers={"default_identifier_name": "my_identifier"},
+    )
+    validator = context.get_validator(
+        batch_request=batch_request,
+        create_expectation_suite_with_name="test",
+    )
+
+    result = validator.expect_column_mean_to_be_positive(column="a")
+
+    assert result.success is True
diff --git a/tests/expectations/core/test_expect_column_values_to_be_in_set.py b/tests/expectations/core/test_expect_column_values_to_be_in_set.py
new file mode 100644
--- /dev/null
+++ b/tests/expectations/core/test_expect_column_values_to_be_in_set.py
@@ -0,0 +1,198 @@
+from datetime import datetime
+
+import pandas as pd
+import pytest
+
+import great_expectations.exceptions.exceptions
+from great_expectations.core.batch import RuntimeBatchRequest
+from great_expectations.data_context import DataContext
+from great_expectations.expectations.core.expect_column_values_to_be_in_set import (
+    ExpectColumnValuesToBeInSet,
+)
+
+
+# <snippet>
+class ExpectColumnValuesToBeTwoLetterCountryCode(ExpectColumnValuesToBeInSet):
+    default_kwarg_values = {
+        "value_set": ["FR", "DE", "CH", "ES", "IT", "BE", "NL", "PL"],
+    }
+
+
+# </snippet>
+def test_expect_column_values_to_be_in_set_fail(
+    data_context_with_datasource_pandas_engine,
+):
+    context: DataContext = data_context_with_datasource_pandas_engine
+
+    df = pd.DataFrame(
+        {
+            "a": [
+                "2021-01-01",
+                "2021-01-31",
+                "2021-02-28",
+                "2021-03-20",
+                "2021-02-21",
+                "2021-05-01",
+                "2021-06-18",
+            ]
+        }
+    )
+
+    batch_request = RuntimeBatchRequest(
+        datasource_name="my_datasource",
+        data_connector_name="default_runtime_data_connector_name",
+        data_asset_name="my_data_asset",
+        runtime_parameters={"batch_data": df},
+        batch_identifiers={"default_identifier_name": "my_identifier"},
+    )
+    validator = context.get_validator(
+        batch_request=batch_request,
+        create_expectation_suite_with_name="test",
+    )
+
+    result = validator.expect_column_values_to_be_in_set(
+        column="a", value_set=["2021-06-18"]
+    )
+    assert result.success is False
+
+
+def test_expect_column_values_in_set_pass(
+    data_context_with_datasource_pandas_engine,
+):
+    context: DataContext = data_context_with_datasource_pandas_engine
+
+    df = pd.DataFrame(
+        {
+            "a": [
+                "2021-01-01",
+                "2021-01-31",
+                "2021-02-28",
+                "2021-03-20",
+                "2021-02-21",
+                "2021-05-01",
+                "2021-06-18",
+            ]
+        }
+    )
+
+    batch_request = RuntimeBatchRequest(
+        datasource_name="my_datasource",
+        data_connector_name="default_runtime_data_connector_name",
+        data_asset_name="my_data_asset",
+        runtime_parameters={"batch_data": df},
+        batch_identifiers={"default_identifier_name": "my_identifier"},
+    )
+    validator = context.get_validator(
+        batch_request=batch_request,
+        create_expectation_suite_with_name="test",
+    )
+
+    result = validator.expect_column_values_to_be_in_set(
+        column="a",
+        value_set=[
+            "2021-01-01",
+            "2021-01-31",
+            "2021-02-28",
+            "2021-03-20",
+            "2021-02-21",
+            "2021-05-01",
+            "2021-06-18",
+        ],
+    )
+    assert result.success is True
+
+
+def test_expect_column_values_country_fail(
+    data_context_with_datasource_pandas_engine,
+):
+    context: DataContext = data_context_with_datasource_pandas_engine
+
+    df = pd.DataFrame(
+        {
+            "a": [
+                "2021-01-01",
+                "2021-01-31",
+                "2021-02-28",
+                "2021-03-20",
+                "2021-02-21",
+                "2021-05-01",
+                "2021-06-18",
+            ]
+        }
+    )
+
+    batch_request = RuntimeBatchRequest(
+        datasource_name="my_datasource",
+        data_connector_name="default_runtime_data_connector_name",
+        data_asset_name="my_data_asset",
+        runtime_parameters={"batch_data": df},
+        batch_identifiers={"default_identifier_name": "my_identifier"},
+    )
+    validator = context.get_validator(
+        batch_request=batch_request,
+        create_expectation_suite_with_name="test",
+    )
+
+    result = validator.expect_column_values_to_be_two_letter_country_code(column="a")
+
+    assert result.success is False
+
+
+def test_expect_column_values_country_pass(
+    data_context_with_datasource_pandas_engine,
+):
+    context: DataContext = data_context_with_datasource_pandas_engine
+
+    df = pd.DataFrame({"a": ["FR", "DE", "CH", "ES", "IT", "BE", "NL", "PL"]})
+
+    batch_request = RuntimeBatchRequest(
+        datasource_name="my_datasource",
+        data_connector_name="default_runtime_data_connector_name",
+        data_asset_name="my_data_asset",
+        runtime_parameters={"batch_data": df},
+        batch_identifiers={"default_identifier_name": "my_identifier"},
+    )
+    validator = context.get_validator(
+        batch_request=batch_request,
+        create_expectation_suite_with_name="test",
+    )
+
+    result = validator.expect_column_values_to_be_two_letter_country_code(column="a")
+
+    assert result.success is True
+
+
+def test_expect_column_values_to_be_in_set_no_set(
+    data_context_with_datasource_pandas_engine,
+):
+    context: DataContext = data_context_with_datasource_pandas_engine
+
+    df = pd.DataFrame(
+        {
+            "a": [
+                "2021-01-01",
+                "2021-01-31",
+                "2021-02-28",
+                "2021-03-20",
+                "2021-02-21",
+                "2021-05-01",
+                "2021-06-18",
+            ]
+        }
+    )
+
+    batch_request = RuntimeBatchRequest(
+        datasource_name="my_datasource",
+        data_connector_name="default_runtime_data_connector_name",
+        data_asset_name="my_data_asset",
+        runtime_parameters={"batch_data": df},
+        batch_identifiers={"default_identifier_name": "my_identifier"},
+    )
+    validator = context.get_validator(
+        batch_request=batch_request,
+        create_expectation_suite_with_name="test",
+    )
+    with pytest.raises(
+        great_expectations.exceptions.exceptions.InvalidExpectationConfigurationError
+    ):
+        result = validator.expect_column_values_to_be_in_set(column="a")