internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: great-expectations/great_expectations

Issue:
Indenting Expectations to enable Pull Request Diff
Hi Team, 

I intend to use GE within a team environment. We will probably be storing them along side our project.
The expectations are stored in a single-lined JSON blob in your expectation store, but with no indentation and thus will make diffing the parameters challenging.

Any ideas on the best way to do pull requests with expectations?

Cheers


Discussion:
@troxil First and foremost you are spot on storing `Expectations` in version control right in your project. 🎉 

We have discussed this as a feature we all want for the exact same reasons. We had held off initially because getting ordered dicts in legacy python was squirrelly at best. But now that we have dropped legacy python support there are no blockers to this important feature.

We'd love a PR if you are so inclined. If this is something you want to contribute to it would be a great first issue!

If not, a workaround we've used in the past is to open your json files with an ide like Atom and use a deterministic JSON prettyfier before you commit them.

Resolving patch:
diff --git a/great_expectations/data_asset/data_asset.py b/great_expectations/data_asset/data_asset.py
--- a/great_expectations/data_asset/data_asset.py
+++ b/great_expectations/data_asset/data_asset.py
@@ -836,7 +836,12 @@ def save_expectation_suite(
             self._data_context.save_expectation_suite(expectation_suite)
         elif filepath is not None:
             with open(filepath, 'w') as outfile:
-                json.dump(expectationSuiteSchema.dump(expectation_suite), outfile, indent=2)
+                json.dump(
+                    expectationSuiteSchema.dump(expectation_suite),
+                    outfile,
+                    indent=2,
+                    sort_keys=True,
+                )
         else:
             raise ValueError("Unable to save config: filepath or data_context must be available.")
 
diff --git a/great_expectations/data_context/store/expectations_store.py b/great_expectations/data_context/store/expectations_store.py
--- a/great_expectations/data_context/store/expectations_store.py
+++ b/great_expectations/data_context/store/expectations_store.py
@@ -33,7 +33,7 @@ def __init__(self, store_backend=None, runtime_environment=None):
         super().__init__(store_backend=store_backend, runtime_environment=runtime_environment)
 
     def serialize(self, key, value):
-        return self._expectationSuiteSchema.dumps(value)
+        return self._expectationSuiteSchema.dumps(value, indent=2, sort_keys=True)
 
     def deserialize(self, key, value):
         return self._expectationSuiteSchema.loads(value)


Test patch:
Repository: great-expectations/great_expectations

Issue:
profile CLI help text mistakenly includes Python parameter docstrings
For CLI commands (like all the other `great_expectations` ones) the docstrings for the Click command shouldn't include traditional docstring elements like argument specifications.

It looks like `great_expectations datasource profile --help` includes this: (see all the `:param:` entries in the second paragraph)

<img width="628" alt="image" src="https://user-images.githubusercontent.com/30357972/78701953-c2e4f580-78ff-11ea-8b2a-3cb8004f8d83.png">



Discussion:

Resolving patch:
diff --git a/great_expectations/cli/datasource.py b/great_expectations/cli/datasource.py
--- a/great_expectations/cli/datasource.py
+++ b/great_expectations/cli/datasource.py
@@ -155,14 +155,6 @@ def datasource_profile(datasource, generator_name, data_assets, profile_all_data
     if the number of data assets in the datasource exceeds the internally defined limit. If it does, it will
     prompt the user to either specify the list of data assets to profile or to profile all.
     If the limit is not exceeded, the profiler will profile all data assets in the datasource.
-
-    :param datasource: name of the datasource to profile
-    :param data_assets: if this comma-separated list of data asset names is provided, only the specified data assets will be profiled
-    :param profile_all_data_assets: if provided, all data assets will be profiled
-    :param directory:
-    :param view: Open the docs in a browser
-    :param additional_batch_kwargs: Additional keyword arguments to be provided to get_batch when loading the data asset.
-    :return:
     """
     cli_message("<yellow>Warning - this is a BETA feature.</yellow>")
     try:


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/core/usage_statistics/schemas.py b/great_expectations/core/usage_statistics/schemas.py
--- a/great_expectations/core/usage_statistics/schemas.py
+++ b/great_expectations/core/usage_statistics/schemas.py
@@ -515,7 +515,7 @@
                         "cli.datasource.new",
                         "data_context.open_data_docs",
                         "data_context.build_data_docs",
-                        "cli.init.ctx_new"
+                        "cli.init.create"
                     ],
                 },
                 "event_payload": {


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/data_asset/data_asset.py b/great_expectations/data_asset/data_asset.py
--- a/great_expectations/data_asset/data_asset.py
+++ b/great_expectations/data_asset/data_asset.py
@@ -602,6 +602,7 @@ def get_expectation_suite(
         discard_include_config_kwargs=True,
         discard_catch_exceptions_kwargs=True,
         suppress_warnings=False,
+        suppress_logging=False,
     ):
         """Returns _expectation_config as a JSON object, and perform some cleaning along the way.
 
@@ -614,6 +615,10 @@ def get_expectation_suite(
                 In returned expectation objects, suppress the `include_config` parameter. Defaults to `True`.
             discard_catch_exceptions_kwargs (boolean): \
                 In returned expectation objects, suppress the `catch_exceptions` parameter.  Defaults to `True`.
+            suppress_warnings (boolean): \
+                If true, do not include warnings in logging information about the operation.
+            suppress_logging (boolean): \
+                If true, do not create a log entry (useful when using get_expectation_suite programmatically)
 
         Returns:
             An expectation suite.
@@ -691,7 +696,8 @@ def get_expectation_suite(
             settings_message += " settings filtered."
 
         expectation_suite.expectations = expectations
-        logger.info(message + settings_message)
+        if not suppress_logging:
+            logger.info(message + settings_message)
         return expectation_suite
 
     def save_expectation_suite(
diff --git a/great_expectations/profile/basic_suite_builder_profiler.py b/great_expectations/profile/basic_suite_builder_profiler.py
--- a/great_expectations/profile/basic_suite_builder_profiler.py
+++ b/great_expectations/profile/basic_suite_builder_profiler.py
@@ -480,7 +480,7 @@ def _profile(cls, dataset, configuration=None):
             dataset = _remove_column_expectations(dataset, excluded_expectations)
         if included_expectations:
             for expectation in dataset.get_expectation_suite(
-                discard_failed_expectations=False
+                discard_failed_expectations=False, suppress_logging=True,
             ).expectations:
                 if expectation.expectation_type not in included_expectations:
                     try:


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -41,11 +41,15 @@
     "sphinxcontrib.contentui",
     "sphinx_gitstamp",
     "sphinx.ext.autosectionlabel",
+    'sphinxcontrib.discourse',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 
+#discourse url connect
+discourse_url = 'https://discuss.greatexpectations.io/'
+
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/cli/cli_messages.py b/great_expectations/cli/cli_messages.py
--- a/great_expectations/cli/cli_messages.py
+++ b/great_expectations/cli/cli_messages.py
@@ -17,6 +17,7 @@
     great_expectations
     |-- great_expectations.yml
     |-- expectations
+    |-- checkpoints 
     |-- notebooks
     |-- plugins
     |-- .gitignore


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/render/renderer/suite_scaffold_notebook_renderer.py b/great_expectations/render/renderer/suite_scaffold_notebook_renderer.py
--- a/great_expectations/render/renderer/suite_scaffold_notebook_renderer.py
+++ b/great_expectations/render/renderer/suite_scaffold_notebook_renderer.py
@@ -130,7 +130,7 @@ def render(self, batch_kwargs=None, **kwargs) -> nbformat.NotebookNode:
 
 This is highly configurable depending on your goals. You can include or exclude
 columns, and include or exclude expectation types (when applicable). [The
-Expectation Glossary](http://docs.greatexpectations.io/en/latest/expectation_glossary.html)
+Expectation Glossary](https://docs.greatexpectations.io/en/latest/reference/glossary_of_expectations.html?utm_source=notebook&utm_medium=scaffold_expectations)
 contains a list of possible expectations."""
         )
         self._add_scaffold_cell()


Test patch:
diff --git a/tests/render/renderer/test_suite_scaffold_notebook_renderer.py b/tests/render/renderer/test_suite_scaffold_notebook_renderer.py
--- a/tests/render/renderer/test_suite_scaffold_notebook_renderer.py
+++ b/tests/render/renderer/test_suite_scaffold_notebook_renderer.py
@@ -75,7 +75,7 @@ def test_render_snapshot_test(titanic_data_context):
 
 This is highly configurable depending on your goals. You can include or exclude
 columns, and include or exclude expectation types (when applicable). [The
-Expectation Glossary](http://docs.greatexpectations.io/en/latest/expectation_glossary.html)
+Expectation Glossary](https://docs.greatexpectations.io/en/latest/reference/glossary_of_expectations.html?utm_source=notebook&utm_medium=scaffold_expectations)
 contains a list of possible expectations.""",
                 "metadata": {},
             },
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/render/renderer/site_builder.py b/great_expectations/render/renderer/site_builder.py
--- a/great_expectations/render/renderer/site_builder.py
+++ b/great_expectations/render/renderer/site_builder.py
@@ -641,7 +641,7 @@ def _get_call_to_action_buttons(self, usage_statistics):
         )
         see_glossary = CallToActionButton(
             "See More Kinds of Expectations",
-            "http://docs.greatexpectations.io/en/latest/reference/expectation_glossary.html",
+            "https://docs.greatexpectations.io/en/latest/reference/glossary_of_expectations.html",
         )
         validation_playground = CallToActionButton(
             "How to Validate Data",


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/data_context/util.py b/great_expectations/data_context/util.py
--- a/great_expectations/data_context/util.py
+++ b/great_expectations/data_context/util.py
@@ -146,7 +146,7 @@ def substitute_config_variable(template_str, config_variables_dict):
     if match:
         config_variable_value = config_variables_dict.get(match.group(1))
 
-        if config_variable_value:
+        if config_variable_value is not None:
             if match.start() == 0 and match.end() == len(template_str):
                 return config_variable_value
             else:


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -32,6 +32,7 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    'autoapi.extension',
     # 'sphinx_rtd_theme',
     "sphinx.ext.autodoc",
     "sphinx.ext.todo",
@@ -44,6 +45,10 @@
     "sphinxcontrib.discourse",
 ]
 
+autoapi_type = 'python'
+autoapi_dirs = ['../great_expectations']
+
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/render/view/view.py b/great_expectations/render/view/view.py
--- a/great_expectations/render/view/view.py
+++ b/great_expectations/render/view/view.py
@@ -47,9 +47,9 @@ class DefaultJinjaView(object):
     * Font Awesome 5.10.1
     * Bootstrap 4.3.1
     * jQuery 3.2.1
-    * Vega 5.3.5
-    * Vega-Lite 3.2.1
-    * Vega-Embed 4.0.0
+    * Vega 5
+    * Vega-Lite 4
+    * Vega-Embed 6
 
     """
 


Test patch:
diff --git a/tests/cli/test_init_pandas.py b/tests/cli/test_init_pandas.py
--- a/tests/cli/test_init_pandas.py
+++ b/tests/cli/test_init_pandas.py
@@ -138,8 +138,6 @@ def test_cli_init_on_new_project(
                         short-logo-vector.svg
                         short-logo.png
                         validation_failed_unexpected_values.gif
-                    scripts/
-                        bootstrap-table-filter-control.min.js
                     styles/
                         data_docs_custom_styles_template.css
                         data_docs_default_styles.css
@@ -616,8 +614,6 @@ def test_cli_init_on_new_project_with_broken_excel_file_try_again_with_different
                         short-logo-vector.svg
                         short-logo.png
                         validation_failed_unexpected_values.gif
-                    scripts/
-                        bootstrap-table-filter-control.min.js
                     styles/
                         data_docs_custom_styles_template.css
                         data_docs_default_styles.css
diff --git a/tests/cli/test_init_sqlite.py b/tests/cli/test_init_sqlite.py
--- a/tests/cli/test_init_sqlite.py
+++ b/tests/cli/test_init_sqlite.py
@@ -154,8 +154,6 @@ def test_cli_init_on_new_project(
                         short-logo-vector.svg
                         short-logo.png
                         validation_failed_unexpected_values.gif
-                    scripts/
-                        bootstrap-table-filter-control.min.js
                     styles/
                         data_docs_custom_styles_template.css
                         data_docs_default_styles.css
diff --git a/tests/data_context/test_data_context.py b/tests/data_context/test_data_context.py
--- a/tests/data_context/test_data_context.py
+++ b/tests/data_context/test_data_context.py
@@ -530,8 +530,6 @@ def test_render_full_static_site_from_empty_project(tmp_path_factory, filesystem
                 short-logo-vector.svg
                 short-logo.png
                 validation_failed_unexpected_values.gif
-            scripts/
-                bootstrap-table-filter-control.min.js
             styles/
                 data_docs_custom_styles_template.css
                 data_docs_default_styles.css
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/datasource/sqlalchemy_datasource.py b/great_expectations/datasource/sqlalchemy_datasource.py
--- a/great_expectations/datasource/sqlalchemy_datasource.py
+++ b/great_expectations/datasource/sqlalchemy_datasource.py
@@ -23,21 +23,22 @@
     logger.debug("Unable to import sqlalchemy.")
 
 
-try:
-    import google.auth
+if sqlalchemy != None:
+    try:
+        import google.auth
 
-    datasource_initialization_exceptions = (
-        sqlalchemy.exc.OperationalError,
-        sqlalchemy.exc.DatabaseError,
-        sqlalchemy.exc.ArgumentError,
-        google.auth.exceptions.GoogleAuthError,
-    )
-except ImportError:
-    datasource_initialization_exceptions = (
-        sqlalchemy.exc.OperationalError,
-        sqlalchemy.exc.DatabaseError,
-        sqlalchemy.exc.ArgumentError,
-    )
+        datasource_initialization_exceptions = (
+            sqlalchemy.exc.OperationalError,
+            sqlalchemy.exc.DatabaseError,
+            sqlalchemy.exc.ArgumentError,
+            google.auth.exceptions.GoogleAuthError,
+        )
+    except ImportError:
+        datasource_initialization_exceptions = (
+            sqlalchemy.exc.OperationalError,
+            sqlalchemy.exc.DatabaseError,
+            sqlalchemy.exc.ArgumentError,
+        )
 
 
 class SqlAlchemyDatasource(Datasource):


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/dataset/sparkdf_dataset.py b/great_expectations/dataset/sparkdf_dataset.py
--- a/great_expectations/dataset/sparkdf_dataset.py
+++ b/great_expectations/dataset/sparkdf_dataset.py
@@ -1132,6 +1132,29 @@ def expect_column_values_to_not_match_regex(
     ):
         return column.withColumn("__success", ~column[0].rlike(regex))
 
+    @DocInherit
+    @MetaSparkDFDataset.column_map_expectation
+    def expect_column_values_to_match_regex_list(
+        self,
+        column,
+        regex_list,
+        match_on="any",
+        mostly=None,
+        result_format=None,
+        include_config=True,
+        catch_exceptions=None,
+        meta=None,
+    ):
+        if match_on == "any":
+            return column.withColumn("__success", column[0].rlike("|".join(regex_list)))
+        elif match_on == "all":
+            formatted_regex_list = ["(?={})".format(regex) for regex in regex_list]
+            return column.withColumn(
+                "__success", column[0].rlike("".join(formatted_regex_list))
+            )
+        else:
+            raise ValueError("match_on must be either 'any' or 'all'")
+
     @DocInherit
     @MetaSparkDFDataset.column_pair_map_expectation
     def expect_column_pair_values_to_be_equal(


Test patch:
diff --git a/tests/test_definitions/column_map_expectations/expect_column_values_to_match_regex_list.json b/tests/test_definitions/column_map_expectations/expect_column_values_to_match_regex_list.json
--- a/tests/test_definitions/column_map_expectations/expect_column_values_to_match_regex_list.json
+++ b/tests/test_definitions/column_map_expectations/expect_column_values_to_match_regex_list.json
@@ -6,6 +6,13 @@
       "x" : ["man", "plan", "canal", "panama", "hat", "bat", "bit", "bot", "but", "bet"],
       "column_name with space": ["man", "plan", "canal", "panama", "hat", "bat", "bit", "bot", "but", "bet"]
     },
+    "schemas": {
+      "spark": {
+        "w": "StringType",
+        "x": "StringType",
+        "column_name with space": "StringType"
+      }
+    },
     "tests" : [{
       "title" : "Basic positive test",
       "exact_match_out" : false,
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -455,7 +455,7 @@ def candidate_test_is_on_temporary_notimplemented_list(context, expectation_type
             # "expect_column_value_lengths_to_equal",
             # "expect_column_values_to_match_regex",
             # "expect_column_values_to_not_match_regex",
-            "expect_column_values_to_match_regex_list",
+            # "expect_column_values_to_match_regex_list",
             "expect_column_values_to_not_match_regex_list",
             # "expect_column_values_to_match_strftime_format",
             "expect_column_values_to_be_dateutil_parseable",
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/datasource/batch_kwargs_generator/s3_batch_kwargs_generator.py b/great_expectations/datasource/batch_kwargs_generator/s3_batch_kwargs_generator.py
--- a/great_expectations/datasource/batch_kwargs_generator/s3_batch_kwargs_generator.py
+++ b/great_expectations/datasource/batch_kwargs_generator/s3_batch_kwargs_generator.py
@@ -43,7 +43,7 @@ class S3GlobReaderBatchKwargsGenerator(BatchKwargsGenerator):
                   my_first_asset:
                     prefix: my_first_asset/
                     regex_filter: .*  # The regex filter will filter the results returned by S3 for the key and prefix to only those matching the regex
-                    directory_assets: True
+                    directory_assets: True # if True, the contents of the directory will be treated as one batch. Notice that this option does not work with Pandas, since Pandas does not support loading multiple files from and S3 bucket into a data frame.
                   access_logs:
                     prefix: access_logs
                     regex_filter: access_logs/2019.*\.csv.gz


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/data_context/types/base.py b/great_expectations/data_context/types/base.py
--- a/great_expectations/data_context/types/base.py
+++ b/great_expectations/data_context/types/base.py
@@ -227,6 +227,7 @@ class Meta:
         keys=fields.Str(), values=fields.Dict(), allow_none=True
     )
     credentials = fields.Raw(allow_none=True)
+    spark_context = fields.Raw(allow_none=True)
 
     @validates_schema
     def validate_schema(self, data, **kwargs):


Test patch: