internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/data_asset/data_asset.py b/great_expectations/data_asset/data_asset.py
--- a/great_expectations/data_asset/data_asset.py
+++ b/great_expectations/data_asset/data_asset.py
@@ -1239,9 +1239,12 @@ def _format_map_output(
                     )
                 ]
             except TypeError:
-                partial_unexpected_counts = [
-                    "partial_exception_counts requires a hashable type"
-                ]
+                partial_unexpected_counts = []
+                if "details" not in return_obj["result"]:
+                    return_obj["result"]["details"] = {}
+                return_obj["result"]["details"][
+                    "partial_unexpected_counts_error"
+                ] = "partial_unexpected_counts requested, but requires a hashable type"
             finally:
                 return_obj["result"].update(
                     {
diff --git a/great_expectations/render/renderer/content_block/validation_results_table_content_block.py b/great_expectations/render/renderer/content_block/validation_results_table_content_block.py
--- a/great_expectations/render/renderer/content_block/validation_results_table_content_block.py
+++ b/great_expectations/render/renderer/content_block/validation_results_table_content_block.py
@@ -124,7 +124,13 @@ def _get_unexpected_table(cls, evr):
         if result.get("partial_unexpected_counts"):
             header_row = ["Unexpected Value", "Count"]
             for unexpected_count in result.get("partial_unexpected_counts"):
-                if unexpected_count.get("value"):
+                if not isinstance(unexpected_count, dict):
+                    # handles case: "partial_exception_counts requires a hashable type"
+                    # this case is also now deprecated (because the error is moved to an errors key
+                    # the error also *should have* been updated to "partial_unexpected_counts ..." long ago.
+                    # NOTE: JPC 20200724 - Consequently, this codepath should be removed by approximately Q1 2021
+                    continue
+                elif unexpected_count.get("value"):
                     table_rows.append(
                         [unexpected_count.get("value"), unexpected_count.get("count")]
                     )


Test patch:
Repository: great-expectations/great_expectations

Issue:
[BUG] Duplicate `usage_statistics` message is sent when building data_docs with CLI
**Describe the bug**
When building DataDocs through the CLI (handled by the `build_docs()` function), the DataContext's `context.build_data_docs()` function is called twice. 

First `context.build_data_docs()` is called using the `dry_run` flag, which returns a structure containing the URLs of the sites that *would* be built, and allowing users to confirm. It is called a second time when the DataDocs are **actually** built.

The way that `context.build_data_docs()` sends `usage_statistics` is by using the `usage_statistics_enabled_method` decorator, which is not able to distinguish a `dry_run` from a real one. This causes a duplicate message to be sent. 

**To Reproduce**
1. Put `print(message)` statement in the `emit()` function in `UsageStatisticsHandler` class in `great_expectations/core/usage_statistics.py`
2. Run `great_expectations docs build` or run `great_expectation init`, choose any data source, say "yes" to all prompts until Data Docs are built. 
3. Observe two `event: data_context.build_data_docs` emitted. 

**Expected behavior**
There should only be one since DataDocs are only being built once.

**Proposed Change**
* `usage_statistics_wrapped_method` decorator will check if `dry_run` is part of the **kwargs, and return undecorated function if `dry_run` is set to True
*  this will allow `context.build_data_docs()` to only emit a message if the `dry_run` is False (as well as handle future functions that may need to be called more than once)

**Environment:**
 - OS: MacOS
 - GE Version: 0.11.8





Discussion:

Resolving patch:
diff --git a/great_expectations/core/usage_statistics/usage_statistics.py b/great_expectations/core/usage_statistics/usage_statistics.py
--- a/great_expectations/core/usage_statistics/usage_statistics.py
+++ b/great_expectations/core/usage_statistics/usage_statistics.py
@@ -244,6 +244,10 @@ def usage_statistics_enabled_method(
 
         @wraps(func)
         def usage_statistics_wrapped_method(*args, **kwargs):
+            # if a function like `build_data_docs()` is being called as a `dry_run`
+            # then we dont want to emit usage_statistics. We just return the function without sending a usage_stats message
+            if "dry_run" in kwargs and kwargs["dry_run"]:
+                return func(*args, **kwargs)
             # Set event_payload now so it can be updated below
             event_payload = {}
             message = {"event_payload": event_payload, "event": event_name}


Test patch:
diff --git a/tests/cli/test_init_pandas.py b/tests/cli/test_init_pandas.py
--- a/tests/cli/test_init_pandas.py
+++ b/tests/cli/test_init_pandas.py
@@ -156,8 +156,7 @@ def test_cli_init_on_new_project(
 """
     )
 
-    # data_context.build_docs is twice (once in dry run mode) and two events are fired
-    assert mock_emit.call_count == 10
+    assert mock_emit.call_count == 9
     assert mock_emit.call_args_list[1] == mock.call(
         {"event_payload": {}, "event": "cli.init.create", "success": True}
     )
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/data_asset/data_asset.py b/great_expectations/data_asset/data_asset.py
--- a/great_expectations/data_asset/data_asset.py
+++ b/great_expectations/data_asset/data_asset.py
@@ -1336,7 +1336,7 @@ def test_expectation_function(self, function, *args, **kwargs):
             define custom classes, etc. To use developed expectations from the command-line tool, you will still need \
             to define custom classes, etc.
 
-            Check out :ref:`custom_expectations_reference` for more information.
+            Check out :ref:`how_to_guides__creating_and_editing_expectations__how_to_create_custom_expectations` for more information.
         """
 
         argspec = inspect.getfullargspec(function)[0][1:]
diff --git a/great_expectations/dataset/dataset.py b/great_expectations/dataset/dataset.py
--- a/great_expectations/dataset/dataset.py
+++ b/great_expectations/dataset/dataset.py
@@ -349,7 +349,7 @@ def test_column_map_expectation_function(self, function, *args, **kwargs):
             define custom classes, etc. To use developed expectations from the command-line tool, you'll still need to \
             define custom classes, etc.
 
-            Check out :ref:`custom_expectations_reference` for more information.
+            Check out :ref:`how_to_guides__creating_and_editing_expectations__how_to_create_custom_expectations` for more information.
         """
 
         new_function = self.column_map_expectation(function)
@@ -371,7 +371,7 @@ def test_column_aggregate_expectation_function(self, function, *args, **kwargs):
             define custom classes, etc. To use developed expectations from the command-line tool, you'll still need to \
             define custom classes, etc.
 
-            Check out :ref:`custom_expectations_reference` for more information.
+            Check out :ref:`how_to_guides__creating_and_editing_expectations__how_to_create_custom_expectations` for more information.
         """
 
         new_function = self.column_aggregate_expectation(function)


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -46,7 +46,7 @@
     "sphinx.ext.napoleon",
     "sphinxcontrib.contentui",
     "sphinx_gitstamp",
-    "sphinx.ext.autosectionlabel",
+    # "sphinx.ext.autosectionlabel",
     "sphinxcontrib.discourse",
     "autoapi.extension",
 ]
@@ -54,6 +54,7 @@
 autoapi_type = "python"
 autoapi_dirs = ["../great_expectations"]
 autoapi_add_toctree_entry = False
+# autoapi_keep_files = True
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]


Test patch:
diff --git a/docs/contributing/testing.rst b/docs/contributing/testing.rst
--- a/docs/contributing/testing.rst
+++ b/docs/contributing/testing.rst
@@ -55,7 +55,7 @@ To accomplish this, Great Expectations encapsulates unit tests for Expectations
 
 Test fixture files are structured as follows:
 
-.. code-block:: json
+.. code-block::
 
     {
         "expectation_type" : "expect_column_max_to_be_between",
@@ -73,7 +73,7 @@ Each item under ``datasets`` includes three entries: ``data``, ``schemas``, and
 
     ...defines a dataframe of sample data to apply Expectations against. The dataframe is defined as a dictionary of lists, with keys containing column names and values containing lists of data entries. All lists within a dataset must have the same length.
 
-    .. code-block:: json
+    .. code-block::
 
         "data" : {
             "w" : [1, 2, 3, 4, 5, 5, 4, 3, 2, 1],
@@ -89,7 +89,7 @@ Each item under ``datasets`` includes three entries: ``data``, ``schemas``, and
 
     ...define the types to be used when instantiating tests against different execution environments, including different SQL dialects. Each schema is defined as dictionary with column names and types as key-value pairs. If the schema isn't specified for a given execution environment, Great Expectations will introspect values and attempt to guess the schema.
 
-    .. code-block:: json
+    .. code-block::
 
         "schemas": {
             "sqlite": {
@@ -120,7 +120,7 @@ Each item under ``datasets`` includes three entries: ``data``, ``schemas``, and
 
     See an example below. For other examples
 
-    .. code-block:: json
+    .. code-block::
 
         "tests" : [{
             "title": "Basic negative test case",
Repository: great-expectations/great_expectations

Issue:
docs build cli not obeying --yes parameter
**Describe the bug**
The `great_expectations docs build --yes` command still asks for confirmation even though the `--yes` parameter has been provided.
```bash
$ great_expectations docs build --yes

The following Data Docs sites will be built:

 - local_site: file:///home/varun.patil/ge_example/great_expectations/uncommitted/data_docs/local_site/index.html

Would you like to proceed? [Y/n]:
```

**To Reproduce**
Steps to reproduce the behavior:
Run the `great_expectations docs build --yes` command.

**Expected behavior**
The command should not ask for user confirmation when the `--yes` argument is provided.

**Environment (please complete the following information):**
 - OS: Ubuntu 18.04
 - GE Version: 0.12.2

**Additional context**
The problem is because of the wrong usage of the click options.
https://github.com/great-expectations/great_expectations/blob/8c06fa6ddd18ba2b58b529ff9a67f53663b21319/great_expectations/cli/docs.py#L36-L41

This should instead be written as:
```python
@click.option(
    "--assume-yes",
    "--yes",
    "-y",
    is_flag=True,
    help="By default request confirmation to build docs unless you specify -y/--yes/--assume-yes flag to skip dialog",
    default=False,
)
```


Discussion:

Resolving patch:
diff --git a/great_expectations/cli/docs.py b/great_expectations/cli/docs.py
--- a/great_expectations/cli/docs.py
+++ b/great_expectations/cli/docs.py
@@ -34,8 +34,10 @@ def docs():
     default=True,
 )
 @click.option(
-    "--assume-yes/--yes",
+    "--assume-yes",
+    "--yes",
     "-y",
+    is_flag=True,
     help="By default request confirmation to build docs unless you specify -y/--yes/--assume-yes flag to skip dialog",
     default=False,
 )


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/render/renderer/suite_edit_notebook_renderer.py b/great_expectations/render/renderer/suite_edit_notebook_renderer.py
--- a/great_expectations/render/renderer/suite_edit_notebook_renderer.py
+++ b/great_expectations/render/renderer/suite_edit_notebook_renderer.py
@@ -8,6 +8,7 @@
 from great_expectations.core.id_dict import BatchKwargs
 from great_expectations.data_context.types.base import (
     NotebookTemplateConfig,
+    NotebookConfig,
     notebookConfigSchema,
 )
 from great_expectations.data_context.util import instantiate_class_from_config


Test patch:
Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`


Discussion:
Done inline during improve_partitions push.

Resolving patch:
diff --git a/great_expectations/render/renderer/site_builder.py b/great_expectations/render/renderer/site_builder.py
--- a/great_expectations/render/renderer/site_builder.py
+++ b/great_expectations/render/renderer/site_builder.py
@@ -642,7 +642,7 @@ def _get_call_to_action_buttons(self, usage_statistics):
         create_expectations = CallToActionButton(
             "How to Create Expectations",
             # TODO update this link to a proper tutorial
-            "https://docs.greatexpectations.io/en/latest/how_to_guides/creating_and_editing_expectations.html",
+            "https://docs.greatexpectations.io/en/latest/guides/how_to_guides/creating_and_editing_expectations.html",
         )
         see_glossary = CallToActionButton(
             "See More Kinds of Expectations",
@@ -655,11 +655,11 @@ def _get_call_to_action_buttons(self, usage_statistics):
         )
         customize_data_docs = CallToActionButton(
             "How to Customize Data Docs",
-            "https://docs.greatexpectations.io/en/latest/reference/data_docs_reference.html#customizing-data-docs",
+            "https://docs.greatexpectations.io/en/latest/reference/core_concepts.html#data-docs",
         )
         team_site = CallToActionButton(
             "How to Set Up a Team Site",
-            "https://docs.greatexpectations.io/en/latest/how_to_guides/configuring_data_docs.html",
+            "https://docs.greatexpectations.io/en/latest/guides/how_to_guides/configuring_data_docs.html",
         )
         # TODO gallery does not yet exist
         # gallery = CallToActionButton(


Test patch:
Repository: great-expectations/great_expectations

Issue:
"$" character in user's response to a CLI prompt is mistakenly interpreted as a variable to substitute
**Describe the bug**
"$" character in user's response to a CLI prompt is mistakenly interpreted as a variable to substitute

![image](https://user-images.githubusercontent.com/80363/94323126-72169700-ff49-11ea-85fc-5dfba04ceba1.png)


**To Reproduce**
Steps to reproduce the behavior:
1. Add a Datasource in the CLI
2. Choose a database, answer one of the prompts (e.g., password) with a string that contains the "$" character
3. See the error

**Expected behavior**
The "$" character should not be treated specially in user's answer to prompts

**Environment (please complete the following information):**
 - OS: N/A
 - GE Version: 0.12.1

**Additional context**
N/A



Discussion:
This issue went through triage - the core team will develop a fix.

Resolving patch:
diff --git a/great_expectations/data_context/util.py b/great_expectations/data_context/util.py
--- a/great_expectations/data_context/util.py
+++ b/great_expectations/data_context/util.py
@@ -165,6 +165,8 @@ def substitute_all_config_variables(data, replace_variables_dict):
     if isinstance(data, dict) or isinstance(data, OrderedDict):
         return {
             k: substitute_all_config_variables(v, replace_variables_dict)
+            if k.lower() not in {"password"}
+            else v
             for k, v in data.items()
         }
     elif isinstance(data, list):


Test patch:
diff --git a/tests/data_context/test_data_context_config_variables.py b/tests/data_context/test_data_context_config_variables.py
--- a/tests/data_context/test_data_context_config_variables.py
+++ b/tests/data_context/test_data_context_config_variables.py
@@ -241,3 +241,15 @@ def test_substitute_env_var_in_config_variable_file(
     ]["reader_options"]["test_variable_sub4"] == {
         "inner_env_sub": "correct_val_of_replace_me"
     }
+    assert (
+        context_config["datasources"]["mydatasource"]["batch_kwargs_generators"][
+            "mygenerator"
+        ]["reader_options"]["password"]
+        == "dont$replaceme"
+    )
+    assert (
+        context_config["datasources"]["mydatasource"]["batch_kwargs_generators"][
+            "mygenerator"
+        ]["password"]
+        == "dont$replace$me$please$$$$thanks"
+    )
diff --git a/tests/test_fixtures/great_expectations_basic_with_variables.yml b/tests/test_fixtures/great_expectations_basic_with_variables.yml
--- a/tests/test_fixtures/great_expectations_basic_with_variables.yml
+++ b/tests/test_fixtures/great_expectations_basic_with_variables.yml
@@ -13,9 +13,11 @@ datasources:
       mygenerator:
         class_name: SubdirReaderBatchKwargsGenerator
         base_directory: ../data
+        password: dont$replace$me$please$$$$thanks
         reader_options:
           sep:
           engine: python
+          password: dont$replaceme
           test_variable_sub1: ${replace_me}
           test_variable_sub2: $replace_me
           test_variable_sub3: ${replace_me_1}
Repository: great-expectations/great_expectations

Issue:
Failed To Provide partial_unexpected_counts in Validation Results
**Describe the bug**
GE is failing to give correct "**partial_unexpected_counts**" in validation  json result after running "**action_list_operator**.
When I am Validating Column Type with **Integer** ,which has mixed data types in it, then in GE JSON result i am seeing like
**"partial_unexpected_counts": ["partial_exception_counts requires a hashable type"]** as one key-value pair.

This is Happening with all versions of GE and this is due to **Counter** that we are using to calculate "partial_unexpected_counts" in **great_expectations\data_asset\data_asset.py** file.
**To Reproduce**
Steps to reproduce the behavior:
1. create a batch with dataframe like df=pd.DataFrame(
    { 
        'Id' :[ 45.02 , 21.123 , 40.12 , 25.122 , "43" ],
        'Name' :[ ' %% JobID %% ' , 21.0 , 40 , 21 , np.nan],
        'Age' :[ ' %% JobID %% ' , 21.0 , 40.0 , 21 , np.nan],
        'Age2' :[ ' %% JobID %% ' , 21.0 , 40.0 , 2 , np.nan]
    }
)
2. create an expectation suite with batch.expect_column_values_to_be_of_type('Id','int')
3. save expectation_suite with above 1 expectation
4. run  

- 1. results = context.run_validation_operator("action_list_operator", assets_to_validate=[batch], run_id=run_id)
- 2. expectation_suite_identifier = ExpectationSuiteIdentifier(expectation_suite_name)
- 3. validation_result_identifier = ValidationResultIdentifier(expectation_suite_identifier=expectation_suite_identifier, 
batch_identifier=batch.batch_kwargs.to_id(), run_id=run_id')

5. Verify the  **partial_unexpected_counts**  value in Json Result under your **Validation_Store** 
Please find GE Result in below attached file
![Screenshot (24)](https://user-images.githubusercontent.com/51196441/96203074-95e46000-0f7e-11eb-95c3-60a7d0df6b57.png)





**Expected behavior**
Json Validation Result should have like this  "partial_unexpected_counts": [{"value": 45.02, "count": 1}, {"value": "21.123", "count": 1},{"value": "25.122", "count": 1}]

**Environment (please complete the following information):**
 - OS: All
 - GE Version: All [Latest Too]
**Additional context**
@alexsherstinsky @eugmandel 


Discussion:
@prajwal-kesava Thank you for providing easy-to-follow steps to reproduced! I see the same error that you are reporting. Marking this "core-team-priority", but not sure when we will get to it. In the meantime, if there is a community PR that addresses this, it will be very much welcome.

Resolving patch:
diff --git a/great_expectations/data_asset/data_asset.py b/great_expectations/data_asset/data_asset.py
--- a/great_expectations/data_asset/data_asset.py
+++ b/great_expectations/data_asset/data_asset.py
@@ -1155,7 +1155,7 @@ def _format_map_output(
                         Counter(unexpected_list).most_common(
                             result_format["partial_unexpected_count"]
                         ),
-                        key=lambda x: (-x[1], x[0]),
+                        key=lambda x: (-x[1], str(x[0])),
                     )
                 ]
             except TypeError:


Test patch:
Repository: great-expectations/great_expectations

Issue:
Creating a new expectation via CLI / SQL Query throws TypeError 'missing execution_engine' error
**Describe the bug**
Just upgraded to version 0.13.0. When creating a new expectation using the CLI, GE throws an error when I provide a SQL query as an alternative to the top 5 tables that are recommended (i.e., _Do not see the table in the list above? Just type the SQL query_)

This throws error: `TypeError: __init__() missing 1 required positional argument: 'execution_engine'`


**To Reproduce**
Steps to reproduce the behavior:
1. great_expectations suite edit my_suite
2. SELECT * FROM my_database.my_schema.my_table

**Expected behavior**
Launch the Jupyter Notebook with the suite editing interface for the corresponding query result.

**Environment (please complete the following information):**
 - OS: Windows
 - GE Version: [e.g. 0.13.0]

**Additional context**  

I am using the Snowflake connector, configured as described in the docs.

I was able to use this workflow without issue prior to updating to 0.13.0.

I am able to generate expectations by running `great_expectations suite edit my_suite` selecting one of the top 5 tables, and then updating the batch_kwargs in the jupyter notebook with the desired table, for what it's worth.


Discussion:

Resolving patch:
diff --git a/great_expectations/cli/datasource.py b/great_expectations/cli/datasource.py
--- a/great_expectations/cli/datasource.py
+++ b/great_expectations/cli/datasource.py
@@ -39,7 +39,7 @@
     BatchKwargsError,
     DatasourceInitializationError,
 )
-from great_expectations.validator.validator import Validator
+from great_expectations.validator.validator import BridgeValidator
 
 logger = logging.getLogger(__name__)
 
@@ -1324,7 +1324,7 @@ def _get_batch_kwargs_for_sqlalchemy_datasource(
             else:
                 batch_kwargs = {"query": query, "datasource": datasource_name}
                 batch_kwargs.update(temp_table_kwargs)
-                Validator(
+                BridgeValidator(
                     batch=datasource.get_batch(batch_kwargs),
                     expectation_suite=ExpectationSuite("throwaway"),
                 ).get_dataset()


Test patch: