Repository: great-expectations/great_expectations

Issue:
Use cleaner solution for non-truncating division in python 2
Prefer `from __future__ import division` to `1.*x/y`
Expect_column_values_to_be_unique failed on files without header
**Describe the bug**
Create Expect_column_values_to_be_unique rule on file without header Failed.

**To Reproduce**
Steps to reproduce the behavior:
1. Use UserConfigurableProfiler to automatic create expectations on files without header
2. Set UserConfigurableProfiler attribute primary_or_compound_key to [ _c0 ]
3. See error
```
Traceback (most recent call last):
  File "/home/biteam/great_expectations/execute_ge.py", line 96, in <module>
    suite = profiler.build_suite()
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/profile/user_configurable_profiler.py", line 212, in build_suite
    return self._profile_and_build_expectation_suite()
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/profile/user_configurable_profiler.py", line 272, in _profile_and_build_expectation_suite
    self._build_expectations_primary_or_compound_key(
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/profile/user_configurable_profiler.py", line 922, in _build_expectations_primary_or_compound_key
    profile_dataset.expect_column_values_to_be_unique(column)
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/validator/validator.py", line 304, in inst_expectation
    raise err
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/validator/validator.py", line 262, in inst_expectation
    validation_result = expectation.validate(
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 741, in validate
    evr = validator.graph_validate(
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/validator/validator.py", line 529, in graph_validate
    raise err
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/validator/validator.py", line 477, in graph_validate
    ] = self.resolve_validation_graph(
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/validator/validator.py", line 707, in resolve_validation_graph
    raise err
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/validator/validator.py", line 673, in resolve_validation_graph
    self._resolve_metrics(
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/validator/validator.py", line 762, in _resolve_metrics
    return execution_engine.resolve_metrics(
  File "/home/biteam/.local/lib/python3.8/site-packages/great_expectations/execution_engine/execution_engine.py", line 298, in resolve_metrics
    raise ge_exceptions.MetricResolutionError(
great_expectations.exceptions.exceptions.MetricResolutionError: unable to parse condition: col("_c0").notnull()
Calculating Metrics:  60%|███████████████████████████████████████████████████████████████████████████████▏                                                    | 6/10 [00:00<00:00, 40.58it/s]
```
**Expected behavior**
Create expectations according to the files without header and Expect_column_values_to_be_unique rule on _c0


**Environment (please complete the following information):**
 - Operating System: Linux (Ubuntu 20.04)
 - Great Expectations Version: 0.13.40

**Additional context**
I create files without header by tutorials file


Discussion:
Done inline during improve_partitions push.


Hey @nf-jimmysu! Thanks for opening up this issue.

Would you mind providing us with a few pertinent details:
- Which tutorial/doc are you following?
- Are you able to instantiate a validator?
- Could you please provide your configuration (what backend are you using)?

We'll take those details and try to narrow down the issue at hand. Thanks!
Hi @cdkini ! Thanks for your reply.
1. I'm following the tutorial of ge_tutorials with data yellow_tripdata_sample_2019-01.csv & yellow_tripdata_sample_2019-02.csv. I load this files to create files without header.
2. I am able to create validator with header and also without header (if I don't add primary_key when I create it)
3. I'm using ubuntu 20.04, great-expecatation 0.13.40 & pyspark 3.2.0

Thank you!
Hey @nf-jimmysu! Are you able to provide your config file (stripped of any sensitive information!) and your batch_request? That would help us better identify the root cause here.
Data Source config
```
datasource_yaml = f"""
name: {datasource_name}
class_name: Datasource
execution_engine:
  class_name: SparkDFExecutionEngine
data_connectors:
  default_inferred_data_connector_name:
    class_name: InferredAssetFilesystemDataConnector
    base_directory: {datasource_path}
    default_regex:
      group_names:
        - data_asset_name
      pattern: (.*)
  default_runtime_data_connector_name:
    class_name: RuntimeDataConnector
    batch_identifiers:
      - default_identifier_name
"""
```

Batch Request
```
batch_request = BatchRequest(
    datasource_name=datasource_name,
    data_connector_name="default_inferred_data_connector_name",
    data_asset_name=example_file_name,
    limit=1000,
    batch_spec_passthrough={"reader_method": "csv", "reader_options": {"header": False, "inferSchema":True}},
)
```

Validation Profiler
```
profiler = UserConfigurableProfiler(
    profile_dataset=validator,
    excluded_expectations=[],
    ignored_columns=[],
    not_null_only=False,
    primary_or_compound_key=['_c0'],
    semantic_types_dict=None,
    table_expectations_only=False,
    value_set_threshold="MANY",
)
```

Thank You for your reply!

Resolving patch:
diff --git a/great_expectations/execution_engine/sparkdf_execution_engine.py b/great_expectations/execution_engine/sparkdf_execution_engine.py
--- a/great_expectations/execution_engine/sparkdf_execution_engine.py
+++ b/great_expectations/execution_engine/sparkdf_execution_engine.py
@@ -592,15 +592,7 @@ class MetricDomainTypes.
     def add_column_row_condition(
         self, domain_kwargs, column_name=None, filter_null=True, filter_nan=False
     ):
-        if filter_nan is False:
-            return super().add_column_row_condition(
-                domain_kwargs=domain_kwargs,
-                column_name=column_name,
-                filter_null=filter_null,
-                filter_nan=filter_nan,
-            )
-
-        # We explicitly handle filter_nan for spark using a spark-native condition
+        # We explicitly handle filter_nan & filter_null for spark using a spark-native condition
         if "row_condition" in domain_kwargs and domain_kwargs["row_condition"]:
             raise GreatExpectationsError(
                 "ExecutionEngine does not support updating existing row_conditions."


Test patch:
diff --git a/tests/execution_engine/test_sparkdf_execution_engine.py b/tests/execution_engine/test_sparkdf_execution_engine.py
--- a/tests/execution_engine/test_sparkdf_execution_engine.py
+++ b/tests/execution_engine/test_sparkdf_execution_engine.py
@@ -1145,7 +1145,7 @@ def test_add_column_row_condition(spark_session, basic_spark_df_execution_engine
     new_domain_kwargs = engine.add_column_row_condition(
         domain_kwargs, filter_null=True, filter_nan=False
     )
-    assert new_domain_kwargs["row_condition"] == 'col("foo").notnull()'
+    assert new_domain_kwargs["row_condition"] == "foo IS NOT NULL"
     df, cd, ad = engine.get_compute_domain(new_domain_kwargs, domain_type="table")
     res = df.collect()
     assert res == [(1,), (2,), (3,), (3,), (2,), (3,), (4,), (5,), (6,)]
@@ -1183,7 +1183,7 @@ def test_add_column_row_condition(spark_session, basic_spark_df_execution_engine
     new_domain_kwargs = engine.add_column_row_condition(
         domain_kwargs, filter_null=True, filter_nan=False
     )
-    assert new_domain_kwargs["row_condition"] == 'col("foo").notnull()'
+    assert new_domain_kwargs["row_condition"] == "foo IS NOT NULL"
     df, cd, ad = engine.get_compute_domain(new_domain_kwargs, domain_type="table")
     res = df.collect()
     expected = [(1,), (2,), (3,), (3,), (np.nan,), (2,), (3,), (4,), (5,), (6,)]
diff --git a/tests/test_definitions/column_map_expectations/expect_column_values_to_be_unique.json b/tests/test_definitions/column_map_expectations/expect_column_values_to_be_unique.json
--- a/tests/test_definitions/column_map_expectations/expect_column_values_to_be_unique.json
+++ b/tests/test_definitions/column_map_expectations/expect_column_values_to_be_unique.json
@@ -9,7 +9,9 @@
       "n" : [null, null, 2, null],
       "unique" : [1, 2, 3, null],
       "null": [null, null, null, null],
-      "mult_dup": ["1", "1", "2", "2"]
+      "mult_dup": ["1", "1", "2", "2"],
+      "0": [1, 2, 3, 4],
+      "_c0": [1, 2, 3, 4]
     },
     "schemas": {
       "spark": {
@@ -20,7 +22,9 @@
         "n": "IntegerType",
         "unique": "IntegerType",
         "null": "NullType",
-        "mult_dup": "StringType"
+        "mult_dup": "StringType",
+        "0": "IntegerType",
+        "_c0": "IntegerType"
       },
       "mssql": {
         "a": "VARCHAR",
@@ -30,7 +34,9 @@
         "n": "INTEGER",
         "unique": "INTEGER",
         "null": "NVARCHAR",
-        "mult_dup": "VARCHAR"
+        "mult_dup": "VARCHAR",
+        "0": "INTEGER",
+        "_c0": "INTEGER"
       },
       "mysql": {
         "a": "CHAR",
@@ -40,7 +46,9 @@
         "n": "INTEGER",
         "unique": "INTEGER",
         "null": "CHAR",
-        "mult_dup": "CHAR"
+        "mult_dup": "CHAR",
+        "0": "INTEGER",
+        "_c0": "INTEGER"
       }
     },
     "tests" : [
@@ -147,6 +155,26 @@
           "unexpected_list": ["1", "1", "2", "2"],
           "unexpected_index_list": [0, 1, 2, 3]
         }
+      },
+      {
+        "title": "positive_integer_column_name",
+        "exact_match_out": false,
+        "in": {
+          "column": "0"
+        },
+        "out": {
+          "success": true
+        }
+      },
+      {
+        "title": "positive_spark_headerless_column_name",
+        "exact_match_out": false,
+        "in": {
+          "column": "_c0"
+      },
+        "out": {
+          "success": true
+        }
       }
     ]
   }]