Repository: great-expectations/great_expectations

Issue:
Regex Error with BigQuery
We are having an issue running the following expectation on a string column:
```python
batch.expect_column_values_to_match_regex("UNIQUE_ID", "^[0-9]*$", catch_exceptions=True)
```

Error is as follow:

DatabaseError: (google.cloud.bigquery.dbapi.exceptions.DatabaseError) 400 Syntax error: Parenthesized expression cannot be parsed as an expression, struct constructor, or subquery at [1:154] [SQL: 'SELECT count(*) AS `element_count`, sum(CASE WHEN (`UNIQUE_ID` IS NULL) THEN %(param_1)s ELSE %(param_2)s END) AS `null_count`, sum(CASE WHEN (NOT (`UNIQUE_ID` REGEXP_CONTAINS %(param_3)s) AND `UNIQUE_ID` IS NOT NULL) THEN %(param_4)s ELSE %(param_5)s END) AS `unexpected_count` \\nFROM `dataset.raw_table`'] [parameters: {'param_1': 1, 'param_2': 0, 'param_3': '^[0-9]*$', 'param_4': 1, 'param_5': 0}] (Background on this error at: http://sqlalche.me/e/4xp6)"

When checking the query in BQ, we see that the NOT in sum(CASE WHEN (**NOT** ( is throwing a syntax error.

Luckily, the regex is quite simple...and as a temporary work around we are casting column as int and doing a null check. More complicated regex would certainly fail. Note: We prefer to run ge on raw tables whenever possible, as maintaining custom SQL is one more thing that we have to update in our pipeline.

Thanks in advance!



Discussion:
Wow -- this one looks tricky.

It appears to me that what's going on is that bigquery updated its query language, and the generated query is straddling the two.

In particular, under the "legacy sql" dialect, this particular parenthesized expression is valid...but the REGEXP_CONTAINS function is not (it was called REGEXP_MATCH and has a slightly different syntax).

On the other hand, the new "standard sql" dialect doesn't support this parenthesized expression. I *suspect* the pybigquery team will be amenable to looking at this more closely (and we could probably get a PR for them). What version of pybigquery are you using?
I've continued to look at this, and actually think the above is at least incomplete: the underlying issue seems to be that GE currently treats all regular expression matching as a binary expression, but in the bigquery dialect, it's actually a (binary) function. I think an easier fix will be to update GE to generate a function in this case.

Resolving patch:
diff --git a/great_expectations/dataset/sqlalchemy_dataset.py b/great_expectations/dataset/sqlalchemy_dataset.py
--- a/great_expectations/dataset/sqlalchemy_dataset.py
+++ b/great_expectations/dataset/sqlalchemy_dataset.py
@@ -976,38 +976,53 @@ def expect_column_values_to_be_unique(self, column, mostly=None,
 
         return sa.column(column).notin_(dup_query)
 
-    def _get_dialect_regex_fn(self, positive=True):
+    def _get_dialect_regex_expression(self, column, regex, positive=True):
         try:
             # postgres
             if isinstance(self.engine.dialect, sa.dialects.postgresql.dialect):
-                return "~" if positive else "!~"
+                if positive:
+                    return BinaryExpression(sa.column(column), literal(regex), custom_op("~"))
+                else:
+                    return BinaryExpression(sa.column(column), literal(regex), custom_op("!~"))
         except AttributeError:
             pass
 
         try:
             # redshift
             if isinstance(self.engine.dialect, sqlalchemy_redshift.dialect.RedshiftDialect):
-                return "~" if positive else "!~"
+                if positive:
+                    return BinaryExpression(sa.column(column), literal(regex), custom_op("~"))
+                else:
+                    return BinaryExpression(sa.column(column), literal(regex), custom_op("!~"))
         except (AttributeError, TypeError):  # TypeError can occur if the driver was not installed and so is None
             pass
         try:
             # Mysql
             if isinstance(self.engine.dialect, sa.dialects.mysql.dialect):
-                return "REGEXP" if positive else "NOT REGEXP"
+                if positive:
+                    return BinaryExpression(sa.column(column), literal(regex), custom_op("REGEXP"))
+                else:
+                    return BinaryExpression(sa.column(column), literal(regex), custom_op("NOT REGEXP"))
         except AttributeError:
             pass
 
         try:
             # Snowflake
             if isinstance(self.engine.dialect, snowflake.sqlalchemy.snowdialect.SnowflakeDialect):
-                return "RLIKE" if positive else "NOT RLIKE"
+                if positive:
+                    return BinaryExpression(sa.column(column), literal(regex), custom_op("RLIKE"))
+                else:
+                    return BinaryExpression(sa.column(column), literal(regex), custom_op("NOT RLIKE"))
         except (AttributeError, TypeError):  # TypeError can occur if the driver was not installed and so is None
             pass
 
         try:
             # Bigquery
             if isinstance(self.engine.dialect, pybigquery.sqlalchemy_bigquery.BigQueryDialect):
-                return "REGEXP_CONTAINS" if positive else "NOT REGEXP_CONTAINS"
+                if positive:
+                    return sa.func.REGEXP_CONTAINS(sa.column(column), literal(regex))
+                else:
+                    return sa.not_(sa.func.REGEXP_CONTAINS(sa.column(column), literal(regex)))
         except (AttributeError, TypeError):  # TypeError can occur if the driver was not installed and so is None
             pass
 
@@ -1019,13 +1034,12 @@ def expect_column_values_to_match_regex(
             mostly=None,
             result_format=None, include_config=True, catch_exceptions=None, meta=None
     ):
-
-        regex_fn = self._get_dialect_regex_fn(positive=True)
-        if regex_fn is None:
+        regex_expression = self._get_dialect_regex_expression(column, regex, positive=True)
+        if regex_expression is None:
             logger.warning("Regex is not supported for dialect %s" % str(self.engine.dialect))
             raise NotImplementedError
 
-        return BinaryExpression(sa.column(column), literal(regex), custom_op(regex_fn))
+        return regex_expression
 
     @MetaSqlAlchemyDataset.column_map_expectation
     def expect_column_values_to_not_match_regex(
@@ -1035,12 +1049,12 @@ def expect_column_values_to_not_match_regex(
             mostly=None,
             result_format=None, include_config=True, catch_exceptions=None, meta=None
     ):
-        regex_fn = self._get_dialect_regex_fn(positive=False)
-        if regex_fn is None:
+        regex_expression = self._get_dialect_regex_expression(column, regex, positive=False)
+        if regex_expression is None:
             logger.warning("Regex is not supported for dialect %s" % str(self.engine.dialect))
             raise NotImplementedError
 
-        return BinaryExpression(sa.column(column), literal(regex), custom_op(regex_fn))
+        return regex_expression
 
     @MetaSqlAlchemyDataset.column_map_expectation
     def expect_column_values_to_match_regex_list(self,
@@ -1054,20 +1068,23 @@ def expect_column_values_to_match_regex_list(self,
         if match_on not in ["any", "all"]:
             raise ValueError("match_on must be any or all")
 
-        regex_fn = self._get_dialect_regex_fn(positive=True)
-        if regex_fn is None:
+        if len(regex_list) == 0:
+            raise ValueError("At least one regex must be supplied in the regex_list.")
+
+        regex_expression = self._get_dialect_regex_expression(column, regex_list[0], positive=True)
+        if regex_expression is None:
             logger.warning("Regex is not supported for dialect %s" % str(self.engine.dialect))
             raise NotImplementedError
 
         if match_on == "any":
             condition = \
                 sa.or_(
-                    *[BinaryExpression(sa.column(column), literal(regex), custom_op(regex_fn)) for regex in regex_list]
+                    *[self._get_dialect_regex_expression(column, regex, positive=True) for regex in regex_list]
                 )
         else:
             condition = \
                 sa.and_(
-                    *[BinaryExpression(sa.column(column), literal(regex), custom_op(regex_fn)) for regex in regex_list]
+                    *[self._get_dialect_regex_expression(column, regex, positive=True) for regex in regex_list]
                 )
         return condition
 
@@ -1075,12 +1092,14 @@ def expect_column_values_to_match_regex_list(self,
     def expect_column_values_to_not_match_regex_list(self, column, regex_list,
                                                      mostly=None,
                                                      result_format=None, include_config=True, catch_exceptions=None, meta=None):
+        if len(regex_list) == 0:
+            raise ValueError("At least one regex must be supplied in the regex_list.")
 
-        regex_fn = self._get_dialect_regex_fn(positive=False)
-        if regex_fn is None:
+        regex_expression = self._get_dialect_regex_expression(column, regex_list[0], positive=False)
+        if regex_expression is None:
             logger.warning("Regex is not supported for dialect %s" % str(self.engine.dialect))
             raise NotImplementedError
 
         return sa.and_(
-            *[BinaryExpression(sa.column(column), literal(regex), custom_op(regex_fn)) for regex in regex_list]
+            *[self._get_dialect_regex_expression(column, regex, positive=False) for regex in regex_list]
         )


Test patch: