Repository: spectacles-ci/spectacles

Issue:
Recommend re-running in hybrid/single mode when errors found in batch mode
When spectacles is run in batch mode and encounters an error, we should suggest the user re-run the explore (maybe even pre-populating the command with the explore argument in the output) in single-dimension mode to zero in on the dimensions causing the error.

Here's a suggested output:

```
To determine the exact dimensions responsible for this error, you can re-run this explore in single-dimension mode.

You can also run this original validation with `--mode hybrid` to do this automatically.
```

Discussion:
DylanBaker: One error: 
```============== Testing 1 explore [batch mode] [concurrency = 10] ===============

✗ jaffle_shop.fct_orders failed

Completed SQL validation in 2 seconds.

================================== fct_orders ==================================

The Google BigQuery Standard SQL database encountered an error while running
this query. Failed to retrieve data - Name customer_lifetime_value_ not found
inside dim_customers at [3:23]

Test SQL: logs/queries/jaffle_shop__fct_orders.sql


To determine the exact dimensions responsible for this error, you can re-run 
Spectacles in single-dimension mode, with `--mode single`.

You can also run this original validation with `--mode hybrid` to do this automatically.
```

Multiple errors:
```
============== Testing 2 explores [batch mode] [concurrency = 10] ==============

✗ jaffle_shop.dim_customers failed
✗ jaffle_shop.fct_orders failed

Completed SQL validation in 3 seconds.

================================== fct_orders ==================================

The Google BigQuery Standard SQL database encountered an error while running
this query. Failed to retrieve data - Name customer_lifetime_value_ not found
inside dim_customers at [3:23]

Test SQL: logs/queries/jaffle_shop__fct_orders.sql

================================= dim_customers ================================

The Google BigQuery Standard SQL database encountered an error while running
this query. Failed to retrieve data - Name customer_lifetime_value_ not found
inside dim_customers at [3:23]

Test SQL: logs/queries/jaffle_shop__dim_customers.sql


To determine the exact dimensions responsible for these errors, you can re-run 
Spectacles in single-dimension mode, with `--mode single`.

You can also run this original validation with `--mode hybrid` to do this automatically.```
codecov[bot]: # [Codecov](https://codecov.io/gh/spectacles-ci/spectacles/pull/190?src=pr&el=h1) Report
> Merging [#190](https://codecov.io/gh/spectacles-ci/spectacles/pull/190?src=pr&el=desc) into [master](https://codecov.io/gh/spectacles-ci/spectacles/commit/66892cc1d35068162b29040f9034733c983059f8&el=desc) will **decrease** coverage by `0.59%`.
> The diff coverage is `25.00%`.

[![Impacted file tree graph](https://codecov.io/gh/spectacles-ci/spectacles/pull/190/graphs/tree.svg?width=650&height=150&src=pr&token=EWgmmQti5j)](https://codecov.io/gh/spectacles-ci/spectacles/pull/190?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #190      +/-   ##
==========================================
- Coverage   70.68%   70.08%   -0.60%     
==========================================
  Files          10       10              
  Lines         904      916      +12     
==========================================
+ Hits          639      642       +3     
- Misses        265      274       +9     
```


| [Impacted Files](https://codecov.io/gh/spectacles-ci/spectacles/pull/190?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [spectacles/cli.py](https://codecov.io/gh/spectacles-ci/spectacles/pull/190/diff?src=pr&el=tree#diff-c3BlY3RhY2xlcy9jbGkucHk=) | `69.38% <0.00%> (-0.48%)` | :arrow_down: |
| [spectacles/lookml.py](https://codecov.io/gh/spectacles-ci/spectacles/pull/190/diff?src=pr&el=tree#diff-c3BlY3RhY2xlcy9sb29rbWwucHk=) | `71.81% <27.27%> (-3.56%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/spectacles-ci/spectacles/pull/190?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/spectacles-ci/spectacles/pull/190?src=pr&el=footer). Last update [66892cc...b046cd6](https://codecov.io/gh/spectacles-ci/spectacles/pull/190?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/spectacles/cli.py b/spectacles/cli.py
index 3bbaa9e..33e702d 100644
--- a/spectacles/cli.py
+++ b/spectacles/cli.py
@@ -560,6 +560,13 @@ def run_sql(
                 dimension=error["metadata"].get("dimension"),
                 lookml_url=error["metadata"].get("lookml_url"),
             )
+        if mode == "batch":
+            logger.info(
+                f"\n\nTo determine the exact dimensions responsible for {'this error' if project.number_of_errors == 1 else 'these errors'}, "
+                f"you can re-run \nSpectacles in single-dimension mode, with `--mode single`.\n\n"
+                "You can also run this original validation with `--mode hybrid` to do this automatically."
+            )
+
         logger.info("")
         raise GenericValidationError
     else:
diff --git a/spectacles/lookml.py b/spectacles/lookml.py
index 0bac8c9..63a967e 100644
--- a/spectacles/lookml.py
+++ b/spectacles/lookml.py
@@ -131,6 +131,19 @@ class Explore(LookMlObject):
     def add_dimension(self, dimension: Dimension):
         self.dimensions.append(dimension)
 
+    @property
+    def number_of_errors(self):
+        if self.errored:
+            if self.error:
+                errors = 1
+            else:
+                errors = len(
+                    [dimension for dimension in self.dimensions if dimension.errored]
+                )
+            return errors
+        else:
+            return 0
+
 
 class Model(LookMlObject):
     def __init__(self, name: str, project_name: str, explores: List[Explore]):
@@ -194,6 +207,12 @@ class Model(LookMlObject):
         ]
         return cls(name, project, explores)
 
+    @property
+    def number_of_errors(self):
+        return sum(
+            [explore.number_of_errors for explore in self.explores if explore.errored]
+        )
+
 
 class Project(LookMlObject):
     def __init__(self, name, models: Sequence[Model]):
@@ -272,3 +291,10 @@ class Project(LookMlObject):
             "tested": tested,
             "errors": errors,
         }
+
+    def __repr__(self):
+        return f"{self.__class__.__name__}(name={self.name}, models={self.models})"
+
+    @property
+    def number_of_errors(self):
+        return sum([model.number_of_errors for model in self.models if model.errored])


Test patch:
diff --git a/tests/test_lookml.py b/tests/test_lookml.py
index 196d91c..5c41859 100644
--- a/tests/test_lookml.py
+++ b/tests/test_lookml.py
@@ -113,3 +113,114 @@ def test_parent_queried_behavior_should_depend_on_its_child(
         assert parent.queried is True
         children.append(b)
         assert parent.queried is True
+
+
+def test_comparison_to_mismatched_type_object_fails(dimension, explore, model, project):
+    assert dimension != 1
+    assert explore != 1
+    assert model != 1
+    assert project != 1
+
+
+def test_explore_number_of_errors_batch_with_errors(dimension, explore, sql_error):
+    explore.dimensions = [dimension]
+    explore.queried = True
+    explore.error = sql_error
+    assert explore.number_of_errors == 1
+
+
+def test_explore_number_of_errors_batch_with_no_errors(dimension, explore, sql_error):
+    explore.dimensions = [dimension]
+    explore.queried = True
+    assert explore.number_of_errors == 0
+
+
+def test_explore_number_of_errors_single_with_errors(dimension, explore, sql_error):
+    dimension.error = sql_error
+    dimension.queried = True
+    explore.dimensions = [dimension, dimension]
+    assert explore.number_of_errors == 2
+
+
+def test_explore_number_of_errors_single_with_no_errors(dimension, explore, sql_error):
+    dimension.queried = True
+    explore.dimensions = [dimension, dimension]
+    assert explore.number_of_errors == 0
+
+
+def test_model_number_of_errors_batch_with_errors(dimension, explore, model, sql_error):
+    explore.dimensions = [dimension]
+    explore.queried = True
+    explore.error = sql_error
+    model.explores = [explore, explore]
+    assert model.number_of_errors == 2
+
+
+def test_model_number_of_errors_batch_with_no_errors(
+    dimension, explore, model, sql_error
+):
+    explore.dimensions = [dimension]
+    explore.queried = True
+    model.explores = [explore, explore]
+    assert model.number_of_errors == 0
+
+
+def test_model_number_of_errors_single_with_errors(
+    dimension, explore, model, sql_error
+):
+    dimension.error = sql_error
+    explore.dimensions = [dimension, dimension]
+    explore.queried = True
+    model.explores = [explore, explore]
+    assert model.number_of_errors == 4
+
+
+def test_model_number_of_errors_single_with_no_errors(
+    dimension, explore, model, sql_error
+):
+    explore.dimensions = [dimension, dimension]
+    explore.queried = True
+    model.explores = [explore, explore]
+    assert model.number_of_errors == 0
+
+
+def test_project_number_of_errors_batch_with_errors(
+    dimension, explore, model, project, sql_error
+):
+    explore.dimensions = [dimension]
+    explore.queried = True
+    explore.error = sql_error
+    model.explores = [explore, explore]
+    project.models = [model]
+    assert project.number_of_errors == 2
+
+
+def test_project_number_of_errors_batch_with_no_errors(
+    dimension, explore, model, project, sql_error
+):
+    explore.dimensions = [dimension]
+    explore.queried = True
+    model.explores = [explore, explore]
+    project.models = [model]
+    assert project.number_of_errors == 0
+
+
+def test_project_number_of_errors_single_with_errors(
+    dimension, explore, model, project, sql_error
+):
+    dimension.error = sql_error
+    explore.dimensions = [dimension, dimension]
+    explore.queried = True
+    model.explores = [explore, explore]
+    project.models = [model, model]
+    assert project.number_of_errors == 8
+
+
+def test_project_number_of_errors_single_with_no_errors(
+    dimension, explore, model, project, sql_error
+):
+    explore.dimensions = [dimension, dimension]
+    explore.queried = True
+    model.explores = [explore, explore]
+    project.models = [model, model]
+    assert project.number_of_errors == 0