Repository: great-expectations/great_expectations

Issue:
Spark Context has stopped on Dataricks
Hi there, 

**Bug Description**
I am attempting to follow the example as shown in the documentation [here](
https://docs.greatexpectations.io/en/latest/guides/how_to_guides/configuring_data_contexts/how_to_instantiate_a_data_context_on_a_databricks_spark_cluster.html) for using great expectations on Databricks. 

However  whenever I try to instantiate the great expectations context, the spark context seems to stop, displaying the following error message:

```
The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached.
```

One of my colleagues also had an example of great expectations that was working on Databricks, until today, when he also got the `spark context has stopped` error.

**To Reproduce**
Attempt to run the first snippet of [this](
https://docs.greatexpectations.io/en/latest/guides/how_to_guides/configuring_data_contexts/how_to_instantiate_a_data_context_on_a_databricks_spark_cluster.html) in a Databricks notebook cell.

**Expected behavior**
The client is instantiated correctly

**Environment (please complete the following information):**
Databricks run time 7.3 LTS


Discussion:
@sawyee Are you setting spark.app.name? If yes, and you are running a recent release of Great Expectation, you should see this behavior. The solution is not to set the custom app name when running interactively in a Databricks notebook. Could you please confirm? Thank you!
Hi @eugmandel I haven't manually set the spark.app.name configuration, but the result of 
`spark.conf.get("spark.app.name")` is `Databricks Shell`

Thanks
Hi @eugmandel, I am also experiencing the same issue with the same Databricks runtime. Is there any work around?

EDIT:

After having a look at the code, I think [this](https://github.com/great-expectations/great_expectations/blob/develop/great_expectations/core/util.py#L482) may be root cause. I don't understand why the spark context needs to be stopped but this just breaks Databricks runtimes.
Downgrading to version 0.13.12 seems to work. 90ed2ba8dc003b1c11581a64e280455d4bd08dad seems to introduce the breaking changes.
@sawyee Thank you for replying re custom app name!, @jarandaf Thank you for pointing to that commit! 

Yes, then this does look like a bug - the core team will take it on.
@alexsherstinsky Could you please take a look at this? Thank you!
I am looking at it now.  Thank you to the users for reporting it.
Thank you, @jarandaf -- it appears that our implementation of wanting to honor the developer's desire to specify his/her own custom `spark.app.name` is not working with `Databricks`, where the system value of this property is `Databricks Shell`.  Apparently, once stopped, `SparkContext` is having difficulties being restarted.  I was able to reproduce the issue -- thanks!

Moreover, we conducted further troubleshooting and discovered that accessing `SparkContext` from a local `Jupyter` notebook (in `Jupyter Lab`) also differs in behavior from that as part of standard `Python` on a local machine (e.g., in `pytest`).  Specifically, certain references (such as `sparkSession.SparkContext`) are `NULL` in a notebook cell, while a proper object reference in the standard `Python`.  However, further dereferencing produces the expected results.

We modified the code to sacrifice a little bit of readability in favor of a more consistent behavior between standard `Python` and `Jupyter` notebooks, and issued PR #2574 .  Next week, we will test on `Databricks` and report our findings herein.

Thank you very much for finding and reporting this issue.  We will communicate again next week.  Have a great weekend!
Hello again @jarandaf -- the above mentioned PR has been merged into the `develop` branch.  If you are able to load `great_expectations` from that branch, could you please re-test your notebook in Databricks and let us know whether or not the issue is still present?  Otherwise, we can test it in a couple of days when the next release is published.  Thank you!
Hi @alexsherstinsky, thank you for the PR. I was able to try the latest develop changes (414e45ba9980fa033abf119e9c38421a807561c1) and It seems the spark context keeps getting stopped though.

Resolving patch:
diff --git a/great_expectations/core/util.py b/great_expectations/core/util.py
--- a/great_expectations/core/util.py
+++ b/great_expectations/core/util.py
@@ -2,6 +2,7 @@
 import datetime
 import decimal
 import logging
+import os
 import sys
 from collections import OrderedDict
 from collections.abc import Mapping
@@ -99,6 +100,16 @@ def in_jupyter_notebook():
         return False  # Probably standard Python interpreter
 
 
+def in_databricks() -> bool:
+    """
+    Tests whether we are in a Databricks environment.
+
+    Returns:
+        bool
+    """
+    return "DATABRICKS_RUNTIME_VERSION" in os.environ
+
+
 def convert_to_json_serializable(data):
     """
     Helper function to convert an object to one that is json serializable
@@ -548,6 +559,11 @@ def get_or_create_spark_session(
 def spark_restart_required(
     current_spark_config: List[tuple], desired_spark_config: dict
 ) -> bool:
+
+    # we can't change spark context config values within databricks runtimes
+    if in_databricks():
+        return False
+
     current_spark_config_dict: dict = {k: v for (k, v) in current_spark_config}
     if desired_spark_config.get("spark.app.name") != current_spark_config_dict.get(
         "spark.app.name"


Test patch:
diff --git a/tests/integration/spark/test_spark_config.py b/tests/integration/spark/test_spark_config.py
--- a/tests/integration/spark/test_spark_config.py
+++ b/tests/integration/spark/test_spark_config.py
@@ -1,6 +1,10 @@
 import logging
+import os
 from typing import Dict, List
 
+import pytest
+from packaging.version import parse as parse_version
+
 logger = logging.getLogger(__name__)
 
 try:
@@ -43,7 +47,19 @@ def test_spark_config_datasource(spark_session_v012):
     assert ("spark.executor.memory", "768m") in conf
 
 
-def test_spark_config_execution_engine(spark_session):
+@pytest.mark.parametrize("databricks_runtime", [False, True])
+def test_spark_config_execution_engine(spark_session, databricks_runtime):
+    # we observe different behaviour depending on Spark version:
+    # https://spark.apache.org/docs/3.0.0/pyspark-migration-guide.html#upgrading-from-pyspark-24-to-30
+    pyspark_version = parse_version(pyspark.__version__)
+
+    # keep track of spark app id
+    old_app_id = spark_session.sparkContext.applicationId
+
+    if databricks_runtime:
+        # simulate a databricks runtime environment by setting the databricks runtime version
+        os.environ["DATABRICKS_RUNTIME_VERSION"] = "7.3"
+
     name: str = "great_expectations-ee-config"
     spark_config: Dict[str, str] = {
         "spark.app.name": name,
@@ -59,8 +75,28 @@ def test_spark_config_execution_engine(spark_session):
     sc_stopped: bool = spark_session.sparkContext._jsc.sc().isStopped()
     assert not sc_stopped
 
-    # Test that our values were set
+    # Test that our values were set if not running in a Databricks runtime
     conf: List[tuple] = execution_engine.spark.sparkContext.getConf().getAll()
-    assert ("spark.app.name", "great_expectations-ee-config") in conf
-    assert ("spark.sql.catalogImplementation", "hive") in conf
-    assert ("spark.executor.memory", "512m") in conf
+    if not databricks_runtime:
+        # we can safely "restart" the SparkContext
+        if pyspark_version.major >= 3:
+            # spark context is restarted because the "spark.app.name" is not the one desired
+            # this happens because from Spark 3, the SparkSession builder won't update config values
+            assert old_app_id != execution_engine.spark.sparkContext.applicationId
+        assert ("spark.app.name", "great_expectations-ee-config") in conf
+        assert ("spark.sql.catalogImplementation", "hive") in conf
+        assert ("spark.executor.memory", "512m") in conf
+    else:
+        # we should not "restart" the SparkContext
+        if pyspark_version.major >= 3:
+            assert old_app_id == execution_engine.spark.sparkContext.applicationId
+            # spark context config values are not changed
+            assert (
+                "spark.app.name",
+                "default_great_expectations_spark_application",
+            ) in conf
+        else:
+            # spark context config values are changed by the builder no matter what
+            assert ("spark.app.name", "great_expectations-ee-config") in conf
+            assert ("spark.sql.catalogImplementation", "hive") in conf
+            assert ("spark.executor.memory", "512m") in conf