Repository: great-expectations/great_expectations

Issue:
spark_context yaml key does not pass configuration to spark
**Describe the bug**
The [spark_context](https://docs.greatexpectations.io/en/latest/guides/how_to_guides/configuring_datasources/how_to_configure_a_self_managed_spark_datasource.html?highlight=spark_context#additional-notes) option for a spark datasource in the .yml config file does not work. Configuration options passed through it will not take effect on the resulting SparkContext object. 

This is problematic for a number of spark usecases, eg: if you want to read files from S3 you need to have spark install hadoop-aws libraries. Typically this is done through `SparkSession.builder.config`, which is what `spark_context` should be passing its arguments to.

**To Reproduce**
Steps to reproduce the behavior:
1. Create a spark datasource with non-default `spark_context` arguments eg:
```
  spark_dataframe:
    data_asset_type:
      class_name: SparkDFDataset
      module_name: great_expectations.dataset
    class_name: SparkDFDatasource
    module_name: great_expectations.datasource
    spark_config:
      spark.jars.packages: 'org.apache.hadoop:hadoop-aws:2.7.3'
      spark.hadoop.fs.s3a.impl: 'org.apache.hadoop.fs.s3a.S3AFileSystem'
      spark.hadoop.fs.s3.impl: 'org.apache.hadoop.fs.s3a.S3AFileSystem'
```
2. Start a great expectations action in the CLI that uses the spark datasource eg: `great_expectations suite new`
3. Select the spark datasource you configured and hit enter. This will start the spark server locally.
4. Visit the spark UI (typically at localhost:4040) and go to the `Environment` tab
5. Search for spark.jars.packages, it will not be found.

**Expected behavior**
spark.jars.packages to exist in the spark.properties section in the environment tab and be populated with the argument  `org.apache.hadoop:hadoop-aws:2.7.3`

**Environment (please complete the following information):**
 - OS: Ubuntu 18.04
 - GE Version: 0.12.9
 - Spark Version 2.4.5




Discussion:

Resolving patch:
diff --git a/great_expectations/data_context/types/base.py b/great_expectations/data_context/types/base.py
--- a/great_expectations/data_context/types/base.py
+++ b/great_expectations/data_context/types/base.py
@@ -304,7 +304,7 @@ class Meta:
         keys=fields.Str(), values=fields.Dict(), required=False, allow_none=True
     )
     credentials = fields.Raw(required=False, allow_none=True)
-    spark_context = fields.Raw(required=False, allow_none=True)
+    spark_config = fields.Raw(required=False, allow_none=True)
 
     @validates_schema
     def validate_schema(self, data, **kwargs):


Test patch:
diff --git a/tests/datasource/test_new_datasource.py b/tests/datasource/test_new_datasource.py
--- a/tests/datasource/test_new_datasource.py
+++ b/tests/datasource/test_new_datasource.py
@@ -88,13 +88,12 @@ def basic_spark_datasource(tmp_path_factory):
 execution_engine:
     class_name: SparkDFExecutionEngine
     spark_config:
-        spark_context:
-            spark.master: local[*]
-            spark.executor.memory: 6g
-            spark.driver.memory: 6g
-            spark.ui.showConsoleProgress: false
-            spark.sql.shuffle.partitions: 2
-            spark.default.parallelism: 4
+        spark.master: local[*]
+        spark.executor.memory: 6g
+        spark.driver.memory: 6g
+        spark.ui.showConsoleProgress: false
+        spark.sql.shuffle.partitions: 2
+        spark.default.parallelism: 4
 data_connectors:
     test_runtime_data_connector:
         module_name: great_expectations.datasource.data_connector
@@ -233,14 +232,12 @@ def test_basic_spark_datasource_self_check(basic_spark_datasource):
             "class_name": "SparkDFExecutionEngine",
             "persist": True,
             "spark_config": {
-                "spark_context": {
-                    "spark.master": "local[*]",
-                    "spark.executor.memory": "6g",
-                    "spark.driver.memory": "6g",
-                    "spark.ui.showConsoleProgress": False,
-                    "spark.sql.shuffle.partitions": 2,
-                    "spark.default.parallelism": 4,
-                }
+                "spark.master": "local[*]",
+                "spark.executor.memory": "6g",
+                "spark.driver.memory": "6g",
+                "spark.ui.showConsoleProgress": False,
+                "spark.sql.shuffle.partitions": 2,
+                "spark.default.parallelism": 4,
             },
         },
         "data_connectors": {