Repository: apache/airflow

Issue:
Spark JDBC Hook fails if spark_conf is not specified
**Apache Airflow version**: 1.10.10

**What happened**:

At SparkJDBCHook, the `spark_conf` parameter has default None, if kept like that it raise an error:
```
Traceback (most recent call last):
  File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/contrib/operators/spark_jdbc_operator.py", line 211, in execute
    self._hook.submit_jdbc_job()
  File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/contrib/hooks/spark_jdbc_hook.py", line 243, in submit_jdbc_job
    "/spark_jdbc_script.py")
  File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 383, in submit
    spark_submit_cmd = self._build_spark_submit_command(application)
  File "/Users/rbottega/Documents/airflow_latest/env/lib/python3.7/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 254, in _build_spark_submit_command
    for key in self._conf:
TypeError: 'NoneType' object is not iterable
```

**What you expected to happen**:

Following the same behaviour than SparkSubmitHook, the a`spark_conf` should have default empty dict "{}"
```
self._conf = conf or {}
```

**How to reproduce it**:
Create a DAG with SparkJDBCOperator and don't specify the parameter `spark_conf`
```
    spark_to_jdbc_job = SparkJDBCOperator(
        cmd_type='spark_to_jdbc',
        jdbc_table="foo",
        spark_jars="${SPARK_HOME}/jars/postgresql-42.2.12.jar",
        jdbc_driver="org.postgresql.Driver",
        metastore_table="bar",
        save_mode="append",
        task_id="spark_to_jdbc_job"
    )
```


**Anything else we need to know**:

I am happy to implement this change.


Discussion:

Resolving patch:
diff --git a/airflow/providers/apache/spark/example_dags/example_spark_dag.py b/airflow/providers/apache/spark/example_dags/example_spark_dag.py
--- a/airflow/providers/apache/spark/example_dags/example_spark_dag.py
+++ b/airflow/providers/apache/spark/example_dags/example_spark_dag.py
@@ -48,7 +48,6 @@
     jdbc_to_spark_job = SparkJDBCOperator(
         cmd_type='jdbc_to_spark',
         jdbc_table="foo",
-        spark_conf={},
         spark_jars="${SPARK_HOME}/jars/postgresql-42.2.12.jar",
         jdbc_driver="org.postgresql.Driver",
         metastore_table="bar",
@@ -60,7 +59,6 @@
     spark_to_jdbc_job = SparkJDBCOperator(
         cmd_type='spark_to_jdbc',
         jdbc_table="foo",
-        spark_conf={},
         spark_jars="${SPARK_HOME}/jars/postgresql-42.2.12.jar",
         jdbc_driver="org.postgresql.Driver",
         metastore_table="bar",
diff --git a/airflow/providers/apache/spark/hooks/spark_jdbc.py b/airflow/providers/apache/spark/hooks/spark_jdbc.py
--- a/airflow/providers/apache/spark/hooks/spark_jdbc.py
+++ b/airflow/providers/apache/spark/hooks/spark_jdbc.py
@@ -147,7 +147,7 @@ def __init__(self,
         super().__init__(*args, **kwargs)
         self._name = spark_app_name
         self._conn_id = spark_conn_id
-        self._conf = spark_conf
+        self._conf = spark_conf or {}
         self._py_files = spark_py_files
         self._files = spark_files
         self._jars = spark_jars


Test patch: