Repository: PrefectHQ/prefect

Issue:
DatabricksSubmitRun incompatible with PrefectSecret
## Description

Documentation on the databricks tasks (https://docs.prefect.io/api/latest/tasks/databricks.html) describes a calling convention that does not work.

 - The relevant keyword argument is `databricks_conn_secret`, not `databricks_conn_string`.
 - `PrefectSecret` is a task, so it cannot be passed to the instantiator, as is done in the code examples. To avoid triggering a `UserWarning`, it must be passed to the call on the instantiated task.
 - Even if these two modifications are made, the task will fail with: `TypeError: argument of type 'NoneType' is not iterable`.

## Expected Behavior
I expected the example code on the documentation page to run without errors (at least up to the point where it hits the databricks API).

## Reproduction

The below is adapted from the documentation. It fails on an unexpected keyword argument:

```python
import prefect
from prefect.tasks.databricks.databricks_submitjob import DatabricksSubmitRun
from prefect.tasks.secrets.base import PrefectSecret

json = {
    "new_cluster": {"spark_version": "2.1.0-db3-scala2.11", "num_workers": 2},
    "notebook_task": {
        "notebook_path": "/Users/prefect@example.com/PrepareData",
    },
}

conn = PrefectSecret("DATABRICKS_CONNECTION_STRING")
notebook_run = DatabricksSubmitRun(databricks_conn_string=conn, json=json)

with prefect.Flow("prefect-mwe") as flow:
    notebook_run()
flow.run()
```

Below is the modified code, which still fails with `TypeError`:

```python
import prefect
from prefect.tasks.databricks.databricks_submitjob import DatabricksSubmitRun
from prefect.tasks.secrets.base import PrefectSecret

json = {
    "new_cluster": {"spark_version": "2.1.0-db3-scala2.11", "num_workers": 2},
    "notebook_task": {
        "notebook_path": "/Users/prefect@example.com/PrepareData",
    },
}

conn = PrefectSecret("DATABRICKS_CONNECTION_STRING")
notebook_run = DatabricksSubmitRun(json=json)

with prefect.Flow("prefect-mwe") as flow:
    notebook_run(databricks_conn_secret=conn)
flow.run()
```



## Environment

The environment variable `PREFECT__CONTEXT__SECRETS__DATABRICKS_CONNECTION_STRING` must contain a JSON representation of the dictionary expected by `databricks_conn_secret`.



Discussion:

Resolving patch:
diff --git a/src/prefect/tasks/databricks/databricks_submitjob.py b/src/prefect/tasks/databricks/databricks_submitjob.py
--- a/src/prefect/tasks/databricks/databricks_submitjob.py
+++ b/src/prefect/tasks/databricks/databricks_submitjob.py
@@ -99,7 +99,8 @@ class DatabricksSubmitRun(Task):
     }
 
     conn = PrefectSecret('DATABRICKS_CONNECTION_STRING')
-    notebook_run = DatabricksSubmitRun(databricks_conn_string=conn, json=json)
+    notebook_run = DatabricksSubmitRun(json=json)
+    notebook_run(databricks_conn_secret=conn)
     ```
 
     Another way to accomplish the same thing is to use the named parameters
@@ -118,9 +119,9 @@ class DatabricksSubmitRun(Task):
 
     conn = PrefectSecret('DATABRICKS_CONNECTION_STRING')
     notebook_run = DatabricksSubmitRun(
-        databricks_conn_string=conn,
         new_cluster=new_cluster,
         notebook_task=notebook_task)
+    notebook_run(databricks_conn_secret=conn)
     ```
 
     In the case where both the json parameter **AND** the named parameters
@@ -136,7 +137,8 @@ class DatabricksSubmitRun(Task):
 
     with Flow('my flow') as flow:
         conn = PrefectSecret('DATABRICKS_CONNECTION_STRING')
-        DatabricksSubmitRun(databricks_conn_string=conn, json=...)
+        notebook_run = DatabricksSubmitRun(json=...)
+        notebook_run(databricks_conn_secret=conn)
     ```
 
     Currently the named parameters that `DatabricksSubmitRun` task supports are
@@ -340,6 +342,10 @@ def run(
         assert isinstance(
             databricks_conn_secret, dict
         ), "`databricks_conn_secret` must be supplied as a valid dictionary."
+        self.databricks_conn_secret = databricks_conn_secret
+
+        if json:
+            self.json = json
 
         # Initialize Databricks Connections
         hook = self.get_hook()
@@ -395,7 +401,8 @@ class DatabricksRunNow(Task):
         }
 
     conn = PrefectSecret('DATABRICKS_CONNECTION_STRING')
-    notebook_run = DatabricksRunNow(databricks_conn_string=conn, json=json)
+    notebook_run = DatabricksRunNow(json=json)
+    notebook_run(databricks_conn_secret=conn)
     ```
 
     Another way to accomplish the same thing is to use the named parameters
@@ -417,11 +424,11 @@ class DatabricksRunNow(Task):
 
     conn = PrefectSecret('DATABRICKS_CONNECTION_STRING')
     notebook_run = DatabricksRunNow(
-        databricks_conn_string=conn,
         notebook_params=notebook_params,
         python_params=python_params,
         spark_submit_params=spark_submit_params
     )
+    notebook_run(databricks_conn_secret=conn)
     ```
 
     In the case where both the json parameter **AND** the named parameters
@@ -437,7 +444,8 @@ class DatabricksRunNow(Task):
 
     with Flow('my flow') as flow:
         conn = PrefectSecret('DATABRICKS_CONNECTION_STRING')
-        DatabricksRunNow(databricks_conn_string=conn, json=...)
+        notebook_run = DatabricksRunNow(json=...)
+        notebook_run(databricks_conn_secret=conn)
     ```
 
     Currently the named parameters that `DatabricksRunNow` task supports are
@@ -631,6 +639,10 @@ def run(
         assert isinstance(
             databricks_conn_secret, dict
         ), "`databricks_conn_secret` must be supplied as a valid dictionary."
+        self.databricks_conn_secret = databricks_conn_secret
+
+        if json:
+            self.json = json
 
         # Initialize Databricks Connections
         hook = self.get_hook()


Test patch: