Project: KICS (Provisioning / Security & Compliance)
File: 375cdab9-3f94-4ae0-b1e3-8fbdf9cdf4d7.md

title: Job's Task is Legacy (sparksubmittask) hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 375cdab9-3f94-4ae0-b1e3-8fbdf9cdf4d7 Query name: Job's Task is Legacy (sparksubmittask) Platform: Terraform Severity: <span style="color:#ff7213">Medium</span> Category: Best Practices URL: Job's Task Is sparksubmittask<br> ```tf title="Positive test num. 1 - tf file" hl_lines="36" resource "databricks_job" "positive" { name = "Job with multiple tasks" job_cluster { jobclusterkey = "j" new_cluster { num_workers = 2 sparkversion = data.databricksspark_version.latest.id nodetypeid = data.databricksnodetype.smallest.id } } task { task_key = "a" new_cluster { num_workers = 1 sparkversion = data.databricksspark_version.latest.id nodetypeid = data.databricksnodetype.smallest.id } notebook_task { notebookpath = databricksnotebook.this.path } } task { task_key = "b" //this task will only run after task a depends_on { task_key = "a" } existingclusterid = databricks_cluster.shared.id sparksubmittask { mainclassname = "com.acme.data.Main" } } task { task_key = "c" jobclusterkey = "j" notebook_task { notebookpath = databricksnotebook.this.path } } //this task starts a Delta Live Tables pipline update task { task_key = "d" pipeline_task { pipelineid = databrickspipeline.this.id } } } ``` ```tf title="Positive test num. 2 - tf file" hl_lines="18" resource "databricks_job" "positive" { name = "Job with multiple tasks" job_cluster { jobclusterkey = "j" new_cluster { num_workers = 2 sparkversion = data.databricksspark_version.latest.id nodetypeid = data.databricksnodetype.smallest.id } } task { task_key = "a" existingclusterid = databricks_cluster.shared.id sparksubmittask { mainclassname = "com.acme.data.Main" } } } ``` ```tf title="Negative test num. 1 - tf file" resource "databricks_job" "negative1" { name = "Job with multiple tasks" job_cluster { jobclusterkey = "j" new_cluster { num_workers = 2 sparkversion = data.databricksspark_version.latest.id nodetypeid = data.databricksnodetype.smallest.id } } task { task_key = "a" new_cluster { num_workers = 1 sparkversion = data.databricksspark_version.latest.id nodetypeid = data.databricksnodetype.smallest.id } notebook_task { notebookpath = databricksnotebook.this.path } } task { task_key = "b" //this task will only run after task a depends_on { task_key = "a" } existingclusterid = databricks_cluster.shared.id sparkjartask { mainclassname = "com.acme.data.Main" } } task { task_key = "c" jobclusterkey = "j" notebook_task { notebookpath = databricksnotebook.this.path } } //this task starts a Delta Live Tables pipline update task { task_key = "d" pipeline_task { pipelineid = databrickspipeline.this.id } } } ``` ```tf title="Negative test num. 2 - tf file" resource "databricks_job" "negative1" { name = "Job with multiple tasks" job_cluster { jobclusterkey = "j" new_cluster { num_workers = 2 sparkversion = data.databricksspark_version.latest.id nodetypeid = data.databricksnodetype.smallest.id } } task { task_key = "a" new_cluster { num_workers = 1 sparkversion = data.databricksspark_version.latest.id nodetypeid = data.databricksnodetype.smallest.id } notebook_task { notebookpath = databricksnotebook.this.path } } } ```