Repository: databricks/dbt-databricks
Pull request #856 (2024-11-25 22:41:21)

Try setting queue enabled for Python notebook jobs
<!---
  Include the number of the issue addressed by this PR above if applicable.

  Example:
    resolves #1234

  Please review our pull request review process in CONTRIBUTING.md before your proceed.
-->

### Description

Hearing again from users about multiple python models having concurrency issues.  We definitely want to queue if necessary, but not sure if this is actually the problem.  Setting this shouldn't hurt though.

### Checklist

- [x] I have run this code in development and it appears to resolve the stated issue
- [x] This PR includes tests, or tests are not required/relevant for this PR
- [x] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-databricks next" section.



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e587078d..e12a3b19 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,7 +2,6 @@
 
 ### Features
 
-- Add config for generating unique tmp table names for enabling parralel merge (thanks @huangxingyi-git!) ([854](https://github.com/databricks/dbt-databricks/pull/854))
 - Add support for serverless job clusters on python models ([706](https://github.com/databricks/dbt-databricks/pull/706))
 - Add 'user_folder_for_python' behavior to switch writing python model notebooks to the user's folder ([835](https://github.com/databricks/dbt-databricks/pull/835))
 - Merge capabilities are extended ([739](https://github.com/databricks/dbt-databricks/pull/739)) to include the support for the following features (thanks @mi-volodin):
@@ -22,6 +21,7 @@
 ### Fixes
 
 - Replace array indexing with 'get' in split_part so as not to raise exception when indexing beyond bounds ([839](https://github.com/databricks/dbt-databricks/pull/839))
+- Set queue enabled for Python notebook jobs ([856](https://github.com/databricks/dbt-databricks/pull/856))
 
 ### Under the Hood
 
diff --git a/dbt/adapters/databricks/python_models/python_submissions.py b/dbt/adapters/databricks/python_models/python_submissions.py
index 27e4303c..5f0f7653 100644
--- a/dbt/adapters/databricks/python_models/python_submissions.py
+++ b/dbt/adapters/databricks/python_models/python_submissions.py
@@ -224,6 +224,7 @@ class PythonJobConfigCompiler:
         if access_control_list:
             job_spec["access_control_list"] = access_control_list
 
+        job_spec["queue"] = {"enabled": True}
         return PythonJobDetails(self.run_name, job_spec, additional_job_config)
 
 
diff --git a/dbt/include/databricks/macros/materializations/incremental/incremental.sql b/dbt/include/databricks/macros/materializations/incremental/incremental.sql
index 55c54324..6096be4a 100644
--- a/dbt/include/databricks/macros/materializations/incremental/incremental.sql
+++ b/dbt/include/databricks/macros/materializations/incremental/incremental.sql
@@ -19,7 +19,7 @@
   {%- set on_schema_change = incremental_validate_on_schema_change(config.get('on_schema_change'), default='ignore') -%}
   {%- set target_relation = this.incorporate(type='table') -%}
   {%- set existing_relation = adapter.get_relation(database=this.database, schema=this.schema, identifier=this.identifier, needs_information=True) -%}
-  {%- if unique_tmp_table_suffix == True and raw_strategy in ['merge', 'replace_where'] and raw_file_format == 'delta' -%}
+  {%- if unique_tmp_table_suffix == True and raw_strategy == 'replace_where' and raw_file_format == 'delta' -%}
     {%- set temp_relation_suffix = adapter.generate_unique_temporary_table_suffix() -%}
   {%- else -%}
     {%- set temp_relation_suffix = '__dbt_tmp' -%}


Test patch:
diff --git a/tests/unit/python/test_python_job_support.py b/tests/unit/python/test_python_job_support.py
index 4eec7355..41f48041 100644
--- a/tests/unit/python/test_python_job_support.py
+++ b/tests/unit/python/test_python_job_support.py
@@ -158,6 +158,7 @@ class TestPythonJobConfigCompiler:
                 "notebook_path": "path",
             },
             "libraries": [],
+            "queue": {"enabled": True},
         }
         assert details.additional_job_config == {}
 
@@ -182,5 +183,6 @@ class TestPythonJobConfigCompiler:
             "cluster_id": "id",
             "libraries": [{"pypi": {"package": "foo"}}],
             "access_control_list": [{"user_name": "user", "permission_level": "IS_OWNER"}],
+            "queue": {"enabled": True},
         }
         assert details.additional_job_config == {"foo": "bar"}