Repository: dbt-labs/dbt-core

Issue:
[CT-2533] [Feature] Support sqlparse 0.4.4
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

`sqlparse` <0.4.4 contains a moderate security vulnerability: https://github.com/advisories/GHSA-rrm6-wvj7-cwh2

dbt-core has started to pin to <0.4.4 in https://github.com/dbt-labs/dbt-core/pull/7394 which makes it difficult for packages using dbt to update to the fixed version.

Would it be possible for dbt to support v0.4.4? 

### Describe alternatives you've considered

The answer might just be "no". In which case we will have to wait until a new version of sqlparse addresses the issue. It does not seem like there is anything upstream sqlparse tracking work to resolve https://github.com/dbt-labs/dbt-core/issues/7396. So, if the answer is no, then I would hope the appropriate effort upstream can be made so we have a path forward eventually.

### Who will this benefit?

All users of dbt-core who want to update to a non-vulnerable sqlparse version.

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_


Discussion:
+1 even a moderate security vulnerability can be a problem for some organizations.
Thanks for reaching out @lukehsiao and @dimoschi 🙏 

We might need to wait until a new version of sqlparse addresses the issue. The best next step would be for someone to open a relevant issue upstream with [`sqlparse` ](https://github.com/andialbrecht/sqlparse)so that it has the opportunity to be fixed.

I can take it as an action item to open an issue with `sqlparse` and provide them with a reproduction case that mimics the failure of the [`test_singular_tests_ephemeral`](https://github.com/dbt-labs/dbt-core/blob/810ef7f5560b4bbe8a3afb0dc182a4380d3a3dcd/tests/adapter/dbt/tests/adapter/basic/test_singular_tests_ephemeral.py) test in our CI [here](https://github.com/dbt-labs/dbt-core/actions/runs/4731122949/jobs/8400070522).

### Background for those just tuning in
We introduced a pin of sqlparse<0.4.4 because our functional testing suite was failing for sqlparse==0.4.4, and you can read more here:
https://github.com/dbt-labs/dbt-core/issues/7396#issuecomment-1517612668

Side note: The CI tests actually worked for macOS Windows and it was only failing for Linux (Ubuntu).
Gonna try reproduce the issue and look if I can open a PR at `sqlparse`. Thanks for providing context @dbeatty10 
Another user just reported (#7521) seeing the same issue with `sqlparse==0.4.4` on `Debian GNU/Linux 11 (bullseye)`
@MauroLuzzatto provided some very useful information [here](https://github.com/dbt-labs/dbt-core/issues/7595#issuecomment-1545263773), namely:

> I have also seen the error disappearing after retriggering the GHA run.

So maybe there's some kind of state that is available to subsequent executions that isn't available to the first one in a fresh environment?

This is the context reported by @MauroLuzzatto:

> So far, I have only seen the errors on
> - `python:3.8-slim-bullseye`
> - `Ubuntu: 22.04.2 LTS` 
> 
> And not in any local development environment:
> - `macOS Ventura Version 13.3.1`
> 
> However, that might also be connected to the fact that the dbt and other modules are installed there (CI and deployment) on every run.

We should replace `sqlparse` with a manual solution.
💡 @emmyoop do we have an issue in GitHub to replace `sqlparse` with a manual solution? Or would that issue need to be created?
Closing in favor of #7791 
Reopening per https://github.com/dbt-labs/dbt-core/issues/7791#issuecomment-1609468046
Looks like #7919 has pivoted back to use sqlparse 0.4.4 as well

Resolving patch:
diff --git a/core/dbt/compilation.py b/core/dbt/compilation.py
--- a/core/dbt/compilation.py
+++ b/core/dbt/compilation.py
@@ -181,7 +181,6 @@ def link_graph(self, manifest: Manifest):
             self.add_node(source.unique_id)
         for semantic_model in manifest.semantic_models.values():
             self.add_node(semantic_model.unique_id)
-
         for node in manifest.nodes.values():
             self.link_node(node, manifest)
         for exposure in manifest.exposures.values():
@@ -301,66 +300,6 @@ def add_ephemeral_prefix(self, name: str):
         relation_cls = adapter.Relation
         return relation_cls.add_ephemeral_prefix(name)
 
-    def _inject_ctes_into_sql(self, sql: str, ctes: List[InjectedCTE]) -> str:
-        """
-        `ctes` is a list of InjectedCTEs like:
-
-            [
-                InjectedCTE(
-                    id="cte_id_1",
-                    sql="__dbt__cte__ephemeral as (select * from table)",
-                ),
-                InjectedCTE(
-                    id="cte_id_2",
-                    sql="__dbt__cte__events as (select id, type from events)",
-                ),
-            ]
-
-        Given `sql` like:
-
-          "with internal_cte as (select * from sessions)
-           select * from internal_cte"
-
-        This will spit out:
-
-          "with __dbt__cte__ephemeral as (select * from table),
-                __dbt__cte__events as (select id, type from events),
-                with internal_cte as (select * from sessions)
-           select * from internal_cte"
-
-        (Whitespace enhanced for readability.)
-        """
-        if len(ctes) == 0:
-            return sql
-
-        parsed_stmts = sqlparse.parse(sql)
-        parsed = parsed_stmts[0]
-
-        with_stmt = None
-        for token in parsed.tokens:
-            if token.is_keyword and token.normalized == "WITH":
-                with_stmt = token
-            elif token.is_keyword and token.normalized == "RECURSIVE" and with_stmt is not None:
-                with_stmt = token
-                break
-            elif not token.is_whitespace and with_stmt is not None:
-                break
-
-        if with_stmt is None:
-            # no with stmt, add one, and inject CTEs right at the beginning
-            first_token = parsed.token_first()
-            with_stmt = sqlparse.sql.Token(sqlparse.tokens.Keyword, "with")
-            parsed.insert_before(first_token, with_stmt)
-        else:
-            # stmt exists, add a comma (which will come after injected CTEs)
-            trailing_comma = sqlparse.sql.Token(sqlparse.tokens.Punctuation, ",")
-            parsed.insert_after(with_stmt, trailing_comma)
-
-        token = sqlparse.sql.Token(sqlparse.tokens.Keyword, ", ".join(c.sql for c in ctes))
-        parsed.insert_after(with_stmt, token)
-
-        return str(parsed)
-
     def _recursively_prepend_ctes(
         self,
         model: ManifestSQLNode,
@@ -435,7 +374,7 @@ def _recursively_prepend_ctes(
 
             _add_prepended_cte(prepended_ctes, InjectedCTE(id=cte.id, sql=sql))
 
-        injected_sql = self._inject_ctes_into_sql(
+        injected_sql = inject_ctes_into_sql(
             model.compiled_code,
             prepended_ctes,
         )
@@ -586,3 +525,74 @@ def compile_node(
         if write:
             self._write_node(node)
         return node
+
+
+def inject_ctes_into_sql(sql: str, ctes: List[InjectedCTE]) -> str:
+    """
+    `ctes` is a list of InjectedCTEs like:
+
+        [
+            InjectedCTE(
+                id="cte_id_1",
+                sql="__dbt__cte__ephemeral as (select * from table)",
+            ),
+            InjectedCTE(
+                id="cte_id_2",
+                sql="__dbt__cte__events as (select id, type from events)",
+            ),
+        ]
+
+    Given `sql` like:
+
+      "with internal_cte as (select * from sessions)
+       select * from internal_cte"
+
+    This will spit out:
+
+      "with __dbt__cte__ephemeral as (select * from table),
+            __dbt__cte__events as (select id, type from events),
+            internal_cte as (select * from sessions)
+       select * from internal_cte"
+
+    (Whitespace enhanced for readability.)
+    """
+    if len(ctes) == 0:
+        return sql
+
+    parsed_stmts = sqlparse.parse(sql)
+    parsed = parsed_stmts[0]
+
+    with_stmt = None
+    for token in parsed.tokens:
+        if token.is_keyword and token.normalized == "WITH":
+            with_stmt = token
+        elif token.is_keyword and token.normalized == "RECURSIVE" and with_stmt is not None:
+            with_stmt = token
+            break
+        elif not token.is_whitespace and with_stmt is not None:
+            break
+
+    if with_stmt is None:
+        # no with stmt, add one, and inject CTEs right at the beginning
+        # [original_sql]
+        first_token = parsed.token_first()
+        with_token = sqlparse.sql.Token(sqlparse.tokens.Keyword, "with")
+        parsed.insert_before(first_token, with_token)
+        # [with][original_sql]
+        injected_ctes = ", ".join(c.sql for c in ctes) + " "
+        injected_ctes_token = sqlparse.sql.Token(sqlparse.tokens.Keyword, injected_ctes)
+        parsed.insert_after(with_token, injected_ctes_token)
+        # [with][joined_ctes][original_sql]
+    else:
+        # with stmt exists so we don't need to add one, but we do need to add a comma
+        # between the injected ctes and the original sql
+        # [with][original_sql]
+        injected_ctes = ", ".join(c.sql for c in ctes)
+        injected_ctes_token = sqlparse.sql.Token(sqlparse.tokens.Keyword, injected_ctes)
+        parsed.insert_after(with_stmt, injected_ctes_token)
+        # [with][joined_ctes][original_sql]
+        comma_token = sqlparse.sql.Token(sqlparse.tokens.Punctuation, ", ")
+        parsed.insert_after(injected_ctes_token, comma_token)
+        # [with][joined_ctes][, ][original_sql]
+
+    return str(parsed)
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -68,9 +68,8 @@
         "pathspec>=0.9,<0.12",
         "isodate>=0.6,<0.7",
         # ----
-        # There is a difficult-to-reproduce bug in sqlparse==0.4.4 for ephemeral model compilation
-        # For context: dbt-core#7396 + dbt-core#7515
-        "sqlparse>=0.2.3,<0.4.4",
+        # There was a pin to below 0.4.4 for a while due to a bug in Ubuntu/sqlparse 0.4.4
+        "sqlparse>=0.2.3",
         # ----
         # These are major-version-0 packages also maintained by dbt-labs. Accept patches.
         "dbt-extractor~=0.4.1",


Test patch:
diff --git a/tests/functional/compile/test_compile.py b/tests/functional/compile/test_compile.py
--- a/tests/functional/compile/test_compile.py
+++ b/tests/functional/compile/test_compile.py
@@ -1,10 +1,11 @@
 import json
 import pathlib
 import pytest
+import re
 
 from dbt.cli.main import dbtRunner
 from dbt.exceptions import DbtRuntimeError, TargetNotFoundError
-from dbt.tests.util import run_dbt, run_dbt_and_capture
+from dbt.tests.util import run_dbt, run_dbt_and_capture, read_file
 from tests.functional.compile.fixtures import (
     first_model_sql,
     second_model_sql,
@@ -17,9 +18,13 @@
 )
 
 
-def get_lines(model_name):
-    from dbt.tests.util import read_file
+def norm_whitespace(string):
+    _RE_COMBINE_WHITESPACE = re.compile(r"\s+")
+    string = _RE_COMBINE_WHITESPACE.sub(" ", string).strip()
+    return string
+
 
+def get_lines(model_name):
     f = read_file("target", "compiled", "test", "models", model_name + ".sql")
     return [line for line in f.splitlines() if line]
 
@@ -90,21 +95,22 @@ def test_last_selector(self, project):
     def test_no_selector(self, project):
         run_dbt(["compile"])
 
-        assert get_lines("first_ephemeral_model") == ["select 1 as fun"]
-        assert get_lines("second_ephemeral_model") == [
-            "with __dbt__cte__first_ephemeral_model as (",
-            "select 1 as fun",
-            ")select * from __dbt__cte__first_ephemeral_model",
-        ]
-        assert get_lines("third_ephemeral_model") == [
-            "with __dbt__cte__first_ephemeral_model as (",
-            "select 1 as fun",
-            "),  __dbt__cte__second_ephemeral_model as (",
-            "select * from __dbt__cte__first_ephemeral_model",
-            ")select * from __dbt__cte__second_ephemeral_model",
-            "union all",
-            "select 2 as fun",
-        ]
+        sql = read_file("target", "compiled", "test", "models", "first_ephemeral_model.sql")
+        assert norm_whitespace(sql) == norm_whitespace("select 1 as fun")
+        sql = read_file("target", "compiled", "test", "models", "second_ephemeral_model.sql")
+        expected_sql = """with __dbt__cte__first_ephemeral_model as (
+            select 1 as fun
+            ) select * from __dbt__cte__first_ephemeral_model"""
+        assert norm_whitespace(sql) == norm_whitespace(expected_sql)
+        sql = read_file("target", "compiled", "test", "models", "third_ephemeral_model.sql")
+        expected_sql = """with __dbt__cte__first_ephemeral_model as (
+            select 1 as fun
+            ),  __dbt__cte__second_ephemeral_model as (
+            select * from __dbt__cte__first_ephemeral_model
+            ) select * from __dbt__cte__second_ephemeral_model
+            union all
+            select 2 as fun"""
+        assert norm_whitespace(sql) == norm_whitespace(expected_sql)
 
     def test_with_recursive_cte(self, project):
         run_dbt(["compile"])
@@ -112,7 +118,7 @@ def test_with_recursive_cte(self, project):
         assert get_lines("with_recursive_model") == [
             "with recursive  __dbt__cte__first_ephemeral_model as (",
             "select 1 as fun",
-            "),t(n) as (",
+            "), t(n) as (",
             "    select * from __dbt__cte__first_ephemeral_model",
             "  union all",
             "    select n+1 from t where n < 100",
diff --git a/tests/unit/test_inject_ctes.py b/tests/unit/test_inject_ctes.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_inject_ctes.py
@@ -0,0 +1,197 @@
+from dbt.compilation import inject_ctes_into_sql
+from dbt.contracts.graph.nodes import InjectedCTE
+import re
+
+
+def norm_whitespace(string):
+    _RE_COMBINE_WHITESPACE = re.compile(r"\s+")
+    string = _RE_COMBINE_WHITESPACE.sub(" ", string).strip()
+    return string
+
+
+def test_inject_ctes_simple1():
+    starting_sql = "select * from __dbt__cte__base"
+    ctes = [
+        InjectedCTE(
+            id="model.test.base",
+            sql=" __dbt__cte__base as (\n\n\nselect * from test16873767336887004702_test_ephemeral.seed\n)",
+        )
+    ]
+    expected_sql = """with __dbt__cte__base as (
+        select * from test16873767336887004702_test_ephemeral.seed
+        ) select * from __dbt__cte__base"""
+
+    generated_sql = inject_ctes_into_sql(starting_sql, ctes)
+    assert norm_whitespace(generated_sql) == norm_whitespace(expected_sql)
+
+
+def test_inject_ctes_simple2():
+    starting_sql = "select * from __dbt__cte__ephemeral_level_two"
+    ctes = [
+        InjectedCTE(
+            id="model.test.ephemeral_level_two",
+            sql=' __dbt__cte__ephemeral_level_two as (\n\nselect * from "dbt"."test16873757769710148165_test_ephemeral"."source_table"\n)',
+        )
+    ]
+    expected_sql = """with __dbt__cte__ephemeral_level_two as (
+        select * from "dbt"."test16873757769710148165_test_ephemeral"."source_table"
+        ) select * from __dbt__cte__ephemeral_level_two"""
+
+    generated_sql = inject_ctes_into_sql(starting_sql, ctes)
+    assert norm_whitespace(generated_sql) == norm_whitespace(expected_sql)
+
+
+def test_inject_ctes_multiple_ctes():
+
+    starting_sql = "select * from __dbt__cte__ephemeral"
+    ctes = [
+        InjectedCTE(
+            id="model.test.ephemeral_level_two",
+            sql=' __dbt__cte__ephemeral_level_two as (\n\nselect * from "dbt"."test16873735573223965828_test_ephemeral"."source_table"\n)',
+        ),
+        InjectedCTE(
+            id="model.test.ephemeral",
+            sql=" __dbt__cte__ephemeral as (\n\nselect * from __dbt__cte__ephemeral_level_two\n)",
+        ),
+    ]
+    expected_sql = """with __dbt__cte__ephemeral_level_two as (
+            select * from "dbt"."test16873735573223965828_test_ephemeral"."source_table"
+        ),  __dbt__cte__ephemeral as (
+            select * from __dbt__cte__ephemeral_level_two
+        ) select * from __dbt__cte__ephemeral"""
+
+    generated_sql = inject_ctes_into_sql(starting_sql, ctes)
+    assert norm_whitespace(generated_sql) == norm_whitespace(expected_sql)
+
+
+def test_inject_ctes_multiple_ctes_more_complex():
+    starting_sql = """select * from __dbt__cte__female_only
+        union all
+        select * from "dbt"."test16873757723266827902_test_ephemeral"."double_dependent" where gender = 'Male'"""
+    ctes = [
+        InjectedCTE(
+            id="model.test.base",
+            sql=" __dbt__cte__base as (\n\n\nselect * from test16873757723266827902_test_ephemeral.seed\n)",
+        ),
+        InjectedCTE(
+            id="model.test.base_copy",
+            sql=" __dbt__cte__base_copy as (\n\n\nselect * from __dbt__cte__base\n)",
+        ),
+        InjectedCTE(
+            id="model.test.female_only",
+            sql=" __dbt__cte__female_only as (\n\n\nselect * from __dbt__cte__base_copy where gender = 'Female'\n)",
+        ),
+    ]
+    expected_sql = """with __dbt__cte__base as (
+            select * from test16873757723266827902_test_ephemeral.seed
+        ),  __dbt__cte__base_copy as (
+            select * from __dbt__cte__base
+        ),  __dbt__cte__female_only as (
+            select * from __dbt__cte__base_copy where gender = 'Female'
+        ) select * from __dbt__cte__female_only
+        union all
+        select * from "dbt"."test16873757723266827902_test_ephemeral"."double_dependent" where gender = 'Male'"""
+
+    generated_sql = inject_ctes_into_sql(starting_sql, ctes)
+    assert norm_whitespace(generated_sql) == norm_whitespace(expected_sql)
+
+
+def test_inject_ctes_starting_with1():
+    starting_sql = """
+       with internal_cte as (select * from sessions)
+       select * from internal_cte
+    """
+    ctes = [
+        InjectedCTE(
+            id="cte_id_1",
+            sql="__dbt__cte__ephemeral as (select * from table)",
+        ),
+        InjectedCTE(
+            id="cte_id_2",
+            sql="__dbt__cte__events as (select id, type from events)",
+        ),
+    ]
+    expected_sql = """with __dbt__cte__ephemeral as (select * from table),
+       __dbt__cte__events as (select id, type from events),
+       internal_cte as (select * from sessions)
+       select * from internal_cte"""
+
+    generated_sql = inject_ctes_into_sql(starting_sql, ctes)
+    assert norm_whitespace(generated_sql) == norm_whitespace(expected_sql)
+
+
+def test_inject_ctes_starting_with2():
+    starting_sql = """with my_other_cool_cte as (
+        select id, name from __dbt__cte__ephemeral
+        where id > 1000
+    )
+    select name, id from my_other_cool_cte"""
+    ctes = [
+        InjectedCTE(
+            id="model.singular_tests_ephemeral.ephemeral",
+            sql=' __dbt__cte__ephemeral as (\n\n\nwith my_cool_cte as (\n  select name, id from "dbt"."test16873917221900185954_test_singular_tests_ephemeral"."base"\n)\nselect id, name from my_cool_cte where id is not null\n)',
+        )
+    ]
+    expected_sql = """with  __dbt__cte__ephemeral as (
+        with my_cool_cte as (
+          select name, id from "dbt"."test16873917221900185954_test_singular_tests_ephemeral"."base"
+        )
+        select id, name from my_cool_cte where id is not null
+        ), my_other_cool_cte as (
+            select id, name from __dbt__cte__ephemeral
+            where id > 1000
+        )
+        select name, id from my_other_cool_cte"""
+
+    generated_sql = inject_ctes_into_sql(starting_sql, ctes)
+    assert norm_whitespace(generated_sql) == norm_whitespace(expected_sql)
+
+
+def test_inject_ctes_comment_with():
+    # Test injection with a comment containing "with"
+    starting_sql = """
+        --- This is sql with a comment
+        select * from __dbt__cte__base
+    """
+    ctes = [
+        InjectedCTE(
+            id="model.test.base",
+            sql=" __dbt__cte__base as (\n\n\nselect * from test16873767336887004702_test_ephemeral.seed\n)",
+        )
+    ]
+    expected_sql = """with __dbt__cte__base as (
+        select * from test16873767336887004702_test_ephemeral.seed
+        ) --- This is sql with a comment
+        select * from __dbt__cte__base"""
+
+    generated_sql = inject_ctes_into_sql(starting_sql, ctes)
+    assert norm_whitespace(generated_sql) == norm_whitespace(expected_sql)
+
+
+def test_inject_ctes_with_recursive():
+    # Test injection with "recursive" keyword
+    starting_sql = """
+        with recursive t(n) as (
+            select * from __dbt__cte__first_ephemeral_model
+          union all
+            select n+1 from t where n < 100
+        )
+        select sum(n) from t
+    """
+    ctes = [
+        InjectedCTE(
+            id="model.test.first_ephemeral_model",
+            sql=" __dbt__cte__first_ephemeral_model as (\n\nselect 1 as fun\n)",
+        )
+    ]
+    expected_sql = """with recursive  __dbt__cte__first_ephemeral_model as (
+        select 1 as fun
+        ), t(n) as (
+            select * from __dbt__cte__first_ephemeral_model
+          union all
+            select n+1 from t where n < 100
+        )
+        select sum(n) from t
+    """
+    generated_sql = inject_ctes_into_sql(starting_sql, ctes)
+    assert norm_whitespace(generated_sql) == norm_whitespace(expected_sql)