Repository: dbt-labs/dbt-core

Issue:
[CT-2411] [Bug] Ephemeral materialization not working properly with recursive CTE
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

When performing the expansion of the referenced ephemeral models in the final query, the generated CTE are misplaced if the query starts with a `WITH RECURSIVE` statement.

### Expected Behavior

The CTE for the ephemeral models should be inserted after the `WITH RECURSIVE` statement.

### Steps To Reproduce

Let's say we have `model_1` using ephemeral materialization:
```sql
{{ config(materialized="ephemeral") }}

SELECT
  "Hello" AS greetings
```

And `model_2` using a `WITH RECURSIVE` statement and referencing `model_1`:
```sql
{{ config(materialized="table") }}

WITH RECURSIVE

  # Not actually recursive CTE.
  target AS (
    SELECT
      "world" AS target
  )

SELECT
  CONCAT(m1.greetings, ", ", t.target, " !!") AS col
FROM
  {{ ref("model_1") }} AS m1
CROSS JOIN
  target AS t

```
The generated SQL is not valid as the `model_1` CTE is inserted between the `WITH` and the `RECURSIVE` keywords:
```sql
   create or replace table `XXXX`.`XXXX`.`model_2`  as (

WITH  __dbt__cte__model_1 as (
SELECT
  "Hello" AS greetings
),RECURSIVE

  # Not actually recursive CTE.
  target AS (
    SELECT
      "world" AS target
  )

SELECT
  CONCAT(m1.greetings, ", ", t.target, " !!") AS col
FROM
  __dbt__cte__model_1 AS m1
CROSS JOIN
  target AS t
    );
```

After looking at the source code, it seems the issue comes from [this](https://github.com/dbt-labs/dbt-core/blob/70c26f5c74c340283a12f3a9270ec4c0f7135da5/core/dbt/compilation.py#L233) code that only looks for a `WITH` keyword for determining at which position to insert the CTE generated for the ephemeral models.

### Relevant log output

_No response_

### Environment

```markdown
- OS: Linux Mint 20.3
- Python: 3.11.2
- dbt: 1.4.5
```


### Which database adapter are you using with dbt?

bigquery

### Additional Context

_No response_


Discussion:
Thank you for reporting this @pcasteran !

Thanks also for your simple reprex; I was able to reproduce the behavior you reported (after updating any BigQuery-specific syntax).

Would you be interested in contributing a PR to fix this by any chance?

If so, [this](https://www.postgresql.org/docs/current/queries-with.html#:~:text=follow%20this%20way.-,7.8.2.%C2%A0Recursive%20Queries,-The%20optional%20RECURSIVE) could be a nice example to include in the testing.
For anyone wondering the value of recursive CTE, here are a couple resources:
- https://duckdb.org/docs/sql/query_syntax/with.html
- https://motherduck.com/blog/solving-advent-code-duckdb-dbt/

Resolving patch:
diff --git a/core/dbt/compilation.py b/core/dbt/compilation.py
--- a/core/dbt/compilation.py
+++ b/core/dbt/compilation.py
@@ -232,6 +232,10 @@ def _inject_ctes_into_sql(self, sql: str, ctes: List[InjectedCTE]) -> str:
         for token in parsed.tokens:
             if token.is_keyword and token.normalized == "WITH":
                 with_stmt = token
+            elif token.is_keyword and token.normalized == "RECURSIVE" and with_stmt is not None:
+                with_stmt = token
+                break
+            elif not token.is_whitespace and with_stmt is not None:
                 break
 
         if with_stmt is None:


Test patch:
diff --git a/tests/functional/compile/fixtures.py b/tests/functional/compile/fixtures.py
--- a/tests/functional/compile/fixtures.py
+++ b/tests/functional/compile/fixtures.py
@@ -32,6 +32,16 @@
 }} as fun
 """
 
+with_recursive_model_sql = """
+{{ config(materialized = 'ephemeral') }}
+with recursive t(n) as (
+    select * from {{ ref('first_ephemeral_model') }}
+  union all
+    select n+1 from t where n < 100
+)
+select sum(n) from t;
+"""
+
 schema_yml = """
 version: 2
 
diff --git a/tests/functional/compile/test_compile.py b/tests/functional/compile/test_compile.py
--- a/tests/functional/compile/test_compile.py
+++ b/tests/functional/compile/test_compile.py
@@ -9,6 +9,7 @@
     first_ephemeral_model_sql,
     second_ephemeral_model_sql,
     third_ephemeral_model_sql,
+    with_recursive_model_sql,
     schema_yml,
     model_multiline_jinja,
 )
@@ -54,6 +55,7 @@ def models(self):
             "first_ephemeral_model.sql": first_ephemeral_model_sql,
             "second_ephemeral_model.sql": second_ephemeral_model_sql,
             "third_ephemeral_model.sql": third_ephemeral_model_sql,
+            "with_recursive_model.sql": with_recursive_model_sql,
         }
 
     def test_first_selector(self, project):
@@ -102,6 +104,20 @@ def test_no_selector(self, project):
             "select 2 as fun",
         ]
 
+    def test_with_recursive_cte(self, project):
+        run_dbt(["compile"])
+
+        assert get_lines("with_recursive_model") == [
+            "with recursive  __dbt__cte__first_ephemeral_model as (",
+            "select 1 as fun",
+            "),t(n) as (",
+            "    select * from __dbt__cte__first_ephemeral_model",
+            "  union all",
+            "    select n+1 from t where n < 100",
+            ")",
+            "select sum(n) from t;",
+        ]
+
 
 class TestCompile:
     @pytest.fixture(scope="class")