Repository: dbt-labs/dbt-core

Issue:
[CT-3023] [Feature] compile --suppress-ephemeral-ctes
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

When linting with SQLFluff, it compiles the model to SQL (2x) and then linting is executed against the true SQL of the model, instead of the raw Jinja template.

The first compilation compiles the model as-is, and the second compilation is of a synthetic model built out of the Jinja blocks and expressions, and some tracking values in between. The result thereof is used to "trace" which parts of the compiled model come from what expressions, and to detect loops, etc.

When compiling a model with direct references to ephemeral models, the resulting injected CTEs violate some assumptions of the tracing and result in a failure to lint the model.

Proposal: for the `compile` command only, if a new flag `--suppress-ephemeral-ctes` is provided, the CTE injection does not happen and the expression referencing to the ephemeral model is replaced with a raw identifier. 

This resulting SQL is valid syntax, and allows the tracing to work as it is presently implemented.

### Describe alternatives you've considered

I attempted to strip out the resulting CTEs from both compilations within the Cloud SQLFluff templater, but this was fraught with typical [regex peril](https://xkcd.com/1171/).

When using SQLFluff's CLI templater today, I believe it works around this by altering the manifest nodes for any ephemeral models to appear to be materialized to table instead, during the mid-lint compilation.

### Who will this benefit?

Cloud IDE usage of SQLFluff will immediately benefit from using this flag.

The SQLFluff maintainers may also be able to benefit from this flag in future versions of their CLI templater implementation.

### Are you interested in contributing this feature?

yes! provided a proposed implementation in #8482 

### Anything else?

_No response_


Discussion:

Resolving patch:
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -330,6 +330,7 @@ def docs_serve(ctx, **kwargs):
 @p.state
 @p.defer_state
 @p.deprecated_state
+@p.compile_inject_ephemeral_ctes
 @p.target
 @p.target_path
 @p.threads
diff --git a/core/dbt/cli/params.py b/core/dbt/cli/params.py
--- a/core/dbt/cli/params.py
+++ b/core/dbt/cli/params.py
@@ -40,6 +40,14 @@
     default=True,
 )
 
+compile_inject_ephemeral_ctes = click.option(
+    "--inject-ephemeral-ctes/--no-inject-ephemeral-ctes",
+    envvar=None,
+    help="Internal flag controlling injection of referenced ephemeral models' CTEs during `compile`.",
+    hidden=True,
+    default=True,
+)
+
 config_dir = click.option(
     "--config-dir",
     envvar=None,
diff --git a/core/dbt/compilation.py b/core/dbt/compilation.py
--- a/core/dbt/compilation.py
+++ b/core/dbt/compilation.py
@@ -320,6 +320,10 @@ def _recursively_prepend_ctes(
         if model.compiled_code is None:
             raise DbtRuntimeError("Cannot inject ctes into an uncompiled node", model)
 
+        # tech debt: safe flag/arg access (#6259)
+        if not getattr(self.config.args, "inject_ephemeral_ctes", True):
+            return (model, [])
+
         # extra_ctes_injected flag says that we've already recursively injected the ctes
         if model.extra_ctes_injected:
             return (model, model.extra_ctes)


Test patch:
diff --git a/tests/functional/materializations/test_ephemeral_compilation.py b/tests/functional/materializations/test_ephemeral_compilation.py
--- a/tests/functional/materializations/test_ephemeral_compilation.py
+++ b/tests/functional/materializations/test_ephemeral_compilation.py
@@ -1,3 +1,5 @@
+from dbt.contracts.graph.nodes import ModelNode
+from dbt.contracts.results import RunExecutionResult, RunResult
 import pytest
 from dbt.tests.util import run_dbt
 
@@ -53,6 +55,16 @@
 """
 
 
+SUPPRESSED_CTE_EXPECTED_OUTPUT = """-- fct_eph_first.sql
+
+
+with int_eph_first as(
+    select * from __dbt__cte__int_eph_first
+)
+
+select * from int_eph_first"""
+
+
 class TestEphemeralCompilation:
     @pytest.fixture(scope="class")
     def models(self):
@@ -67,5 +79,13 @@ def test_ephemeral_compilation(self, project):
         results = run_dbt(["run"])
         assert len(results) == 0
 
-        results = run_dbt(["test"])
-        len(results) == 4
+    def test__suppress_injected_ctes(self, project):
+        compile_output = run_dbt(
+            ["compile", "--no-inject-ephemeral-ctes", "--select", "fct_eph_first"]
+        )
+        assert isinstance(compile_output, RunExecutionResult)
+        node_result = compile_output.results[0]
+        assert isinstance(node_result, RunResult)
+        node = node_result.node
+        assert isinstance(node, ModelNode)
+        assert node.compiled_code == SUPPRESSED_CTE_EXPECTED_OUTPUT