Repository: dbt-labs/dbt-core

Issue:
[CT-2057] [Regression] Flaky generic tests on multiple threads in dbt 1.4.1
### Is this a regression in a recent version of dbt-core?

- [X] I believe this is a regression in dbt-core functionality
- [X] I have searched the existing issues, and I could not find an existing issue for this regression

### Current Behavior

Hello,
Our project faced failures in CI test jobs after upgrading to the latest DBT version (1.4.1). The failures were due to flaky generic tests.  After investigating the issue I was able to end up having 2 generic tests running on a single model, which causes the failure.

We are still running the older version `1.3.2` on main and everything is working fine and reliably.

The issue appears to be related to a concurrency/race condition problem. This was confirmed when the following was discovered:
1. Setting the number of threads to 1 caused the failing tests to pass.
2. Testing each of the two generic tests individually (by commenting out the other one) showed that they both succeeded.
3. Sometimes the mentioned tests succeed altogether, but it's rare.

The issues is related to:
https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/compilation.py#L243
```python
token = sqlparse.sql.Token(sqlparse.tokens.Keyword, ", ".join(c.sql for c in ctes))
parsed.insert_after(with_stmt, token)
```

Some of the CTEs don't have their `sql` attribute set:
```python
[
    InjectedCTE(id='model.onb.int_loan_monitor__investment_derived_kpis_actuals', sql=None),
    InjectedCTE(id='model.onb.int_loan_monitor__investment_derived_kpis_scenarios', sql=None),
    ...
]
```

I'm not familiar with the algorithm how the entities of `InjectedCTE` are created in the concurrent mode, but it would be good to hear from you if you have any suggestions/workarounds for this error.

### Expected/Previous Behavior

We expect all the tests to succeed in multi-threaded environment with DBT version 1.4.1

### Steps To Reproduce
I haven't tried yet setting a new project and running tests on a clean setup, but based on our findings, we should focus on:

1. Set more than 1 thread in your `profiles.yml`
2. Create at least 2 generics tests for different fields on a single model (in our case `not_null`)
4. Run all generics tests for a single model:
```
pipenv run dbt --debug test --select model_name
```

### Relevant log output

```shell
12:56:39.551807 [debug] [Thread-1 (]: Traceback (most recent call last):
  File ".local/share/virtualenvs/data-platform-s3J5s7em/lib/python3.10/site-packages/dbt/task/base.py", line 367, in safe_run
    result = self.compile_and_execute(manifest, ctx)
  File ".local/share/virtualenvs/data-platform-s3J5s7em/lib/python3.10/site-packages/dbt/task/base.py", line 304, in compile_and_execute
    ctx.node = self.compile(manifest)
  File ".local/share/virtualenvs/data-platform-s3J5s7em/lib/python3.10/site-packages/dbt/task/compile.py", line 37, in compile
    return compiler.compile_node(self.node, manifest, {})
  File ".local/share/virtualenvs/data-platform-s3J5s7em/lib/python3.10/site-packages/dbt/compilation.py", line 521, in compile_node
    node, _ = self._recursively_prepend_ctes(node, manifest, extra_context)
  File ".local/share/virtualenvs/data-platform-s3J5s7em/lib/python3.10/site-packages/dbt/compilation.py", line 319, in _recursively_prepend_ctes
    injected_sql = self._inject_ctes_into_sql(
  File ".local/share/virtualenvs/data-platform-s3J5s7em/lib/python3.10/site-packages/dbt/compilation.py", line 242, in _inject_ctes_into_sql
    token = sqlparse.sql.Token(sqlparse.tokens.Keyword, ", ".join(c.sql for c in ctes))
TypeError: sequence item 0: expected str instance, NoneType found
```


### Environment

```markdown
- OS: macOS 12.6.1 / Ubuntu 22.04.1
- Python: 3.10.8 / 3.10.9
- dbt (working version): 1.3.2
- dbt (regression version): 1.4.1
```


### Which database adapter are you using with dbt?

bigquery

### Additional Context

_No response_


Discussion:
@gastlich Thanks for opening! Are the two generic tests both defined on an ephemeral model? Or a model that depends on an ephemeral model?

The code you linked is related to how ephemeral models are compiled into downstream nodes (as CTEs). We made a few changes to that code in v1.4, which were behind-the-scenes refactoring and cleanup of our node types (https://github.com/dbt-labs/dbt-core/pull/6384/, https://github.com/dbt-labs/dbt-core/pull/6427). While those chnages shouldn't have had any functional effect, it's of course possible that this is an unintended consequence.

A clear & reliable reproduction case would help us get to the bottom of this.

I've tried reproducing with:
```sql
-- models/my_model.sql
{{ config(materialized = 'ephemeral') }}
select 1 as id
```
```yml
# models/config.yml
version: 2
models:
  - name: my_model
    columns:
      - name: id
        tests:
          - unique
          - not_null
```
But I haven't been able to reproduce the error. I'm able to run both tests with multiple threads successfully.
hello @jtcohen6 . Thanks for coming back to me so quickly :)

I've managed to recreate the issue on a clean project:

### Setup
```yaml
# schema.yml
version: 2

models:
  - name: int_eph_first
    columns:
      - name: first_column
        tests:
          - not_null
      - name: second_column
        tests:
          - not_null

  - name: fct_eph_first
    columns:
      - name: first_column
        tests:
          - not_null
      - name: second_column
        tests:
          - not_null
```

```sql
-- int_eph_first.sql
{{ config(materialized='ephemeral') }}

select
    1 as first_column,
    2 as second_column,
```

```sql
-- fct_eph_first.sql
{{ config(materialized='ephemeral') }}

with int_eph_first as(
    select * from {{ ref('int_eph_first') }}
)

select * from int_eph_first
```

<img width="648" alt="Screenshot 2023-02-07 at 20 39 59" src="https://user-images.githubusercontent.com/1146430/217360525-f98a59ca-fd5b-41bd-94bf-39cce68b3adf.png">

### Single-threaded outcome
```
20:40:24  Running with dbt=1.4.1
20:40:24  Found 2 models, 4 tests, 0 snapshots, 0 analyses, 334 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
20:40:24
20:40:24  Concurrency: 1 threads (target='dev')
20:40:24
20:40:24  1 of 4 START test not_null_fct_eph_first_first_column .......................... [RUN]
20:40:25  1 of 4 PASS not_null_fct_eph_first_first_column ................................ [PASS in 1.54s]
20:40:25  2 of 4 START test not_null_fct_eph_first_second_column ......................... [RUN]
20:40:26  2 of 4 PASS not_null_fct_eph_first_second_column ............................... [PASS in 0.82s]
20:40:26  3 of 4 START test not_null_int_eph_first_first_column .......................... [RUN]
20:40:27  3 of 4 PASS not_null_int_eph_first_first_column ................................ [PASS in 0.78s]
20:40:27  4 of 4 START test not_null_int_eph_first_second_column ......................... [RUN]
20:40:28  4 of 4 PASS not_null_int_eph_first_second_column ............................... [PASS in 1.03s]
20:40:28
20:40:28  Finished running 4 tests in 0 hours 0 minutes and 4.18 seconds (4.18s).
20:40:28
20:40:28  Completed successfully
20:40:28
20:40:28  Done. PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4
```

### Multi-threaded outcome
```
20:41:51  Running with dbt=1.4.1
20:41:51  Unable to do partial parsing because profile has changed
20:41:52  Found 2 models, 4 tests, 0 snapshots, 0 analyses, 334 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
20:41:52
20:41:52  Concurrency: 10 threads (target='dev')
20:41:52
20:41:52  1 of 4 START test not_null_fct_eph_first_first_column .......................... [RUN]
20:41:52  2 of 4 START test not_null_fct_eph_first_second_column ......................... [RUN]
20:41:52  3 of 4 START test not_null_int_eph_first_first_column .......................... [RUN]
20:41:52  4 of 4 START test not_null_int_eph_first_second_column ......................... [RUN]
20:41:52  Unhandled error while executing test.jaffle_shop.not_null_fct_eph_first_second_column.19482e284e
sequence item 0: expected str instance, NoneType found
20:41:52  2 of 4 ERROR not_null_fct_eph_first_second_column .............................. [ERROR in 0.03s]
20:41:53  3 of 4 PASS not_null_int_eph_first_first_column ................................ [PASS in 1.30s]
20:41:53  1 of 4 PASS not_null_fct_eph_first_first_column ................................ [PASS in 1.32s]
20:41:53  4 of 4 PASS not_null_int_eph_first_second_column ............................... [PASS in 1.60s]
20:41:53
20:41:53  Finished running 4 tests in 0 hours 0 minutes and 1.61 seconds (1.61s).
20:41:53
20:41:53  Completed with 1 error and 0 warnings:
20:41:53
20:41:53  sequence item 0: expected str instance, NoneType found
20:41:53
20:41:53  Done. PASS=3 WARN=0 ERROR=1 SKIP=0 TOTAL=4
```
```
20:43:33.491653 [debug] [Thread-1 (]: Began running node test.jaffle_shop.not_null_fct_eph_first_first_column.47390b8f26
20:43:33.491759 [debug] [Thread-2 (]: Began running node test.jaffle_shop.not_null_fct_eph_first_second_column.19482e284e
20:43:33.491907 [debug] [Thread-3 (]: Began running node test.jaffle_shop.not_null_int_eph_first_first_column.7fd5ab6772
20:43:33.492013 [debug] [Thread-4 (]: Began running node test.jaffle_shop.not_null_int_eph_first_second_column.f4df3457a3
20:43:33.492243 [info ] [Thread-1 (]: 1 of 4 START test not_null_fct_eph_first_first_column .......................... [RUN]
20:43:33.492427 [info ] [Thread-2 (]: 2 of 4 START test not_null_fct_eph_first_second_column ......................... [RUN]
20:43:33.492551 [info ] [Thread-3 (]: 3 of 4 START test not_null_int_eph_first_first_column .......................... [RUN]
20:43:33.492669 [info ] [Thread-4 (]: 4 of 4 START test not_null_int_eph_first_second_column ......................... [RUN]
20:43:33.492995 [debug] [Thread-1 (]: Acquiring new bigquery connection 'test.jaffle_shop.not_null_fct_eph_first_first_column.47390b8f26'
20:43:33.493316 [debug] [Thread-2 (]: Acquiring new bigquery connection 'test.jaffle_shop.not_null_fct_eph_first_second_column.19482e284e'
20:43:33.493584 [debug] [Thread-3 (]: Acquiring new bigquery connection 'test.jaffle_shop.not_null_int_eph_first_first_column.7fd5ab6772'
20:43:33.493831 [debug] [Thread-4 (]: Acquiring new bigquery connection 'test.jaffle_shop.not_null_int_eph_first_second_column.f4df3457a3'
20:43:33.493952 [debug] [Thread-1 (]: Began compiling node test.jaffle_shop.not_null_fct_eph_first_first_column.47390b8f26
20:43:33.494067 [debug] [Thread-2 (]: Began compiling node test.jaffle_shop.not_null_fct_eph_first_second_column.19482e284e
20:43:33.494175 [debug] [Thread-3 (]: Began compiling node test.jaffle_shop.not_null_int_eph_first_first_column.7fd5ab6772
20:43:33.494293 [debug] [Thread-4 (]: Began compiling node test.jaffle_shop.not_null_int_eph_first_second_column.f4df3457a3
20:43:33.502155 [debug] [Thread-1 (]: Writing injected SQL for node "test.jaffle_shop.not_null_fct_eph_first_first_column.47390b8f26"
20:43:33.504563 [debug] [Thread-2 (]: Timing info for test.jaffle_shop.not_null_fct_eph_first_second_column.19482e284e (compile): 2023-02-07 20:43:33.502304 => 2023-02-07 20:43:33.504528
20:43:33.509717 [debug] [Thread-3 (]: Writing injected SQL for node "test.jaffle_shop.not_null_int_eph_first_first_column.7fd5ab6772"
20:43:33.512284 [debug] [Thread-4 (]: Writing injected SQL for node "test.jaffle_shop.not_null_int_eph_first_second_column.f4df3457a3"
20:43:33.512788 [error] [Thread-2 (]: Unhandled error while executing test.jaffle_shop.not_null_fct_eph_first_second_column.19482e284e
sequence item 0: expected str instance, NoneType found
20:43:33.516319 [debug] [Thread-1 (]: Writing injected SQL for node "test.jaffle_shop.not_null_fct_eph_first_first_column.47390b8f26"
20:43:33.516659 [debug] [Thread-3 (]: Timing info for test.jaffle_shop.not_null_int_eph_first_first_column.7fd5ab6772 (compile): 2023-02-07 20:43:33.504692 => 2023-02-07 20:43:33.516634
20:43:33.516912 [debug] [Thread-4 (]: Timing info for test.jaffle_shop.not_null_int_eph_first_second_column.f4df3457a3 (compile): 2023-02-07 20:43:33.509885 => 2023-02-07 20:43:33.516889
20:43:33.517100 [debug] [Thread-3 (]: Began executing node test.jaffle_shop.not_null_int_eph_first_first_column.7fd5ab6772
20:43:33.517922 [debug] [Thread-1 (]: Writing injected SQL for node "test.jaffle_shop.not_null_fct_eph_first_first_column.47390b8f26"
20:43:33.518077 [debug] [Thread-4 (]: Began executing node test.jaffle_shop.not_null_int_eph_first_second_column.f4df3457a3
20:43:33.526652 [debug] [Thread-3 (]: Writing runtime sql for node "test.jaffle_shop.not_null_int_eph_first_first_column.7fd5ab6772"
20:43:33.527066 [debug] [Thread-2 (]: Traceback (most recent call last):
  File ".local/share/virtualenvs/dbt-test-gTAqrA9S/lib/python3.11/site-packages/dbt/task/base.py", line 367, in safe_run
    result = self.compile_and_execute(manifest, ctx)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/share/virtualenvs/dbt-test-gTAqrA9S/lib/python3.11/site-packages/dbt/task/base.py", line 304, in compile_and_execute
    ctx.node = self.compile(manifest)
               ^^^^^^^^^^^^^^^^^^^^^^
  File ".local/share/virtualenvs/dbt-test-gTAqrA9S/lib/python3.11/site-packages/dbt/task/compile.py", line 37, in compile
    return compiler.compile_node(self.node, manifest, {})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/share/virtualenvs/dbt-test-gTAqrA9S/lib/python3.11/site-packages/dbt/compilation.py", line 519, in compile_node
    node, _ = self._recursively_prepend_ctes(node, manifest, extra_context)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/share/virtualenvs/dbt-test-gTAqrA9S/lib/python3.11/site-packages/dbt/compilation.py", line 317, in _recursively_prepend_ctes
    injected_sql = self._inject_ctes_into_sql(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/share/virtualenvs/dbt-test-gTAqrA9S/lib/python3.11/site-packages/dbt/compilation.py", line 241, in _inject_ctes_into_sql
    token = sqlparse.sql.Token(sqlparse.tokens.Keyword, ", ".join(c.sql for c in ctes))
                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sequence item 0: expected str instance, NoneType found

20:43:33.528679 [debug] [Thread-4 (]: Writing runtime sql for node "test.jaffle_shop.not_null_int_eph_first_second_column.f4df3457a3"
20:43:33.528863 [debug] [Thread-1 (]: Timing info for test.jaffle_shop.not_null_fct_eph_first_first_column.47390b8f26 (compile): 2023-02-07 20:43:33.494398 => 2023-02-07 20:43:33.528834
20:43:33.528994 [error] [Thread-2 (]: 2 of 4 ERROR not_null_fct_eph_first_second_column .............................. [ERROR in 0.04s]
20:43:33.529244 [debug] [Thread-1 (]: Began executing node test.jaffle_shop.not_null_fct_eph_first_first_column.47390b8f26
20:43:33.529416 [debug] [Thread-3 (]: Opening a new connection, currently in state init
20:43:33.530175 [debug] [Thread-2 (]: Finished running node test.jaffle_shop.not_null_fct_eph_first_second_column.19482e284e
```

@gastlich Thanks for the full reproduction case! I was able to reliably produce the error while running locally with `1.4.latest`. I also saw the error after removing the tests on `fct_eph_first`, and defining tests on `fct_eph_first` only. I don't see the error while running with v1.3.

It sounds like the needed ingredients are:
- two ephemeral models, one referencing the other
- two generic tests defined on the second model
- running `dbt test` with multiple threads

We'll need to dig in some more to understand what's actually going on behind the scenes. It sounds like a thread safety issue with `_inject_ctes_into_sql` / `_recursively_prepend_ctes`.

Resolving patch:
diff --git a/core/dbt/compilation.py b/core/dbt/compilation.py
--- a/core/dbt/compilation.py
+++ b/core/dbt/compilation.py
@@ -95,10 +95,11 @@ def _generate_stats(manifest: Manifest):
 
 def _add_prepended_cte(prepended_ctes, new_cte):
     for cte in prepended_ctes:
-        if cte.id == new_cte.id:
+        if cte.id == new_cte.id and new_cte.sql:
             cte.sql = new_cte.sql
             return
-    prepended_ctes.append(new_cte)
+    if new_cte.sql:
+        prepended_ctes.append(new_cte)
 
 
 def _extend_prepended_ctes(prepended_ctes, new_prepended_ctes):
@@ -262,16 +263,18 @@ def _recursively_prepend_ctes(
         inserting CTEs into the SQL.
         """
         if model.compiled_code is None:
-            raise DbtRuntimeError("Cannot inject ctes into an unparsed node", model)
+            raise DbtRuntimeError("Cannot inject ctes into an uncompiled node", model)
+
+        # extra_ctes_injected flag says that we've already recursively injected the ctes
         if model.extra_ctes_injected:
             return (model, model.extra_ctes)
 
         # Just to make it plain that nothing is actually injected for this case
-        if not model.extra_ctes:
+        if len(model.extra_ctes) == 0:
+            # SeedNodes don't have compilation attributes
             if not isinstance(model, SeedNode):
                 model.extra_ctes_injected = True
-            manifest.update_node(model)
-            return (model, model.extra_ctes)
+            return (model, [])
 
         # This stores the ctes which will all be recursively
         # gathered and then "injected" into the model.
@@ -280,7 +283,8 @@ def _recursively_prepend_ctes(
         # extra_ctes are added to the model by
         # RuntimeRefResolver.create_relation, which adds an
         # extra_cte for every model relation which is an
-        # ephemeral model.
+        # ephemeral model. InjectedCTEs have a unique_id and sql.
+        # extra_ctes start out with sql set to None, and the sql is set in this loop.
         for cte in model.extra_ctes:
             if cte.id not in manifest.nodes:
                 raise DbtInternalError(
@@ -293,23 +297,23 @@ def _recursively_prepend_ctes(
             if not cte_model.is_ephemeral_model:
                 raise DbtInternalError(f"{cte.id} is not ephemeral")
 
-            # This model has already been compiled, so it's been
-            # through here before
-            if getattr(cte_model, "compiled", False):
+            # This model has already been compiled and extra_ctes_injected, so it's been
+            # through here before. We already checked above for extra_ctes_injected, but
+            # checking again because updates maybe have happened in another thread.
+            if cte_model.compiled is True and cte_model.extra_ctes_injected is True:
                 new_prepended_ctes = cte_model.extra_ctes
 
             # if the cte_model isn't compiled, i.e. first time here
             else:
                 # This is an ephemeral parsed model that we can compile.
-                # Compile and update the node
-                cte_model = self._compile_node(cte_model, manifest, extra_context)
-                # recursively call this method
+                # Render the raw_code and set compiled to True
+                cte_model = self._compile_code(cte_model, manifest, extra_context)
+                # recursively call this method, sets extra_ctes_injected to True
                 cte_model, new_prepended_ctes = self._recursively_prepend_ctes(
                     cte_model, manifest, extra_context
                 )
-                # Save compiled SQL file and sync manifest
+                # Write compiled SQL file
                 self._write_node(cte_model)
-                manifest.sync_update_node(cte_model)
 
             _extend_prepended_ctes(prepended_ctes, new_prepended_ctes)
 
@@ -323,20 +327,21 @@ def _recursively_prepend_ctes(
             model.compiled_code,
             prepended_ctes,
         )
-        model._pre_injected_sql = model.compiled_code
-        model.compiled_code = injected_sql
-        model.extra_ctes_injected = True
-        model.extra_ctes = prepended_ctes
-        model.validate(model.to_dict(omit_none=True))
-        manifest.update_node(model)
+        # Check again before updating for multi-threading
+        if not model.extra_ctes_injected:
+            model._pre_injected_sql = model.compiled_code
+            model.compiled_code = injected_sql
+            model.extra_ctes = prepended_ctes
+            model.extra_ctes_injected = True
 
-        return model, prepended_ctes
+        # if model.extra_ctes is not set to prepended ctes, something went wrong
+        return model, model.extra_ctes
 
-    # Sets compiled fields in the ManifestSQLNode passed in,
+    # Sets compiled_code and compiled flag in the ManifestSQLNode passed in,
     # creates a "context" dictionary for jinja rendering,
     # and then renders the "compiled_code" using the node, the
     # raw_code and the context.
-    def _compile_node(
+    def _compile_code(
         self,
         node: ManifestSQLNode,
         manifest: Manifest,
@@ -345,16 +350,6 @@ def _compile_node(
         if extra_context is None:
             extra_context = {}
 
-        data = node.to_dict(omit_none=True)
-        data.update(
-            {
-                "compiled": False,
-                "compiled_code": None,
-                "extra_ctes_injected": False,
-                "extra_ctes": [],
-            }
-        )
-
         if node.language == ModelLanguage.python:
             context = self._create_node_context(node, manifest, extra_context)
 
@@ -374,6 +369,8 @@ def _compile_node(
                 node,
             )
 
+        node.compiled = True
+
         # relation_name is set at parse time, except for tests without store_failures,
         # but cli param can turn on store_failures, so we set here.
         if (
@@ -386,8 +383,6 @@ def _compile_node(
             relation_name = str(relation_cls.create_from(self.config, node))
             node.relation_name = relation_name
 
-        node.compiled = True
-
         return node
 
     def write_graph_file(self, linker: Linker, manifest: Manifest):
@@ -525,11 +520,11 @@ def compile_node(
     ) -> ManifestSQLNode:
         """This is the main entry point into this code. It's called by
         CompileRunner.compile, GenericRPCRunner.compile, and
-        RunTask.get_hook_sql. It calls '_compile_node' to convert
-        the node into a compiled node, and then calls the
+        RunTask.get_hook_sql. It calls '_compile_code' to render
+        the node's raw_code into compiled_code, and then calls the
         recursive method to "prepend" the ctes.
         """
-        node = self._compile_node(node, manifest, extra_context)
+        node = self._compile_code(node, manifest, extra_context)
 
         node, _ = self._recursively_prepend_ctes(node, manifest, extra_context)
         if write:
diff --git a/core/dbt/contracts/graph/manifest.py b/core/dbt/contracts/graph/manifest.py
--- a/core/dbt/contracts/graph/manifest.py
+++ b/core/dbt/contracts/graph/manifest.py
@@ -648,24 +648,6 @@ def __post_deserialize__(cls, obj):
         obj._lock = MP_CONTEXT.Lock()
         return obj
 
-    def sync_update_node(self, new_node: ManifestNode) -> ManifestNode:
-        """update the node with a lock. The only time we should want to lock is
-        when compiling an ephemeral ancestor of a node at runtime, because
-        multiple threads could be just-in-time compiling the same ephemeral
-        dependency, and we want them to have a consistent view of the manifest.
-
-        If the existing node is not compiled, update it with the new node and
-        return that. If the existing node is compiled, do not update the
-        manifest and return the existing node.
-        """
-        with self._lock:
-            existing = self.nodes[new_node.unique_id]
-            if getattr(existing, "compiled", False):
-                # already compiled
-                return existing
-            _update_into(self.nodes, new_node)
-            return new_node
-
     def update_exposure(self, new_exposure: Exposure):
         _update_into(self.exposures, new_exposure)
 
diff --git a/core/dbt/contracts/graph/nodes.py b/core/dbt/contracts/graph/nodes.py
--- a/core/dbt/contracts/graph/nodes.py
+++ b/core/dbt/contracts/graph/nodes.py
@@ -414,8 +414,10 @@ def set_cte(self, cte_id: str, sql: str):
         do if extra_ctes were an OrderedDict
         """
         for cte in self.extra_ctes:
+            # Because it's possible that multiple threads are compiling the
+            # node at the same time, we don't want to overwrite already compiled
+            # sql in the extra_ctes with empty sql.
             if cte.id == cte_id:
-                cte.sql = sql
                 break
         else:
             self.extra_ctes.append(InjectedCTE(id=cte_id, sql=sql))


Test patch:
diff --git a/test/unit/test_compiler.py b/test/unit/test_compiler.py
--- a/test/unit/test_compiler.py
+++ b/test/unit/test_compiler.py
@@ -303,30 +303,6 @@ def test__prepend_ctes__cte_not_compiled(self):
             raw_code='select * from source_table',
             checksum=FileHash.from_contents(''),
         )
-        compiled_ephemeral = ModelNode(
-            name='ephemeral',
-            database='dbt',
-            schema='analytics',
-            alias='ephemeral',
-            resource_type=NodeType.Model,
-            unique_id='model.root.ephemeral',
-            fqn=['root', 'ephemeral'],
-            package_name='root',
-            refs=[],
-            sources=[],
-            depends_on=DependsOn(),
-            config=ephemeral_config,
-            tags=[],
-            path='ephemeral.sql',
-            original_file_path='ephemeral.sql',
-            language='sql',
-            raw_code='select * from source_table',
-            compiled=True,
-            compiled_code='select * from source_table',
-            extra_ctes_injected=True,
-            extra_ctes=[],
-            checksum=FileHash.from_contents(''),
-        )
         manifest = Manifest(
             macros={},
             nodes={
@@ -367,24 +343,17 @@ def test__prepend_ctes__cte_not_compiled(self):
         )
 
         compiler = dbt.compilation.Compiler(self.config)
-        with patch.object(compiler, '_compile_node') as compile_node:
-            compile_node.return_value = compiled_ephemeral
-
-            result, _ = compiler._recursively_prepend_ctes(
-                manifest.nodes['model.root.view'],
-                manifest,
-                {}
-            )
-            compile_node.assert_called_once_with(
-                parsed_ephemeral, manifest, {})
-
-        self.assertEqual(result,
-                         manifest.nodes.get('model.root.view'))
+        node = compiler.compile_node(
+            manifest.nodes['model.root.view'],
+            manifest,
+            {},
+            False
+        )
 
         self.assertTrue(manifest.nodes['model.root.ephemeral'].compiled)
-        self.assertTrue(result.extra_ctes_injected)
+        self.assertTrue(node.extra_ctes_injected)
         self.assertEqualIgnoreWhitespace(
-            result.compiled_code,
+            node.compiled_code,
             ('with __dbt__cte__ephemeral as ('
              'select * from source_table'
              ') '
diff --git a/tests/functional/materializations/test_ephemeral_compilation.py b/tests/functional/materializations/test_ephemeral_compilation.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/materializations/test_ephemeral_compilation.py
@@ -0,0 +1,71 @@
+import pytest
+from dbt.tests.util import run_dbt
+
+# Note: This tests compilation only, so is a dbt Core test and not an adapter test.
+# There is some complicated logic in core/dbt/compilation.py having to do with
+# ephemeral nodes and handling multiple threads at the same time. This test
+# fails fairly regularly if that is broken, but does occasionally work (depending
+# on the order in which things are compiled). It requires multi-threading to fail.
+
+
+fct_eph_first_sql = """
+-- fct_eph_first.sql
+{{ config(materialized='ephemeral') }}
+
+with int_eph_first as(
+    select * from {{ ref('int_eph_first') }}
+)
+
+select * from int_eph_first
+"""
+
+int_eph_first_sql = """
+-- int_eph_first.sql
+{{ config(materialized='ephemeral') }}
+
+select
+    1 as first_column,
+    2 as second_column
+"""
+
+schema_yml = """
+version: 2
+
+models:
+  - name: int_eph_first
+    columns:
+      - name: first_column
+        tests:
+          - not_null
+      - name: second_column
+        tests:
+          - not_null
+
+  - name: fct_eph_first
+    columns:
+      - name: first_column
+        tests:
+          - not_null
+      - name: second_column
+        tests:
+          - not_null
+
+"""
+
+
+class TestEphemeralCompilation:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "int_eph_first.sql": int_eph_first_sql,
+            "fct_eph_first.sql": fct_eph_first_sql,
+            "schema.yml": schema_yml,
+        }
+
+    def test_ephemeral_compilation(self, project):
+        # Note: There are no models that run successfully. This testcase tests running tests.
+        results = run_dbt(["run"])
+        assert len(results) == 0
+
+        results = run_dbt(["test"])
+        len(results) == 4