Repository: dbt-labs/dbt-core

Issue:
Jinja context variable for selected resources
### Describe the feature

A Jinja context variable that includes the list of resources (by `unique_id`) selected in the current invocation:
```
{% if 'model.package_name.my_model' in selected_resources %}
  ...
{% endif %}
```

This would have similar caveats to `graph` and `execute`, because it wouldn't be populated at parse time, only at compile/execute/runtime.

Bonus points if it includes DAG information, i.e. the order in which resources will be queued for execution.

Use cases:
- An `on-run-start` hook that wants to know all resources about to execute. This is available in the `on-run-end` context, in the form of the [`Results`](https://docs.getdbt.com/reference/dbt-jinja-functions/on-run-end-context#results) object.
- Use standard node selection syntax as the input to a macro. E.g. run [`audit_helper.compare_relations()`](https://github.com/fishtown-analytics/dbt-audit-helper/tree/0.3.0/#compare_relations-source) for each resource selected/built by a CI job, to ensure no regressions, by looping over `selected_resources` in a `run-operation` (https://github.com/fishtown-analytics/dbt/issues/2740)
    - Could we do a half-decent version of this with `Results` today in an `on-run-end` hook?
    - Is this better as `post-hook` on each model?

### Describe alternatives you've considered

- Adding a property to manifest nodes: `selected: true|false`. Then users would access the same `graph` context variable and use that property. I understand that some users already do something similar, e.g. inspect the `tags` on each node, to emulate selection criteria. We could also consider `selection_criteria`, detailing _why_ a given node has been selected. This could be included in the `list` output and a helpful tool for visualizing selection groupings.

- Make the selection criteria directly available, via `flags.args.models` and `flags.args.exclude`: https://github.com/fishtown-analytics/dbt/issues/2253#issuecomment-606637983

### Additional context

Further down the road:
- The `run-operation` task should accept `--select` and `--selector`
- Custom tasks (https://github.com/fishtown-analytics/dbt/issues/2381) that can run nodes in parallel using threads, rather than in sequence

### Who will this benefit?

This comes up in a surprising number of more-complex use cases.


Discussion:
Is there any sort of workaround available for something like this right now? Not a blocker at the moment, but would make deployment logic in my use case much terser and more intuitive.

### My use case

I'm using a custom `generate_schema_name` macro that includes a `__dbt_tmp_<timestamp>` suffix on the _schema_ to do blue/green style promotion at the _schema_ level instead of just the table level.

However I'm trying to make it so that nodes that are outside the selection for rebuilding are ref'd from their un-suffixed schema name rather than the schema with temp suffix (in which they don't exist).

Currently, because my main project is split into three (root project → source project 1 & source project 2), and the only selections I currently do in my build process are at the package level (i.e. _either_ source project1 _or_ source project 2 _or_ root project), right now I just check whether `node.package_name == project_name` in my `generate_schema_name` macro. However I'd love to replace that check with `node.unique_id in selected_nodes`!
Hi! Would it be ok for me to contribute to this issue? There are a few occasions where I wished I had this information in the Jinja context.

From what I see we would need to add a `@contexproperty` function in `context/providers.py` and potentially retrieve the resources from `get_graph_queue()`. Does that seem right?

I have not thought about tests so far but feel free to mention where would be the best place to add tests around the feature.
@b-per I'm all for you taking a crack at it. The broad overview you've got sounds about right; my instinct would be to retrieve the resources (list of `unique_id`) from `selector.get_selected`, rather than `GraphQueue`.

We'll want to very clearly document that, just as with `flags`, users should _not_ use `selected_resources` as the input to any parse-time logic (dependencies or configs). I imagine `selected_resources` should be an empty list at parse time.
After having given a further look, it doesn't look as straightforward as I thought it would be 😄 .

`get_selected` is executed on a `NodeSelector` that requires a `Graph` (not available in the current context) and a `Manifest` (available in `context/providers.py`) and takes as input a `spec` object (not available in the current context).

```python
class ProviderContext(ManifestContext):
    # subclasses are MacroContext, ModelContext, TestContext
    def __init__(
        self,
        model,
        config: RuntimeConfig,
        manifest: Manifest,
        provider: Provider,
        context_config: Optional[ContextConfig],
    ) -> None:
```

Should the approach be to import the `graph` and `spec` objects in `context/providers.py` and calculate `get_selected` in `selected_resources` or to calculate it outside of the context and import it already calculated?


Not surprised it's a bit tricky :)

> calculate it outside of the context and import it already calculated?

This one feels like the better approach. I'm not sure which object it should live on, or the extent to which it would require rewiring in our provider. Someone from the Language pr Execution team might be able to provide clearer guidance (since this functionality really spans the two).

Resolving patch:
diff --git a/core/dbt/context/providers.py b/core/dbt/context/providers.py
--- a/core/dbt/context/providers.py
+++ b/core/dbt/context/providers.py
@@ -62,6 +62,8 @@
 
 from dbt.utils import merge, AttrDict, MultiDict
 
+from dbt import selected_resources
+
 import agate
 
 
@@ -1143,6 +1145,15 @@ def env_var(self, var: str, default: Optional[str] = None) -> str:
             msg = f"Env var required but not provided: '{var}'"
             raise_parsing_error(msg)
 
+    @contextproperty
+    def selected_resources(self) -> List[str]:
+        """The `selected_resources` variable contains a list of the resources
+        selected based on the parameters provided to the dbt command.
+        Currently, is not populated for the command `run-operation` that
+        doesn't support `--select`.
+        """
+        return selected_resources.SELECTED_RESOURCES
+
 
 class MacroContext(ProviderContext):
     """Internally, macros can be executed like nodes, with some restrictions:
diff --git a/core/dbt/graph/selector.py b/core/dbt/graph/selector.py
--- a/core/dbt/graph/selector.py
+++ b/core/dbt/graph/selector.py
@@ -17,6 +17,8 @@
 from dbt.contracts.graph.manifest import Manifest
 from dbt.contracts.state import PreviousState
 
+from dbt import selected_resources
+
 
 def get_package_names(nodes):
     return set([node.split(".")[1] for node in nodes])
@@ -269,6 +271,7 @@ def get_graph_queue(self, spec: SelectionSpec) -> GraphQueue:
         dependecies.
         """
         selected_nodes = self.get_selected(spec)
+        selected_resources.set_selected_resources(selected_nodes)
         new_graph = self.full_graph.get_subset_graph(selected_nodes)
         # should we give a way here for consumers to mutate the graph?
         return GraphQueue(new_graph.graph, self.manifest, selected_nodes)
diff --git a/core/dbt/selected_resources.py b/core/dbt/selected_resources.py
new file mode 100644
--- /dev/null
+++ b/core/dbt/selected_resources.py
@@ -0,0 +1,6 @@
+SELECTED_RESOURCES = []
+
+
+def set_selected_resources(selected_resources):
+    global SELECTED_RESOURCES
+    SELECTED_RESOURCES = list(selected_resources)


Test patch:
diff --git a/test/unit/test_context.py b/test/unit/test_context.py
--- a/test/unit/test_context.py
+++ b/test/unit/test_context.py
@@ -221,6 +221,7 @@ def assert_has_keys(
     'sql',
     'sql_now',
     'adapter_macro',
+    'selected_resources'
 }
 REQUIRED_MODEL_KEYS = REQUIRED_MACRO_KEYS | {'this'}
 MAYBE_KEYS = frozenset({'debug'})
diff --git a/tests/functional/selected_resources/fixtures.py b/tests/functional/selected_resources/fixtures.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/selected_resources/fixtures.py
@@ -0,0 +1,35 @@
+on_run_start_macro_assert_selected_models_expected_list = """
+{% macro assert_selected_models_expected_list(expected_list) %}
+
+  {% if execute and (expected_list is not none) %}
+
+    {% set sorted_selected_resources = selected_resources | sort %}
+    {% set sorted_expected_list = expected_list | sort %}
+
+    {% if sorted_selected_resources != sorted_expected_list %}
+      {{ exceptions.raise_compiler_error("FAIL: sorted_selected_resources" ~ sorted_selected_resources ~ " is different from " ~ sorted_expected_list) }}
+    {% endif %}
+
+  {% endif %}
+
+{% endmacro %}
+"""
+
+
+my_model1 = """
+select 1 as id
+"""
+
+my_model2 = """
+select * from {{ ref('model1') }}
+"""
+
+my_snapshot = """
+{% snapshot cc_all_snapshot %}
+    {{ config(
+        check_cols='all', unique_key='id', strategy='check',
+        target_database=database, target_schema=schema
+    ) }}
+    select * from {{ ref('model2') }}
+{% endsnapshot %}
+"""
diff --git a/tests/functional/selected_resources/test_selected_resources.py b/tests/functional/selected_resources/test_selected_resources.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/selected_resources/test_selected_resources.py
@@ -0,0 +1,104 @@
+import pytest
+from dbt.tests.util import run_dbt
+from tests.functional.selected_resources.fixtures import (
+    on_run_start_macro_assert_selected_models_expected_list,
+    my_model1,
+    my_model2,
+    my_snapshot,
+)
+
+
+@pytest.fixture(scope="class")
+def macros():
+    return {
+        "assert_selected_models_expected_list.sql": on_run_start_macro_assert_selected_models_expected_list,
+    }
+
+
+@pytest.fixture(scope="class")
+def models():
+    return {"model1.sql": my_model1, "model2.sql": my_model2}
+
+
+@pytest.fixture(scope="class")
+def snapshots():
+    return {
+        "my_snapshot.sql": my_snapshot,
+    }
+
+
+@pytest.fixture(scope="class")
+def project_config_update():
+    return {
+        "on-run-start": "{{ assert_selected_models_expected_list(var('expected_list',None)) }}",
+    }
+
+
+@pytest.fixture
+def build_all(project):
+    run_dbt(["build"])
+
+
+@pytest.mark.usefixtures("build_all")
+class TestSelectedResources:
+    def test_selected_resources_build_selector(self, project):
+        results = run_dbt(
+            [
+                "build",
+                "--select",
+                "model1+",
+                "--vars",
+                '{"expected_list": ["model.test.model1", "model.test.model2", "snapshot.test.cc_all_snapshot"]}',
+            ]
+        )
+        assert results[0].status == "success"
+
+    def test_selected_resources_build_selector_subgraph(self, project):
+        results = run_dbt(
+            [
+                "build",
+                "--select",
+                "model2+",
+                "--vars",
+                '{"expected_list": ["model.test.model2", "snapshot.test.cc_all_snapshot"]}',
+            ]
+        )
+        assert results[0].status == "success"
+
+    def test_selected_resources_run(self, project):
+        results = run_dbt(
+            [
+                "run",
+                "--select",
+                "model1+",
+                "--vars",
+                '{"expected_list": ["model.test.model2", "model.test.model1"]}',
+            ]
+        )
+        assert results[0].status == "success"
+
+    def test_selected_resources_build_no_selector(self, project):
+        results = run_dbt(
+            [
+                "build",
+                "--vars",
+                '{"expected_list": ["model.test.model1", "model.test.model2", "snapshot.test.cc_all_snapshot"]}',
+            ]
+        )
+        assert results[0].status == "success"
+
+    def test_selected_resources_build_no_model(self, project):
+        results = run_dbt(
+            [
+                "build",
+                "--select",
+                "model_that_does_not_exist",
+                "--vars",
+                '{"expected_list": []}',
+            ]
+        )
+        assert not results
+
+    def test_selected_resources_test_no_model(self, project):
+        results = run_dbt(["test", "--select", "model1+", "--vars", '{"expected_list": []}'])
+        assert not results