Repository: dbt-labs/dbt-core

Issue:
Pass `parsed_node` into `generate_schema_name`.
# Feature description
Recently, I submitted a feature to allow dbt users to specify a `generate_alias_name` macro in their project. Therefore, for example, the name of the table that the model is being built in could be implemented as a function of context like the `target` of the build or `tags` on the model. When implementing that feature, we decided to pass a `node` object into the `get_alias` function. https://github.com/fishtown-analytics/dbt/blob/4b7bddb48189c926e2fc6b753bd5328ce4f953bc/core/dbt/parser/base.py#L213

We have a use case where we would like to use different `schema` depending on `tags` that might be included in a model (e.g., we might launch a model in production with a `experimental` tag). While investigating if I could update our `generate_schema_name` macro to access the model's `tags`, I realized that there is no `model` or `node` object in the available context.

My proposal is that we make some kind of node or model object available at time of rendering a model's custom schema. I think it will be as easy as updating `get_schema` to also accept `parsed_node`: https://github.com/fishtown-analytics/dbt/blob/4b7bddb48189c926e2fc6b753bd5328ce4f953bc/core/dbt/parser/base.py#L209
(Note, we would probably have to add it as a second input and not the first. If that's the case, we should reverse the order of the inputs in `get_alias` prior to release of custom aliases, for consistency).

## Who will this benefit?
My organization! But also, anyone who might want to dynamically generate schema names based on things in the model `config`, like tags, sortkeys, distkeys, materialization, etc. Not just, for example, the `target` and `schema` in `config`.



Discussion:
+1 
@nydnarb thanks for making this issue! 

The core challenge with operating in this part of dbt's codebase is that the state of the resources that dbt operates on are still in flux! The schema name for a model is determined during parsing, so there are still some attributes of the ParsedNode that haven't been finalized yet.

Check out the [_update_parsed_node_info](https://github.com/fishtown-analytics/dbt/blob/73607b85b7825b7fca01a340c52469317eaeae7a/core/dbt/parser/base.py#L200-L230) function. The following attributes of the ParsedNode are all finalized in this order:
1. schema
2. alias
3. database
4. tags
5. config
6. pre-/post-hooks

Looking at this code again, I think we're actually not doing the strictly correct thing when generating an alias name! Specifically, the `database`, `tags`, `config`, and `hooks` that are present in the `parsed_node` on Line 213 do not account for configs that were specified in the model, eg:

```
{{ config(tags=['abc', '123']) }}

select 1 as id
```

Only configs that are specified in `dbt_project.yml` are present in the `parsed_node` object at the time when `get_alias` is currently being called! If you try to `log(parsed_node.tags)` in the `generate_alias_name` macro, I think you'll find that it results in an empty list using the model shown above.

I think there's a quick fix though - we just need to wait until these other attributes are finalized before calling `get_schema` and `get_alias`. There is some fundamental problem here though: either one of `get_schema` or `get_alias` will need to happen first, so the one that happens _last_ will potentially have the wrong value for the one that happens _first_. We either need to run dbt on quantum computers, or prevent folks from operating on these two values in their `generate_schema_name` and `generate_alias_name` macros (either via good docs, or something programatic).

Apologies for leading you astray in your `generate_alias_name` PR -- I think I recommended using `parsed_node` there, and I didn't realize that this would be a problem.

Overall though, I love the suggestion presented here, and think it will be a great addition to dbt if we can get the specifics right.

> Note, we would probably have to add it as a second input and not the first. If that's the case, we should reverse the order of the inputs in get_alias prior to release of custom aliases, for consistency

Yep, totally agree! I didn't realize `generate_alias_name` wasn't released yet - let's definitely swap the order of the args in `generate_alias_name` before that code is released! Really good catch :) 

cc @beckjake - did I miss anything here?


Resolving patch:
diff --git a/core/dbt/deprecations.py b/core/dbt/deprecations.py
--- a/core/dbt/deprecations.py
+++ b/core/dbt/deprecations.py
@@ -36,6 +36,17 @@ class SeedDropExistingDeprecation(DBTDeprecation):
   will be removed in a future version of dbt."""
 
 
+class GenerateSchemaNameSingleArgDeprecated(DBTDeprecation):
+    name = 'generate-schema-name-single-arg'
+    description = '''As of dbt v0.14.0, the `generate_schema_name` macro
+  accepts a second "node" argument. The one-argument form of `generate_schema_name`
+  is deprecated, and will become unsupported in a future release.
+
+  For more information, see:
+    https://docs.getdbt.com/v0.14/docs/upgrading-to-014
+  '''  # noqa
+
+
 _adapter_renamed_description = """\
 The adapter function `adapter.{old_name}` is deprecated and will be removed in
  a future release of dbt. Please use `adapter.{new_name}` instead.
@@ -72,6 +83,7 @@ def warn(name, *args, **kwargs):
 deprecations_list = [
     DBTRepositoriesDeprecation(),
     SeedDropExistingDeprecation(),
+    GenerateSchemaNameSingleArgDeprecated(),
 ]
 
 deprecations = {d.name: d for d in deprecations_list}
diff --git a/core/dbt/parser/base.py b/core/dbt/parser/base.py
--- a/core/dbt/parser/base.py
+++ b/core/dbt/parser/base.py
@@ -13,6 +13,7 @@
 from dbt.logger import GLOBAL_LOGGER as logger
 from dbt.contracts.graph.parsed import ParsedNode
 from dbt.parser.source_config import SourceConfig
+from dbt import deprecations
 
 
 class BaseParser(object):
@@ -82,8 +83,9 @@ def get_schema_func(self):
                 'generate_schema_name',
                 GLOBAL_PROJECT_NAME
             )
+        # this is only true in tests!
         if get_schema_macro is None:
-            def get_schema(_):
+            def get_schema(custom_schema_name=None, node=None):
                 return self.default_schema
         else:
             root_context = dbt.context.parser.generate_macro(
@@ -117,8 +119,10 @@ def get_alias_func(self):
                 'generate_alias_name',
                 GLOBAL_PROJECT_NAME
             )
+
+        # the generate_alias_name macro might not exist
         if get_alias_macro is None:
-            def get_alias(node, custom_alias_name=None):
+            def get_alias(custom_alias_name, node):
                 if custom_alias_name is None:
                     return node.name
                 else:
@@ -206,11 +210,22 @@ def _update_parsed_node_info(self, parsed_node, config):
         # definition, not the current package
         schema_override = config.config.get('schema')
         get_schema = self.get_schema_func()
-        parsed_node.schema = get_schema(schema_override).strip()
+        try:
+            schema = get_schema(schema_override, parsed_node)
+        except dbt.exceptions.CompilationException as exc:
+            too_many_args = (
+                "macro 'dbt_macro__generate_schema_name' takes not more than "
+                "1 argument(s)"
+            )
+            if too_many_args not in str(exc):
+                raise
+            deprecations.warn('generate-schema-name-single-arg')
+            schema = get_schema(schema_override)
+        parsed_node.schema = schema.strip()
 
         alias_override = config.config.get('alias')
         get_alias = self.get_alias_func()
-        parsed_node.alias = get_alias(parsed_node, alias_override).strip()
+        parsed_node.alias = get_alias(alias_override, parsed_node).strip()
 
         parsed_node.database = config.config.get(
             'database', self.default_database


Test patch:
diff --git a/test/integration/006_simple_dependency_test/schema_override_macros/schema.sql b/test/integration/006_simple_dependency_test/schema_override_macros/schema.sql
--- a/test/integration/006_simple_dependency_test/schema_override_macros/schema.sql
+++ b/test/integration/006_simple_dependency_test/schema_override_macros/schema.sql
@@ -1,6 +1,6 @@
 
-{% macro generate_schema_name(schema_name) -%}
+{% macro generate_schema_name(schema_name, node) -%}
 
-    {{ schema_name }}_{{ target.schema }}_macro
+    {{ schema_name }}_{{ node.schema }}_macro
 
 {%- endmacro %}
diff --git a/test/integration/006_simple_dependency_test/test_local_dependency.py b/test/integration/006_simple_dependency_test/test_local_dependency.py
--- a/test/integration/006_simple_dependency_test/test_local_dependency.py
+++ b/test/integration/006_simple_dependency_test/test_local_dependency.py
@@ -7,8 +7,7 @@
 import dbt.exceptions
 
 
-class TestSimpleDependency(DBTIntegrationTest):
-
+class BaseDependencyTest(DBTIntegrationTest):
     @property
     def schema(self):
         return "local_dependency_006"
@@ -17,6 +16,12 @@ def schema(self):
     def models(self):
         return "test/integration/006_simple_dependency_test/local_models"
 
+    def base_schema(self):
+        return self.unique_schema()
+
+    def configured_schema(self):
+        return self.unique_schema() + '_configured'
+
     @property
     def packages_config(self):
         return {
@@ -27,6 +32,17 @@ def packages_config(self):
             ]
         }
 
+
+class TestSimpleDependency(BaseDependencyTest):
+
+    @property
+    def schema(self):
+        return "local_dependency_006"
+
+    @property
+    def models(self):
+        return "test/integration/006_simple_dependency_test/local_models"
+
     def base_schema(self):
         return self.unique_schema()
 
diff --git a/test/integration/012_deprecation_tests/boring-models/boring.sql b/test/integration/012_deprecation_tests/boring-models/boring.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/012_deprecation_tests/boring-models/boring.sql
@@ -0,0 +1 @@
+select 1 as id
diff --git a/test/integration/012_deprecation_tests/macros/schema.sql b/test/integration/012_deprecation_tests/macros/schema.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/012_deprecation_tests/macros/schema.sql
@@ -0,0 +1,7 @@
+{% macro generate_schema_name(schema_name) -%}
+	{%- if schema_name is none -%}
+		{{ target.schema }}
+	{%- else -%}
+		{{ schema_name }}
+	{%- endif -%}
+{%- endmacro %}
diff --git a/test/integration/012_deprecation_tests/test_deprecations.py b/test/integration/012_deprecation_tests/test_deprecations.py
--- a/test/integration/012_deprecation_tests/test_deprecations.py
+++ b/test/integration/012_deprecation_tests/test_deprecations.py
@@ -4,9 +4,9 @@
 import dbt.exceptions
 
 
-class TestDeprecations(DBTIntegrationTest):
+class BaseTestDeprecations(DBTIntegrationTest):
     def setUp(self):
-        super(TestDeprecations, self).setUp()
+        super(BaseTestDeprecations, self).setUp()
         deprecations.reset_deprecations()
 
     @property
@@ -21,6 +21,8 @@ def dir(path):
     def models(self):
         return self.dir("models")
 
+
+class TestDeprecations(BaseTestDeprecations):
     @use_profile('postgres')
     def test_postgres_deprecations_fail(self):
         self.run_dbt(strict=True, expect_pass=False)
@@ -29,5 +31,29 @@ def test_postgres_deprecations_fail(self):
     def test_postgres_deprecations(self):
         self.assertEqual(deprecations.active_deprecations, set())
         self.run_dbt(strict=False)
-        self.assertEqual({'adapter:already_exists'},
-                         deprecations.active_deprecations)
+        expected = {'adapter:already_exists'}
+        self.assertEqual(expected, deprecations.active_deprecations)
+
+
+class TestMacroDeprecations(BaseTestDeprecations):
+    @property
+    def models(self):
+        return self.dir('boring-models')
+
+    @property
+    def project_config(self):
+        return {
+            'macro-paths': [self.dir('macros')],
+        }
+
+    @use_profile('postgres')
+    def test_postgres_deprecations_fail(self):
+        with self.assertRaises(dbt.exceptions.CompilationException):
+            self.run_dbt(strict=True)
+
+    @use_profile('postgres')
+    def test_postgres_deprecations(self):
+        self.assertEqual(deprecations.active_deprecations, set())
+        self.run_dbt(strict=False)
+        expected = {'generate-schema-name-single-arg'}
+        self.assertEqual(expected, deprecations.active_deprecations)
diff --git a/test/integration/024_custom_schema_test/macros/schema.sql b/test/integration/024_custom_schema_test/macros/schema.sql
--- a/test/integration/024_custom_schema_test/macros/schema.sql
+++ b/test/integration/024_custom_schema_test/macros/schema.sql
@@ -1,5 +1,5 @@
 
-{% macro generate_schema_name(schema_name) %}
+{% macro generate_schema_name(schema_name, node) %}
 
     {{ schema_name }}_{{ target.schema }}_macro
 
diff --git a/test/integration/043_custom_aliases_test/macros/macros.sql b/test/integration/043_custom_aliases_test/macros/macros.sql
--- a/test/integration/043_custom_aliases_test/macros/macros.sql
+++ b/test/integration/043_custom_aliases_test/macros/macros.sql
@@ -1,5 +1,5 @@
 
-{% macro generate_alias_name(node, custom_alias_name=none) -%}
+{% macro generate_alias_name(custom_alias_name, node) -%}
     {%- if custom_alias_name is none -%}
         {{ node.name }}
     {%- else -%}