Repository: dbt-labs/dbt-core

Issue:
Improve utility of database & schema information present in on-run-end hooks
### Describe the bug
When running `dbt run` with a profile that includes the example macro of https://docs.getdbt.com/docs/on-run-end-context and writes to multiple databases, the grants are only executed on one of the databases, not the correct database of the schema

### Steps To Reproduce
Use the macro at https://docs.getdbt.com/docs/on-run-end-context

Use a `dbt_project.yml` similar to:
```
models:
  dbt_project:
    staging:
      schema: staging
      database: data_warehouse
    dim_models:
      schema: dim_models
      database: data_warehouse
    data_marts:
      schema: data_marts
      database: data_warehouse
    exports:
      schema: dropbox
      database: exports
on-run-end:
  - "{{ grant_on_schemas('all privileges', schemas, 'accountadmin', true) }}"
```

When any model in `exports` is run, confirm that the `grant` statement fails because it attempts to run on the database `data_warehouse` instead, where `dropbox` schema doesn't exist

### Expected behavior
The macro should execute on the database of the schema(s) passed in

### Screenshots and log output

```
12:09:17 | Concurrency: 16 threads (target='prod')
12:09:17 |
.
.
.
12:09:40 |
12:09:40 | Running 3 on-run-end hooks
12:09:40 | 1 of 3 START hook: dbt_project.on-run-end.0.............................. [RUN]
Database error while running on-run-end
Encountered an error:
Database Error
  002003 (02000): 01904edd-0042-9085-0000-243d012d02b2: SQL compilation error:
  Schema 'DATA_WAREHOUSE.DROPBOX' does not exist or not authorized.
```

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [ ] bigquery
- [x] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
installed version: 0.14.3
   latest version: 0.14.4

Your version of dbt is out of date! You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
```

**The operating system you're using:**
OS X 10.14.5

**The output of `python --version`:**
Python 3.7.5

### Additional context
N/A



Discussion:
Thanks for the report @dinedal - I updated the title of this issue to make it a little bit more actionable. I think the big problem here is that dbt renders out `schemas` in the `on-run-end` hook, but really, we want to include the database _and_ the schema for use in hook code.

I wouldn't feel great about changing the `schemas` list to be a set of `<database>.<schema>` as that would probably break a lot of existing projects. Instead, we can add a new context variable (tbd what that would be called) which contains information about the databases _and_ schemas that dbt objects were created in.

One other thing to note here: it might be worth including information about node types. I know some folks want to grant access to snapshots and models separately - it's not currently possible to programatically differentiate between schemas that dbt built snapshots in vs. schemas the dbt built models in.

Last: I think `on-run-end` has sort of run it's course. As dbt projects are becoming more complicated, I think we'll need a more flexible way to define project-level hooks based on environment variables, the dbt command that was run, etc. TBD if we tackle this as a part of #1842 or a different issue, but did want to surface it here.

Thanks!
> Instead, we can add a new context variable (tbd what that would be called) which contains information about the databases and schemas that dbt objects were created in.

This would work fine for now

> Last: I think on-run-end has sort of run it's course.

It seems like #1842 is quite a large leap, would love to see dbt progress there for sure, however to just add the database to the context of the macro seems like a much smaller change?
>One other thing to note here: it might be worth including information about node types. I know some folks want to grant access to snapshots and models separately - it's not currently possible to programatically differentiate between schemas that dbt built snapshots in vs. schemas the dbt built models in.

That information is actually available, as is the information you want (though it's available in a very clunky way): Each entry in the `results` list in the context has a `node` attribute, and the `node` attribute has a `database` attribute, a `schema` attribute, and a `resource_type` attribute. You could do something horrible like this:
```
{% set db_schemas = [] %}
{% for result in results %}
  {% if result.error is none and not result.failed and not result.skipped %}
    {% set db_schema = (result.node.database, result.node.schema) %}
    {% if db_schema not in db_schemas %}
      {% do db_schemas.append(db_schema) %}
    {% endif %}
  {% endif %}
{% endfor %}
```

That's basically what dbt does currently to generate the `schemas` context value, just with `database` considered as well.

That said, I think it does make a lot of sense to add a `database_schemas` set of unique `(database, schema)` pairs as we can do that much more efficiently. 

I think if users want to do complicated things like think about node types, they should probably just implement something like the stuff above. Especially because you can just check the value of `results[0].node.resource_type`, and if it's `'snapshot'`, you know you are in `dbt snapshot` and can react accordingly.
@beckjake 
This code seems to return a value of `database` on the key `result.node.database` sometimes, not sure what causes it or why, otherwise it does appear to work, so thank you at least for this.
Hmm. I'm not sure why that would be. I thought that might be some sort of sneaky old default lingering in dbt, but I don't see anything in the code like that. Is it possible that your profiles.yml config has `database: database`?

Resolving patch:
diff --git a/core/dbt/task/run.py b/core/dbt/task/run.py
--- a/core/dbt/task/run.py
+++ b/core/dbt/task/run.py
@@ -1,6 +1,6 @@
 import functools
 import time
-from typing import List
+from typing import List, Dict, Any, Set, Tuple, Optional
 
 from dbt.logger import (
     GLOBAL_LOGGER as logger,
@@ -148,7 +148,9 @@ def run_hooks(self, adapter, hook_type: RunHookType, extra_context):
         with TextOnly():
             print_timestamped_line("")
 
-    def safe_run_hooks(self, adapter, hook_type: RunHookType, extra_context):
+    def safe_run_hooks(
+        self, adapter, hook_type: RunHookType, extra_context: Dict[str, Any]
+    ) -> None:
         try:
             self.run_hooks(adapter, hook_type, extra_context)
         except dbt.exceptions.RuntimeException:
@@ -178,18 +180,23 @@ def before_run(self, adapter, selected_uids):
             self.safe_run_hooks(adapter, RunHookType.Start, {})
 
     def after_run(self, adapter, results):
-        # in on-run-end hooks, provide the value 'schemas', which is a list of
-        # unique schemas that successfully executed models were in
-        # errored failed skipped
-        schemas = list(set(
-            r.node.schema for r in results
+        # in on-run-end hooks, provide the value 'database_schemas', which is a
+        # list  of unique database, schema pairs that successfully executed
+        # models  were in. for backwards compatibility, include the old
+        # 'schemas', which did not include database information.
+        database_schema_set: Set[Tuple[Optional[str], str]] = {
+            (r.node.database, r.node.schema) for r in results
             if not any((r.error is not None, r.fail, r.skipped))
-        ))
-
+        }
         self._total_executed += len(results)
+
+        extras = {
+            'schemas': list({s for _, s in database_schema_set}),
+            'results': results,
+            'database_schemas': list(database_schema_set),
+        }
         with adapter.connection_named('master'):
-            self.safe_run_hooks(adapter, RunHookType.End,
-                                {'schemas': schemas, 'results': results})
+            self.safe_run_hooks(adapter, RunHookType.End, extras)
 
     def after_hooks(self, adapter, results, elapsed):
         self.print_results_line(results, elapsed)


Test patch:
diff --git a/core/dbt/task/test.py b/core/dbt/task/test.py
--- a/core/dbt/task/test.py
+++ b/core/dbt/task/test.py
@@ -1,5 +1,7 @@
+from typing import Dict, Any
+
 from dbt.node_runners import TestRunner
-from dbt.node_types import NodeType
+from dbt.node_types import NodeType, RunHookType
 from dbt.task.run import RunTask
 
 
@@ -12,7 +14,9 @@ class TestTask(RunTask):
     def raise_on_first_error(self):
         return False
 
-    def safe_run_hooks(self, adapter, hook_type, extra_context):
+    def safe_run_hooks(
+        self, adapter, hook_type: RunHookType, extra_context: Dict[str, Any]
+    ) -> None:
         # Don't execute on-run-* hooks for tests
         pass
 
diff --git a/test/integration/014_hook_tests/test_run_hooks.py b/test/integration/014_hook_tests/test_run_hooks.py
--- a/test/integration/014_hook_tests/test_run_hooks.py
+++ b/test/integration/014_hook_tests/test_run_hooks.py
@@ -1,5 +1,6 @@
 from test.integration.base import DBTIntegrationTest, use_profile
 
+
 class TestPrePostRunHooks(DBTIntegrationTest):
 
     def setUp(self):
@@ -45,7 +46,9 @@ def project_config(self):
                 "create table {{ target.schema }}.end_hook_order_test ( id int )",
                 "drop table {{ target.schema }}.end_hook_order_test",
                 "create table {{ target.schema }}.schemas ( schema text )",
-                "insert into {{ target.schema }}.schemas values ({% for schema in schemas %}( '{{ schema }}' ){% if not loop.last %},{% endif %}{% endfor %})",
+                "insert into {{ target.schema }}.schemas (schema) values {% for schema in schemas %}( '{{ schema }}' ){% if not loop.last %},{% endif %}{% endfor %}",
+                "create table {{ target.schema }}.db_schemas ( db text, schema text )",
+                "insert into {{ target.schema }}.db_schemas (db, schema) values {% for db, schema in database_schemas %}('{{ db }}', '{{ schema }}' ){% if not loop.last %},{% endif %}{% endfor %}",
             ],
             'seeds': {
                 'quote_columns': False,
@@ -73,6 +76,12 @@ def assert_used_schemas(self):
         self.assertEqual(len(results), 1)
         self.assertEqual(results[0][0], self.unique_schema())
 
+        db_schemas_query = 'select * from {}.db_schemas'.format(self.unique_schema())
+        results = self.run_sql(db_schemas_query, fetch='all')
+        self.assertEqual(len(results), 1)
+        self.assertEqual(results[0][0], self.default_database)
+        self.assertEqual(results[0][1], self.unique_schema())
+
     def check_hooks(self, state):
         ctx = self.get_ctx_vars(state)