Repository: dbt-labs/dbt-core

Issue:
Scrub environment variables in target + logs
### Describe the feature

Could we scrub environment variables (values templated by `env_var`) from `target/` and `logs/`?

We could consider doing this for:
- All environment variables
- Environment variables prefixed with `SECRET_`

I don't have a great sense of what would be involved here. The best approach would tie into each actual `env_var` call. A less-good approach would grep strings for env vars before writing to files.

### Describe alternatives you've considered

The primary motivation for this request is dbt Cloud. dbt Core feels like the best place to handle it, however, and many Core users have asked for it ([e.g.](https://github.com/fishtown-analytics/dbt-spark/issues/147#issuecomment-843627607)).

### Additional context

- Not specific to any database
- Scrubbing environment variables from JSON artifacts also scrubs them in dbt-docs

### Who will this benefit?

- More complex, security-conscious deployments


Discussion:
@drewbanin Tagged as requested! Would love to hear your thoughts here, as it sounds like we've done something similar with logbook before.
@jtcohen6 I am not 100% sure of the canonical way to do this, but I think that we can add a log processor to the logbook LogManager

This is where we set up different log handlers (they emit logs) and log processors (they modify log lines). See some examples [here](https://logbook.readthedocs.io/en/stable/setups.html#deeply-nested-setups)

So, we'd add a new processor around here:
https://github.com/dbt-labs/dbt/blob/b9d5123aa37786425f7429c91f4513839eea6a65/core/dbt/logger.py#L467-L486

And that processor could look something like:
```
class ScrubProcessor(logbook.Processor):
    def process(self, record):
        record.message = record.message.replace("dbt", "♡")
```

Think we'll have some work to do to capture relevant environment variables for scrubbing, but if we do it this way, all log output (whether to stdout, stderr, json/text formatting, written to a file) should be scrubbed of specified secrets :)
@drewbanin Thanks for the pointers! That approach seems eminently doable, I just tried it locally and had some success:

```python
SECRET_ENV_PREFIX = 'DBT_ENV_SECRET_'

def get_metadata_env() -> List[str]:
    return [
        v for k, v in os.environ.items()
        if k.startswith(SECRET_ENV_PREFIX)
    ]

class ScrubProcessor(logbook.Processor):
    def process(self, record):
        for secret in get_metadata_env():
            record.message = record.message.replace(secret, "♡" * len(secret))
```
```
Completed with 2 errors and 0 warnings:

Database Error in model view_1 (models/view_1.sql)
  relation "♡♡♡♡♡♡♡♡♡♡♡.table1" does not exist
  LINE 10: FROM ♡♡♡♡♡♡♡♡♡♡♡.table1
                ^
  compiled SQL at target/run/my_project/models/view_1.sql

Database Error in model view_2 (models/view_2.sql)
  relation "♡♡♡♡♡♡♡♡♡♡♡.table2" does not exist
  LINE 14: FROM ♡♡♡♡♡♡♡♡♡♡♡.table2
                ^
  compiled SQL at target/run/my_project/models/view_2.sql
``` 

Worth noting that this won't scrub env var output in dbt artifacts (`manifest.json`, `run_results.json`), but that doesn't tend to be the most sensitive information. At that point, something is being written to documentation or the database. We're much more interested in use cases like:
- Git access tokens for `dbt deps`
- Cloud storage secrets, e.g. for reading external data
cool! Just want to be clear about this - I have no problem at all with using a ❤️ in these log lines. It's 2021, apps should support unicode & I think Snowflake used to use smiley faces :p

I think that if you're compiling env vars into a model (or similar) then it's reasonable to see their unredacted versions in the build artifacts. I could buy an argument for scrubbing them, but don't know that it needs to happen in the v1 of this thing.

Resolving patch:
diff --git a/core/dbt/logger.py b/core/dbt/logger.py
--- a/core/dbt/logger.py
+++ b/core/dbt/logger.py
@@ -43,6 +43,15 @@
     '{record.message}'
 )
 
+SECRET_ENV_PREFIX = 'DBT_ENV_SECRET_'
+
+
+def get_secret_env() -> List[str]:
+    return [
+        v for k, v in os.environ.items()
+        if k.startswith(SECRET_ENV_PREFIX)
+    ]
+
 
 ExceptionInformation = str
 
@@ -333,6 +342,12 @@ def process(self, record):
         record.extra[self.name] = datetime.utcnow().isoformat()
 
 
+class ScrubSecrets(logbook.Processor):
+    def process(self, record):
+        for secret in get_secret_env():
+            record.message = record.message.replace(secret, "*****")
+
+
 logger = logbook.Logger('dbt')
 # provide this for the cache, disabled by default
 CACHE_LOGGER = logbook.Logger('dbt.cache')
@@ -473,7 +488,8 @@ def __init__(self, stdout=colorama_stdout, stderr=sys.stderr):
         self._file_handler = DelayedFileHandler()
         self._relevel_processor = Relevel(allowed=['dbt', 'werkzeug'])
         self._state_processor = DbtProcessState('internal')
-        # keep track of wheter we've already entered to decide if we should
+        self._scrub_processor = ScrubSecrets()
+        # keep track of whether we've already entered to decide if we should
         # be actually pushing. This allows us to log in main() and also
         # support entering dbt execution via handle_and_check.
         self._stack_depth = 0
@@ -483,6 +499,7 @@ def __init__(self, stdout=colorama_stdout, stderr=sys.stderr):
             self._file_handler,
             self._relevel_processor,
             self._state_processor,
+            self._scrub_processor
         ])
 
     def push_application(self):


Test patch:
diff --git a/test/integration/013_context_var_tests/models/context.sql b/test/integration/013_context_var_tests/models/context.sql
--- a/test/integration/013_context_var_tests/models/context.sql
+++ b/test/integration/013_context_var_tests/models/context.sql
@@ -27,4 +27,6 @@ select
     '{{ run_started_at }}' as run_started_at,
     '{{ invocation_id }}'  as invocation_id,
 
-    '{{ env_var("DBT_TEST_013_ENV_VAR") }}' as env_var
+    '{{ env_var("DBT_TEST_013_ENV_VAR") }}' as env_var,
+    '{{ env_var("DBT_ENV_SECRET_013_SECRET") }}' as env_var_secret,
+    '{{ env_var("DBT_TEST_013_NOT_SECRET") }}' as env_var_not_secret
diff --git a/test/integration/013_context_var_tests/test_context_vars.py b/test/integration/013_context_var_tests/test_context_vars.py
--- a/test/integration/013_context_var_tests/test_context_vars.py
+++ b/test/integration/013_context_var_tests/test_context_vars.py
@@ -1,3 +1,4 @@
+from dbt.logger import SECRET_ENV_PREFIX
 from test.integration.base import DBTIntegrationTest, use_profile
 
 import os
@@ -15,6 +16,8 @@ def setUp(self):
 
         os.environ["DBT_TEST_013_USER"] = "root"
         os.environ["DBT_TEST_013_PASS"] = "password"
+        os.environ[SECRET_ENV_PREFIX + "013_SECRET"] = "secret_variable"
+        os.environ["DBT_TEST_013_NOT_SECRET"] = "regular_variable"
 
         self.fields = [
             'this',
@@ -138,6 +141,12 @@ def test_postgres_env_vars_prod(self):
         self.assertEqual(ctx['target.pass'], '')
         self.assertEqual(ctx['env_var'], '1')
 
+    @use_profile('postgres')
+    def test_postgres_env_vars_secrets(self):
+        _, log_output = self.run_dbt_and_capture(['--debug', 'run', '--target', 'prod'])
+
+        self.assertFalse("secret_variable" in log_output)
+        self.assertTrue("regular_variable" in log_output)
 
 class TestEmitWarning(DBTIntegrationTest):
     @property