Repository: dbt-labs/dbt-core

Issue:
Running dbt with an invalid table reference returns a cryptic `can't pickle ParserMacroCapture objects`.
Edit: originally a bug, but more of an improvement so labels might not be adequate anymore.

### Describe the bug
When running a dbt command (`dbt run` or `dbt seed`)  with an invalid reference table (no quote) as part of a model (`{{ ref(keywords) }}`  instead of  `{{ ref('keywords') }} `) for instance, the terminal returns the following:

`can't pickle ParserMacroCapture objects`

Weirdly, the behavior seems to be present in dbt 0.15.0, 0.15.1, 0.15.2 **but not** in dbt 0.14.4.


### Steps To Reproduce
In my project [here](https://github.com/GuillaumeLegoy/runeterra-tracker) the bug happens when running the command described above: `pipenv run dbt seed --profile-dir=.` from the root folder. Fixing the ref model by adding single quotes around its name fixes the issue.

### Expected behavior
A better error message when a ref is not valid, for instance: `Invalid ref able, missing single quotes around identifier`.

### Screenshots and log output
![image](https://user-images.githubusercontent.com/11860362/74107175-c6613800-4b6d-11ea-8d98-786dcc6b7c53.png)


### System information
**Which database are you using dbt with?**
- [X ] postgres
- [ ] redshift
- [ ] bigquery
- [ ] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
dbt 0.15.2
dbt 0.15.1
dbt 0.15.0
```
Again works with `dbt 0.14.4` and below.

**The operating system you're using:**
Linux PopOs

**The output of `python --version`:**
3.7.5



Discussion:
Thanks for the report @GuillaumeLegoy! I'm leaving this one as a `bug`, but I do appreciate your followup on the root cause here! If you're curious, the issue is that the `keywords` variable in your example code is a jinja2 `Undefined` object. dbt "pickles" these objects to pass them between threads, or to serialize them to disk in a `.pickle` file, for instance. Since the `Undefined` object is not pickleable, this raises an exception.

I'm think the fix here might be to make the `ParserMacroCapture` object pickable so that this particular error can be avoided (and a more helpful error can be surfaced instead).

Edit: just because it's worth saying: bad/unhandled error messages like this one are bugs IMO! 
@drewbanin I think that's actually a little bit hard - the `ParserMacroCapture` class is defined in a closure. From the [pickle docs](https://docs.python.org/3.8/library/pickle.html#what-can-be-pickled-and-unpickled)
>Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.

We'd have to change how we define the `ParserMacroCapture` and bind it to things.

A better path might be removing the idea of "capturing" macros at all, seeing as we don't use that code anyway. (I'm pretty sure if we were actually using it, we'd be _not_ including macros in the model parsing env). I'm not sure what that would break, but probably a lot!
100% agree on the error message being terrible though - we can do better than this, for sure.
@beckjake sure, I'm happy to rip out macro capturing if we can!
@beckjake is this caused by post-parse pickling for partial parsing? would using JSON (or another format) fix this error?

we should still rip out ParserMacroCapture, just curious how this connects to other specific partial parsing work that we are prioritizing
Let's spend ~1 day on this, and if we don't feel like we have a good answer, we should reconvene about the best way to solve this problem @beckjake 

Resolving patch:
diff --git a/core/dbt/clients/jinja.py b/core/dbt/clients/jinja.py
--- a/core/dbt/clients/jinja.py
+++ b/core/dbt/clients/jinja.py
@@ -311,38 +311,18 @@ def _is_dunder_name(name):
     return name.startswith('__') and name.endswith('__')
 
 
-def create_macro_capture_env(node):
-
-    class ParserMacroCapture(jinja2.Undefined):
-        """
-        This class sets up the parser to capture macros.
-        """
+def create_undefined(node=None):
+    class Undefined(jinja2.Undefined):
         def __init__(self, hint=None, obj=None, name=None, exc=None):
             super().__init__(hint=hint, name=name)
             self.node = node
             self.name = name
-            self.package_name = node.package_name
+            self.hint = hint
             # jinja uses these for safety, so we have to override them.
             # see https://github.com/pallets/jinja/blob/master/jinja2/sandbox.py#L332-L339 # noqa
             self.unsafe_callable = False
             self.alters_data = False
 
-        def __deepcopy__(self, memo):
-            path = os.path.join(self.node.root_path,
-                                self.node.original_file_path)
-
-            logger.debug(
-                'dbt encountered an undefined variable, "{}" in node {}.{} '
-                '(source path: {})'
-                .format(self.name, self.node.package_name,
-                        self.node.name, path))
-
-            # match jinja's message
-            raise_compiler_error(
-                "{!r} is undefined".format(self.name),
-                node=self.node
-            )
-
         def __getitem__(self, name):
             # Propagate the undefined value if a caller accesses this as if it
             # were a dictionary
@@ -355,15 +335,17 @@ def __getattr__(self, name):
                     .format(type(self).__name__, name)
                 )
 
-            self.package_name = self.name
             self.name = name
 
-            return self
+            return self.__class__(hint=self.hint, name=self.name)
 
         def __call__(self, *args, **kwargs):
             return self
 
-    return ParserMacroCapture
+        def __reduce__(self):
+            raise_compiler_error(f'{self.name} is undefined', node=node)
+
+    return Undefined
 
 
 def get_environment(node=None, capture_macros=False):
@@ -372,7 +354,7 @@ def get_environment(node=None, capture_macros=False):
     }
 
     if capture_macros:
-        args['undefined'] = create_macro_capture_env(node)
+        args['undefined'] = create_undefined(node)
 
     args['extensions'].append(MaterializationExtension)
     args['extensions'].append(DocumentationExtension)


Test patch:
diff --git a/test/integration/003_simple_reference_test/invalid-models/descendant.sql b/test/integration/003_simple_reference_test/invalid-models/descendant.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/003_simple_reference_test/invalid-models/descendant.sql
@@ -0,0 +1,2 @@
+-- should be ref('model')
+select * from {{ ref(model) }}
diff --git a/test/integration/003_simple_reference_test/invalid-models/model.sql b/test/integration/003_simple_reference_test/invalid-models/model.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/003_simple_reference_test/invalid-models/model.sql
@@ -0,0 +1 @@
+select 1 as id
diff --git a/test/integration/003_simple_reference_test/test_simple_reference.py b/test/integration/003_simple_reference_test/test_simple_reference.py
--- a/test/integration/003_simple_reference_test/test_simple_reference.py
+++ b/test/integration/003_simple_reference_test/test_simple_reference.py
@@ -1,5 +1,10 @@
+import os
+
+from dbt.exceptions import CompilationException
+
 from test.integration.base import DBTIntegrationTest, use_profile
 
+
 class TestSimpleReference(DBTIntegrationTest):
     @property
     def schema(self):
@@ -184,3 +189,20 @@ def test__snowflake__simple_reference_with_models_and_children(self):
 
         self.assertTrue('EPHEMERAL_SUMMARY' in created_models)
         self.assertEqual(created_models['EPHEMERAL_SUMMARY'], 'table')
+
+
+class TestErrorReference(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "simple_reference_003"
+
+    @property
+    def models(self):
+        return "invalid-models"
+
+    @use_profile('postgres')
+    def test_postgres_undefined_value(self):
+        with self.assertRaises(CompilationException) as exc:
+            self.run_dbt(['compile'])
+        path = os.path.join('invalid-models', 'descendant.sql')
+        self.assertIn(path, str(exc.exception))