Repository: dbt-labs/dbt-core

Issue:
[CT-1146] [Bug] providers.py raises unhelpful error if dispatch provides package name that doesn't exist
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

I want to preface this by saying I am not surprised that no one has run into this bug before. It takes a special kind of person to make changes to every macro reference in their package and then forget to update the package name in the dispatch line. That being said, here is the bug.

If the adapter.dispatch provides the name of a package that doesn't exist, the error handling build into providers.py doesn't account for this pattern.

Example: The namespace provided `metrics` **exists** so it doesn't go into the first path. In the second path, it looks for the namespace in `get_macro_search_order` but that can't complete because the namespace doesn't exist in the `dbt_projects` path. The end result is returning a None object.  

```
        if namespace is None:
            search_packages = [None]
        elif isinstance(namespace, str):
            search_packages = self._adapter.config.get_macro_search_order(namespace)
            if not search_packages and namespace in self._adapter.config.dependencies:
                search_packages = [self.config.project_name, namespace]
```

This None object then can't be iterated in line 167:
```
        for package_name in search_packages:
```

This returns an unhelpful error message shown in relevant log output.

#### Potential Fix?
@dbeatty10 and I were able to solve this problem by adding another path after line 154:

```
            if not search_packages and namespace in self._adapter.config.dependencies:
                search_packages = [self.config.project_name, namespace]
            else:
                search_packages = [None]
```

This allows the search_packages variable to be iterated over with and helps raise a much more helpful error message around what is going wrong. 

### Expected Behavior

This error message

```
dbt.exceptions.CompilationException: Compilation Error in model develop_metric (models/metric_testing_models/develop_metric.sql)
  In dispatch: No macro named 'develop' found
      Searched for: 'postgres__develop', 'default__develop'
  
  > in macro develop (macros/develop.sql)
  > called by model develop_metric (models/metric_testing_models/develop_metric.sql)
```

### Steps To Reproduce

Loom explaining and showing issue: https://www.loom.com/share/347a5dc783cc45f2be5c8d7eede01622

### Relevant log output

```shell
File "/Users/callummccann/repos/personal_dotfiles/poetry_environments/dbt-testing-metrics/.venv/lib/python3.10/site-packages/dbt/context/providers.py", line 171, in dispatch
    for package_name in search_packages:
TypeError: 'NoneType' object is not iterable
```


### Environment

```markdown
- OS: MacOS
- Python: 3.10.6
- dbt: 1.2.1
```


### Which database adapter are you using with dbt?

postgres

### Additional Context

Happy to submit the fix if the team thinks that the proposed fix above matches what the team thinks it is good code!


Discussion:
I think the fix is fine. Please submit a PR. Creating the test will probably be a lot more work than the fix :)
I prefer the fix like this (which I didn't actually try out 😅):

```python
        search_packages = None

        if namespace is None:
            search_packages = [None]
        elif isinstance(namespace, str):
            search_packages = self._adapter.config.get_macro_search_order(namespace)
            if not search_packages and namespace in self._adapter.config.dependencies:
                search_packages = [self.config.project_name, namespace]
        else:
            raise CompilationException(
                f"In adapter.dispatch, got a {type(namespace)} macro_namespace argument "
                f'("{macro_namespace}"), but macro_namespace should be None or a string.'
            )

        if search_packages is None:
            search_packages = [None]
```

The first line ensures that there is an initial default value for `search_packages`, and the last two lines makes sure it has the shape expected by the logic that uses it downstream.

Even better:
- wrap all of this up in a function named `_get_search_packages` (which would make the `dispatch` function shorter and easier to read and is probably more easy to unit test too):
```python
    def _get_search_packages(self, namespace: Optional[str] = None):
        ...
        return search_packages
```

And then call it:
```python
    search_packages = self._get_search_packages(macro_namespace)
```

Then this extraneous line can probably be removed too:
```python
namespace = macro_namespace
```

Resolving patch:
diff --git a/core/dbt/context/providers.py b/core/dbt/context/providers.py
--- a/core/dbt/context/providers.py
+++ b/core/dbt/context/providers.py
@@ -133,6 +133,25 @@ def _get_adapter_macro_prefixes(self) -> List[str]:
         search_prefixes = get_adapter_type_names(self._adapter.type()) + ["default"]
         return search_prefixes
 
+    def _get_search_packages(self, namespace: Optional[str] = None) -> List[Optional[str]]:
+        search_packages: List[Optional[str]] = [None]
+
+        if namespace is None:
+            search_packages = [None]
+        elif isinstance(namespace, str):
+            macro_search_order = self._adapter.config.get_macro_search_order(namespace)
+            if macro_search_order:
+                search_packages = macro_search_order
+            elif not macro_search_order and namespace in self._adapter.config.dependencies:
+                search_packages = [self.config.project_name, namespace]
+        else:
+            raise CompilationError(
+                f"In adapter.dispatch, got a {type(namespace)} macro_namespace argument "
+                f'("{namespace}"), but macro_namespace should be None or a string.'
+            )
+
+        return search_packages
+
     def dispatch(
         self,
         macro_name: str,
@@ -154,20 +173,7 @@ def dispatch(
         if packages is not None:
             raise MacroDispatchArgError(macro_name)
 
-        namespace = macro_namespace
-
-        if namespace is None:
-            search_packages = [None]
-        elif isinstance(namespace, str):
-            search_packages = self._adapter.config.get_macro_search_order(namespace)
-            if not search_packages and namespace in self._adapter.config.dependencies:
-                search_packages = [self.config.project_name, namespace]
-        else:
-            # Not a string and not None so must be a list
-            raise CompilationError(
-                f"In adapter.dispatch, got a list macro_namespace argument "
-                f'("{macro_namespace}"), but macro_namespace should be None or a string.'
-            )
+        search_packages = self._get_search_packages(macro_namespace)
 
         attempts = []
 
@@ -191,7 +197,7 @@ def dispatch(
                     return macro
 
         searched = ", ".join(repr(a) for a in attempts)
-        msg = f"In dispatch: No macro named '{macro_name}' found\n    Searched for: {searched}"
+        msg = f"In dispatch: No macro named '{macro_name}' found within namespace: '{macro_namespace}'\n    Searched for: {searched}"
         raise CompilationError(msg)
 
 


Test patch:
diff --git a/tests/functional/macros/fixtures.py b/tests/functional/macros/fixtures.py
--- a/tests/functional/macros/fixtures.py
+++ b/tests/functional/macros/fixtures.py
@@ -129,3 +129,28 @@
     {{ adapter_macro('some_macro', arg1, arg2) }}
 {%- endmacro %}
 """
+
+macros__incorrect_dispatch = """
+{% macro cowsay() %}
+  {{ return(adapter.dispatch('cowsay', 'farm_utils')()) }}
+{%- endmacro %}
+
+{% macro default__cowsay() %}
+  'moo'
+{% endmacro %}
+"""
+
+# Note the difference between `test_utils` below and `farm_utils` above
+models__incorrect_dispatch = """
+select {{ test_utils.cowsay() }} as cowsay
+"""
+
+dbt_project__incorrect_dispatch = """
+name: 'test_utils'
+version: '1.0'
+config-version: 2
+
+profile: 'default'
+
+macro-paths: ["macros"]
+"""
diff --git a/tests/functional/macros/test_macros.py b/tests/functional/macros/test_macros.py
--- a/tests/functional/macros/test_macros.py
+++ b/tests/functional/macros/test_macros.py
@@ -10,18 +10,22 @@
     check_relations_equal,
 )
 
+from dbt.tests.fixtures.project import write_project_files
 from tests.functional.macros.fixtures import (
+    dbt_project__incorrect_dispatch,
     models__dep_macro,
     models__with_undefined_macro,
     models__local_macro,
     models__ref_macro,
     models__override_get_columns_macros,
     models__deprecated_adapter_macro_model,
+    models__incorrect_dispatch,
     macros__my_macros,
     macros__no_default_macros,
     macros__override_get_columns_macros,
     macros__package_override_get_columns_macros,
     macros__deprecated_adapter_macro,
+    macros__incorrect_dispatch,
 )
 
 
@@ -203,6 +207,43 @@ def test_overrides(self, project):
         run_dbt()
 
 
+class TestMisnamedMacroNamespace:
+    @pytest.fixture(scope="class", autouse=True)
+    def setUp(self, project_root):
+        test_utils_files = {
+            "dbt_project.yml": dbt_project__incorrect_dispatch,
+            "macros": {
+                "cowsay.sql": macros__incorrect_dispatch,
+            },
+        }
+        write_project_files(project_root, "test_utils", test_utils_files)
+
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "my_model.sql": models__incorrect_dispatch,
+        }
+
+    @pytest.fixture(scope="class")
+    def packages(self):
+        return {
+            "packages": [
+                {"local": "test_utils"},
+            ]
+        }
+
+    def test_misnamed_macro_namespace(
+        self,
+        project,
+    ):
+        run_dbt(["deps"])
+
+        with pytest.raises(dbt.exceptions.CompilationError) as exc:
+            run_dbt()
+
+        assert "In dispatch: No macro named 'cowsay' found" in str(exc.value)
+
+
 class TestAdapterMacroDeprecated:
     @pytest.fixture(scope="class")
     def models(self):