Repository: dbt-labs/dbt-core
Pull request: Cache plugin modules
resolves #9029

Local results:
- Before: make integration  1377.96s user 124.94s system 834% cpu 3:00.16 total
- After: make integration  1057.26s user 61.76s system 719% cpu 2:35.55 total


### Problem

Scanning all available python modules is slow.

### Solution

For the purposes of detecting dbt plugins, only do this once per process invocation and cache the result.

### Checklist

- [x] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me  
- [x] I have run this code in development and it appears to resolve the stated issue  
- [x] This PR includes tests, or tests are not required/relevant for this PR
- [x] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [x] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions


Problem statement:
[CT-3336] [Test Performance] Optimize the set_up_plugin_manager() function
### Short description

Profiling shows that the set_up_plugin_manager() function takes up 30% of the runtime on the main thread during integration testing. The hot path is set_up_plugin_manager() -> from_modules() -> iter_modules(). By caching the results we need after the first time iter_modules() is called, nearly all this time can be won back.

### Acceptance criteria

- The iter_modules() function is only called once in the above call tree for each test process.
- We have documented a notable improvement in performance in GitHub or via local profile.

### Impact to Other Teams

None. They may enjoy the speedier tests, though.

### Will backports be required?

No.

Hints from issue discussion:

Commit message: Add changelog entry

Patch to review:
diff --git a/.changes/unreleased/Under the Hood-20231107-191546.yaml b/.changes/unreleased/Under the Hood-20231107-191546.yaml
new file mode 100644
index 00000000000..d81c0448c63
--- /dev/null
+++ b/.changes/unreleased/Under the Hood-20231107-191546.yaml	
@@ -0,0 +1,6 @@
+kind: Under the Hood
+body: Cache dbt plugin modules to improve integration test performance
+time: 2023-11-07T19:15:46.170151-05:00
+custom:
+  Author: peterallenwebb
+  Issue: "9029"
diff --git a/core/dbt/plugins/manager.py b/core/dbt/plugins/manager.py
index cc71d5ab6bf..aa651a387ea 100644
--- a/core/dbt/plugins/manager.py
+++ b/core/dbt/plugins/manager.py
@@ -1,6 +1,8 @@
+import functools
 import importlib
 import pkgutil
-from typing import Dict, List, Callable
+from types import ModuleType
+from typing import Dict, List, Callable, Mapping
 
 from dbt.contracts.graph.manifest import Manifest
 from dbt.exceptions import DbtRuntimeError
@@ -63,6 +65,17 @@ def get_manifest_artifacts(self, manifest: Manifest) -> PluginArtifacts:
         raise NotImplementedError(f"get_manifest_artifacts hook not implemented for {self.name}")
 
 
+@functools.cache
+def _get_dbt_modules() -> Mapping[str, ModuleType]:
+    # This is an expensive function, especially in the context of testing, when
+    # it is called repeatedly, so we break it out and cache the result globally.
+    return {
+        name: importlib.import_module(name)
+        for _, name, _ in pkgutil.iter_modules()
+        if name.startswith(PluginManager.PLUGIN_MODULE_PREFIX)
+    }
+
+
 class PluginManager:
     PLUGIN_MODULE_PREFIX = "dbt_"
     PLUGIN_ATTR_NAME = "plugins"
@@ -91,11 +104,7 @@ def __init__(self, plugins: List[dbtPlugin]) -> None:
 
     @classmethod
     def from_modules(cls, project_name: str) -> "PluginManager":
-        discovered_dbt_modules = {
-            name: importlib.import_module(name)
-            for _, name, _ in pkgutil.iter_modules()
-            if name.startswith(cls.PLUGIN_MODULE_PREFIX)
-        }
+        discovered_dbt_modules = _get_dbt_modules()
 
         plugins = []
         for name, module in discovered_dbt_modules.items():


Code review comments:
diff_hunk: @@ -63,6 +65,17 @@ def get_manifest_artifacts(self, manifest: Manifest) -> PluginArtifacts:
         raise NotImplementedError(f"get_manifest_artifacts hook not implemented for {self.name}")
 
 
+@functools.cache
path: core/dbt/plugins/manager.py
text: @user2:
@author Per [docs](https://docs.python.org/3/library/functools.html#functools.cache), sounds like `@user1.cache` might just be 3.9+ syntactic sugar for `@user1.lru_cache(maxsize=None)`, which is available on earlier versions?