Repository: dbt-labs/dbt-core

Issue:
[CT-2236] [Feature] Remove lib.py [as much as possible]
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Remove current lib.py

### Describe alternatives you've considered

_No response_

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_


Discussion:
### In `dbt-server`, there are currently the following use cases of `lib.py`
1. create a `RuntimeConfig` to **reuse**
   - this is always paired with 2 or 3 
2. create a `Manifest` python object from an existing `manifest.msgpack` on disk, to **reuse**
   - this use case is only needed for SL worker initialization
3. create a `Manifest` python object from project file
4. serialize a `Manifest` python object, this is to save things to different locations.
   - saving things to different locations is for use case 2 listed above, and partial parsing
   - we already save a `manifest.msgpack` for partial parsing to a location [here](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/parser/manifest.py#LL551C11-L551C11)

#### In #6547 , we can add the following functions:
- return a `Manifest` python object in result
- return a `RuntimeConfig`(Doesn't have to RuntimeConfig, and we shouldn't guarantee any interface on this object.)
- return the partial parsing file write location 
This can fulfill the use case 1, 3, 4 above, 4 is fulfilled here given that is the location of a file is returned, `dbt-server` can implement logic to duplicate that file to other locations if needed.

**Then what's left are use case 1, 2 combination.** _This use case is only for SL dbt-server initialization._(@racheldaniel  correct me if I am wrong but `compile query` is the **only** operation in dbt-server that doesn't access local files at all, `preview` actually still read local files) We are a bit unclear on whether we need this function in `dbt-server` in the future, but leaning towards not needed side. And the timing for it is unclear
In the mean time, to make sure `dbt-core` 1.5 release smoothly, we have the following options
- Add a new code path in `parse` to skip read project files, just construct RuntimeConfig, read a `manifest.msgpack` into python object, and return them?
- create a new command `load`(?) hidden from CLI user, but available via dbtRunner, to do the above mentioned work
- leave function `deserialize_manifest` and `get_dbt_config` as is in `lib.py`, remove in later releases when we no longer need them.

I am leaning towards the last option here for now. It leave something in `lib.py`, but we should have a clear path on removing it if SL layer use case is no longer needed in `dbt-server`. And we can pivot to one of the first two if needed.

Thoughts? Any background/context that feels missing? @jtcohen6 @racheldaniel @aranke @stu-k 
Let's take a step back and think about what interfaces are required here?
Currently, we're thinking too much about implementations, but we should get away from that.
Let's leave these two functions in `lib.py` for v1.5:
- `deserialize_manifest`
- `get_dbt_config`

This is our way of saying, We're not determining the interfaces here until later. In the meantime, we'll continue to own the current implementation.

The acceptance criterion for this ticket is deleting all other methods from `lib.py`.

Resolving patch:
diff --git a/core/dbt/lib.py b/core/dbt/lib.py
deleted file mode 100644
--- a/core/dbt/lib.py
+++ /dev/null
@@ -1,209 +0,0 @@
-import os
-from dbt.config.project import Project
-from dbt.contracts.results import RunningStatus, collect_timing_info
-from dbt.events.functions import fire_event
-from dbt.events.types import NodeCompiling, NodeExecuting
-from dbt.exceptions import DbtRuntimeError
-from dbt.task.sql import SqlCompileRunner
-from dataclasses import dataclass
-from dbt.cli.resolvers import default_profiles_dir
-from dbt.config.runtime import load_profile, load_project
-from dbt.flags import set_from_args
-
-
-@dataclass
-class RuntimeArgs:
-    project_dir: str
-    profiles_dir: str
-    single_threaded: bool
-    profile: str
-    target: str
-
-
-class SqlCompileRunnerNoIntrospection(SqlCompileRunner):
-    def compile_and_execute(self, manifest, ctx):
-        """
-        This version of this method does not connect to the data warehouse.
-        As a result, introspective queries at compilation will not be supported
-        and will throw an error.
-
-        TODO: This is a temporary solution to more complex permissions requirements
-        for the semantic layer, and thus largely duplicates the code in the parent class
-        method. Once conditional credential usage is enabled, this should be removed.
-        """
-        result = None
-        ctx.node.update_event_status(node_status=RunningStatus.Compiling)
-        fire_event(
-            NodeCompiling(
-                node_info=ctx.node.node_info,
-            )
-        )
-        with collect_timing_info("compile") as timing_info:
-            # if we fail here, we still have a compiled node to return
-            # this has the benefit of showing a build path for the errant
-            # model
-            ctx.node = self.compile(manifest)
-        ctx.timing.append(timing_info)
-
-        # for ephemeral nodes, we only want to compile, not run
-        if not ctx.node.is_ephemeral_model:
-            ctx.node.update_event_status(node_status=RunningStatus.Executing)
-            fire_event(
-                NodeExecuting(
-                    node_info=ctx.node.node_info,
-                )
-            )
-            with collect_timing_info("execute") as timing_info:
-                result = self.run(ctx.node, manifest)
-                ctx.node = result.node
-
-            ctx.timing.append(timing_info)
-
-        return result
-
-
-def load_profile_project(project_dir, profile_name_override=None):
-    profile = load_profile(project_dir, {}, profile_name_override)
-    project = load_project(project_dir, False, profile, {})
-    return profile, project
-
-
-def get_dbt_config(project_dir, args=None, single_threaded=False):
-    from dbt.config.runtime import RuntimeConfig
-    import dbt.adapters.factory
-    import dbt.events.functions
-
-    if os.getenv("DBT_PROFILES_DIR"):
-        profiles_dir = os.getenv("DBT_PROFILES_DIR")
-    else:
-        profiles_dir = default_profiles_dir()
-
-    profile_name = getattr(args, "profile", None)
-
-    runtime_args = RuntimeArgs(
-        project_dir=project_dir,
-        profiles_dir=profiles_dir,
-        single_threaded=single_threaded,
-        profile=profile_name,
-        target=getattr(args, "target", None),
-    )
-
-    # set global flags from arguments
-    set_from_args(runtime_args, None)
-    profile, project = load_profile_project(project_dir, profile_name)
-    assert type(project) is Project
-
-    config = RuntimeConfig.from_parts(project, profile, runtime_args)
-
-    # the only thing this set_from_args does differently than
-    # the one above is that it pass runtime config over, I don't think
-    # we need that. but leaving this for now for future reference
-
-    # flags.set_from_args(runtime_args, config)
-
-    # This is idempotent, so we can call it repeatedly
-    dbt.adapters.factory.register_adapter(config)
-
-    # Make sure we have a valid invocation_id
-    dbt.events.functions.set_invocation_id()
-    dbt.events.functions.reset_metadata_vars()
-
-    return config
-
-
-def get_task_by_type(type):
-    from dbt.task.run import RunTask
-    from dbt.task.list import ListTask
-    from dbt.task.seed import SeedTask
-    from dbt.task.test import TestTask
-    from dbt.task.build import BuildTask
-    from dbt.task.snapshot import SnapshotTask
-    from dbt.task.run_operation import RunOperationTask
-
-    if type == "run":
-        return RunTask
-    elif type == "test":
-        return TestTask
-    elif type == "list":
-        return ListTask
-    elif type == "seed":
-        return SeedTask
-    elif type == "build":
-        return BuildTask
-    elif type == "snapshot":
-        return SnapshotTask
-    elif type == "run_operation":
-        return RunOperationTask
-
-    raise DbtRuntimeError("not a valid task")
-
-
-def create_task(type, args, manifest, config):
-    task = get_task_by_type(type)
-
-    def no_op(*args, **kwargs):
-        pass
-
-    task = task(args, config)
-    task.load_manifest = no_op
-    task.manifest = manifest
-    return task
-
-
-def _get_operation_node(manifest, project_path, sql, node_name):
-    from dbt.parser.manifest import process_node
-    from dbt.parser.sql import SqlBlockParser
-    import dbt.adapters.factory
-
-    config = get_dbt_config(project_path)
-    block_parser = SqlBlockParser(
-        project=config,
-        manifest=manifest,
-        root_project=config,
-    )
-
-    adapter = dbt.adapters.factory.get_adapter(config)
-    sql_node = block_parser.parse_remote(sql, node_name)
-    process_node(config, manifest, sql_node)
-    return config, sql_node, adapter
-
-
-def compile_sql(manifest, project_path, sql, node_name="query"):
-    config, node, adapter = _get_operation_node(manifest, project_path, sql, node_name)
-    allow_introspection = str(os.environ.get("__DBT_ALLOW_INTROSPECTION", "1")).lower() in (
-        "true",
-        "1",
-        "on",
-    )
-
-    if allow_introspection:
-        runner = SqlCompileRunner(config, adapter, node, 1, 1)
-    else:
-        runner = SqlCompileRunnerNoIntrospection(config, adapter, node, 1, 1)
-    return runner.safe_run(manifest)
-
-
-def execute_sql(manifest, project_path, sql, node_name="query"):
-    from dbt.task.sql import SqlExecuteRunner
-
-    config, node, adapter = _get_operation_node(manifest, project_path, sql, node_name)
-
-    runner = SqlExecuteRunner(config, adapter, node, 1, 1)
-
-    return runner.safe_run(manifest)
-
-
-def parse_to_manifest(config):
-    from dbt.parser.manifest import ManifestLoader
-
-    return ManifestLoader.get_full_manifest(config)
-
-
-def deserialize_manifest(manifest_msgpack):
-    from dbt.contracts.graph.manifest import Manifest
-
-    return Manifest.from_msgpack(manifest_msgpack)
-
-
-def serialize_manifest(manifest):
-    return manifest.to_msgpack()


Test patch:
diff --git a/test/unit/test_lib.py b/test/unit/test_lib.py
deleted file mode 100644
--- a/test/unit/test_lib.py
+++ /dev/null
@@ -1,62 +0,0 @@
-import os
-import unittest
-from unittest import mock
-from dbt.contracts.results import RunningStatus
-from dbt.lib import compile_sql
-from dbt.adapters.postgres import Plugin
-
-from test.unit.utils import clear_plugin, inject_adapter
-
-
-class MockContext:
-    def __init__(self, node):
-        self.timing = []
-        self.node = mock.MagicMock()
-        self.node._event_status = {"node_status": RunningStatus.Started}
-        self.node.is_ephemeral_model = True
-
-
-def noop_ephemeral_result(*args):
-    return None
-
-
-class TestSqlCompileRunnerNoIntrospection(unittest.TestCase):
-    def setUp(self):
-        self.manifest = {"mock": "manifest"}
-        self.adapter = Plugin.adapter({})
-        self.adapter.connection_for = mock.MagicMock()
-        self.ephemeral_result = lambda: None
-        inject_adapter(self.adapter, Plugin)
-
-    def tearDown(self):
-        clear_plugin(Plugin)
-
-    @mock.patch("dbt.lib._get_operation_node")
-    @mock.patch("dbt.task.sql.GenericSqlRunner.compile")
-    @mock.patch("dbt.task.sql.GenericSqlRunner.ephemeral_result", noop_ephemeral_result)
-    @mock.patch("dbt.task.base.ExecutionContext", MockContext)
-    def test__compile_and_execute__with_connection(self, mock_compile, mock_get_node):
-        """
-        By default, env var for allowing introspection is true, and calling this
-        method should defer to the parent method.
-        """
-        mock_get_node.return_value = ({}, None, self.adapter)
-        compile_sql(self.manifest, "some/path", None)
-
-        mock_compile.assert_called_once_with(self.manifest)
-        self.adapter.connection_for.assert_called_once()
-
-    @mock.patch("dbt.lib._get_operation_node")
-    @mock.patch("dbt.task.sql.GenericSqlRunner.compile")
-    @mock.patch("dbt.task.sql.GenericSqlRunner.ephemeral_result", noop_ephemeral_result)
-    @mock.patch("dbt.task.base.ExecutionContext", MockContext)
-    def test__compile_and_execute__without_connection(self, mock_compile, mock_get_node):
-        """
-        Ensure that compile is called but does not attempt warehouse connection
-        """
-        with mock.patch.dict(os.environ, {"__DBT_ALLOW_INTROSPECTION": "0"}):
-            mock_get_node.return_value = ({}, None, self.adapter)
-            compile_sql(self.manifest, "some/path", None)
-
-            mock_compile.assert_called_once_with(self.manifest)
-            self.adapter.connection_for.assert_not_called()