Repository: dbt-labs/dbt-core

Issue:
[CT-299] Decouple project configs from Task logic
### Is there an existing feature request for this?

- [X] I have searched the existing issues

### Describe the Feature

Generated from #4630 

## User Story:
In order to access project data earlier in the runtime process dbt needs to be able to generate a ProjectConfig object before starting the task.

## TL;DR
I can envision a few different ways to meet his need with varying degrees of ease and correctness
* Method 1: Duplicate all the logic needed to generate project configs and isolate it without any of the Task related specificity.
    * Pros: Fast and easy-ish.
    * Cons: Meh at best, not DRY, not easy to maintain, adds more complexity.
* Method 2: Thread together the various class methods needed to generate the project config. Only the basic flow code 
    * Pros: Almost as easy as option 1, Less duplicated code than option 1.
    * Cons: Still messy and confusing, doesn't address cli/base task mess.
* Method 3: Create the project config (and the profile config) at the outset, then pass them into the selected Task.  will require doing quite a bit of fussing with the cli and Task code.
    * Pros: Would greatly improve our Task orchestration code's readability and simplicity, Would make testing much easier. 
    * Cons: Not easy


### Describe alternatives you've considered

_No response_

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

_No response_

### Anything else?

## Research:
### Task dependency graph
<img width="1122" alt="image" src="https://user-images.githubusercontent.com/81931810/155925959-7eab3bcf-edce-4211-b92d-be3fde1a236a.png">


### Notes on Pre-task configs, tracing through the codebase to generation
#### Arguments 
_From command line_
* `main.py`:
    * `sys.argv` ->
    * `main()` -> 
    * `handle_and_check()` -> 
    * `parse_args()` ->
        * Config:  `parsed_args <class 'argparse.Namespace'>`

#### Flags
_Partially depends on `parsed_args` @_ **core/dbt/flags.py:102**
* `main.py`:
* `flags.py`: (imported)
    * Config: `flags (global module w module constants)`

#### Contents of user profile
_Read multiple times, this is the first one_
_Depends on `flags`_
* `main.py`:
    * `handle_and_check()` -> 
* `config/profile.py`:
    * `read_user_config()` -> 
        * Config: `user_config <class 'dbt.contracts.project.UserConfig'>`

<br>

>The three configs above are passed into each task. They also determine which task is called 
>@ **core/dbt/main.py:214**

### Task configs
#### Runtime / UnsetProfile configs
* `task/base.py`:
    * `ConfiguredTask.from_args()` ->
    * `move_to_nearest_project_dir()` -> (dir structure and CWD as config?!)
    * `BaseTask.from_args()` ->
* `core/dbt/task/any_given_class.py`:
    * `ConfigType.from_args()` ->
* `core/dbt/config/runtime.py`:
    * `RuntimeConfig.from_args()` ->
    * `RuntimeConfig.collect_parts()` ->
        * Config: `project <class 'dbt.config.project.Project'>`
        * Config: `profile <class 'dbt.config.profile.Profile'>`
    * `RuntimeConfig.from_parts()` ->
        * Config: `retval <class 'dbt.config.runtime.RuntimeConfig'>`
    OR
    * `UnsetProfileConfig.from_args()` ->
    * `RuntimeConfig.collect_parts()` ->
        * Config: `project <class 'dbt.config.project.Project'>`
        * Config: `profile <class 'dbt.config.profile.Profile'>`
    * `UnsetProfileConfig.from_args()` ->
        * Config: `retval <class 'dbt.config.runtime.UnsetProfileConfig'>`

>Tasks require either RuntimeConfig or UnsetProfileConfig, both of which are created from a 
>Project, Profile, and args (excepting a few that don't require config data at all, see diagram)


Discussion:

Resolving patch:
diff --git a/core/dbt/config/utils.py b/core/dbt/config/utils.py
--- a/core/dbt/config/utils.py
+++ b/core/dbt/config/utils.py
@@ -1,9 +1,15 @@
-from typing import Dict, Any
+from argparse import Namespace
+from typing import Any, Dict, Optional, Union
+from xmlrpc.client import Boolean
+from dbt.contracts.project import UserConfig
 
+import dbt.flags as flags
 from dbt.clients import yaml_helper
+from dbt.config import Profile, Project, read_user_config
+from dbt.config.renderer import DbtProjectYamlRenderer, ProfileRenderer
 from dbt.events.functions import fire_event
-from dbt.exceptions import raise_compiler_error, ValidationException
 from dbt.events.types import InvalidVarsYAML
+from dbt.exceptions import ValidationException, raise_compiler_error
 
 
 def parse_cli_vars(var_string: str) -> Dict[str, Any]:
@@ -21,3 +27,47 @@ def parse_cli_vars(var_string: str) -> Dict[str, Any]:
     except ValidationException:
         fire_event(InvalidVarsYAML())
         raise
+
+
+def get_project_config(
+    project_path: str,
+    profile_name: str,
+    args: Namespace = Namespace(),
+    cli_vars: Optional[Dict[str, Any]] = None,
+    profile: Optional[Profile] = None,
+    user_config: Optional[UserConfig] = None,
+    return_dict: Boolean = True,
+) -> Union[Project, Dict]:
+    """Returns a project config (dict or object) from a given project path and profile name.
+
+    Args:
+        project_path: Path to project
+        profile_name: Name of profile
+        args: An argparse.Namespace that represents what would have been passed in on the
+            command line (optional)
+        cli_vars: A dict of any vars that would have been passed in on the command line (optional)
+            (see parse_cli_vars above for formatting details)
+        profile: A dbt.config.profile.Profile object (optional)
+        user_config: A dbt.contracts.project.UserConfig object (optional)
+        return_dict: Return a dict if true, return the full dbt.config.project.Project object if false
+
+    Returns:
+        A full project config
+
+    """
+    # Generate a profile if not provided
+    if profile is None:
+        # Generate user_config if not provided
+        if user_config is None:
+            user_config = read_user_config(flags.PROFILES_DIR)
+        # Update flags
+        flags.set_from_args(args, user_config)
+        profile = Profile.render_from_args(args, ProfileRenderer(cli_vars), profile_name)
+    # Generate a project
+    project = Project.from_project_root(
+        project_path,
+        DbtProjectYamlRenderer(profile),
+        verify_version=bool(flags.VERSION_CHECK),
+    )
+    # Return
+    return project.to_project_config() if return_dict else project


Test patch:
diff --git a/test/integration/039_config_tests/models-ok/ok.sql b/test/integration/039_config_tests/models-ok/ok.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/039_config_tests/models-ok/ok.sql
@@ -0,0 +1 @@
+select * from {{ ref('seed') }}
\ No newline at end of file
diff --git a/test/integration/039_config_tests/project/dbt_project.yml b/test/integration/039_config_tests/project/dbt_project.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/039_config_tests/project/dbt_project.yml
@@ -0,0 +1,8 @@
+--- 
+config-version: 2
+model-paths: 
+  - models-ok
+name: test
+profile: test
+test-paths: []
+version: "1.0"
\ No newline at end of file
diff --git a/test/integration/039_config_tests/project/profiles.yml b/test/integration/039_config_tests/project/profiles.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/039_config_tests/project/profiles.yml
@@ -0,0 +1,24 @@
+--- 
+config: 
+  send_anonymous_usage_stats: false
+test: 
+  outputs: 
+    default2: 
+      dbname: dbt
+      host: localhost
+      pass: password
+      port: 5432
+      schema: test16486474128741104874_config_039
+      threads: 4
+      type: postgres
+      user: root
+    noaccess: 
+      dbname: dbt
+      host: localhost
+      pass: password
+      port: 5432
+      schema: test16486474128741104874_config_039
+      threads: 4
+      type: postgres
+      user: noaccess
+  target: default2
\ No newline at end of file
diff --git a/test/integration/039_config_tests/test_configs.py b/test/integration/039_config_tests/test_configs.py
--- a/test/integration/039_config_tests/test_configs.py
+++ b/test/integration/039_config_tests/test_configs.py
@@ -1,9 +1,11 @@
+from datetime import datetime
 import os
 import shutil
 
 from test.integration.base import DBTIntegrationTest, use_profile
 from dbt.exceptions import CompilationException
-
+from dbt.config.utils import get_project_config
+from argparse import Namespace
 
 class TestConfigs(DBTIntegrationTest):
     @property
@@ -243,6 +245,7 @@ def test_postgres_warn_unused_configuration_paths(self):
 
         self.run_dbt(['seed'])
 
+
 class TestConfigIndivTests(DBTIntegrationTest):
     @property
     def schema(self):
@@ -298,7 +301,6 @@ def test_postgres_configuring_individual_tests(self):
         self.assertEqual(results[1].status, 'fail')
 
 
-
 class TestConfigGetDefault(DBTIntegrationTest):
     @property
     def schema(self):
@@ -316,3 +318,33 @@ def test_postgres_config_with_get_default(self):
         self.assertEqual(len(results), 1)
         self.assertEqual(str(results[0].status), 'error')
         self.assertIn('column "default_value" does not exist', results[0].message)
+
+
+class TestConfigUtils(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "config_039"
+
+    @property
+    def models(self):
+        return "models"
+
+    @property
+    def project_config(self):
+        project_path = os.path.dirname(os.path.realpath(__file__)) + "/project"
+        project_config = get_project_config(
+            project_path, "test", Namespace(profiles_dir=project_path)
+        )
+
+        # thanks yaml :(
+        project_config["name"] = str(project_config["name"])
+        project_config["version"] = str(project_config["version"])
+        return project_config
+
+    @use_profile("postgres")
+    def test_postgres_get_project_config(self):
+
+        seed_result = self.run_dbt(["seed"], expect_pass=True)
+        run_result = self.run_dbt(["run"], expect_pass=True)
+        self.assertEqual(len(seed_result), 1)
+        self.assertEqual(len(run_result), 1)