Repository: dbt-labs/dbt-core

Issue:
[CT-2721] [Feature] support specify partial parsing file location
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

All inputs for `dbt-core` should be configurable independently, with sane default when necessary. This makes running dbt-core on cloud easier to reason about. Because you can clearly configure all of your input for a `dbt` invocation.

Currently `dbt-core` will try to load up `partial_parse.msgpack` from the target dir([link](https://github.com/dbt-labs/dbt-core/blob/39e0c22353d2230fe1bc8b4852bbb0afa27fd2c0/core/dbt/parser/manifest.py#L949)). There's no way to configure it without configure target dir to the same place.

We should provide a parameter(for all commands that does parsing) to being able to specify location to load `partial_parse.msgpack`, and if not specified `dbt-core` will fall back to use the current target dir.


### Describe alternatives you've considered

Not do anything.

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_


Discussion:
@ChenyuLInx Given how internal this is to `dbt-core`, I'd love if there were a way to support configuring this without another flag/parameter. Ideas:
- Support passing this into `dbtRunner` &rarr; `@requires.manifest` programmatically
- Support this as an env var (without associated CLI flag) - similar to [configurable location for hub / package registry](https://github.com/dbt-labs/dbt-core/blob/de65697ff9e0746e0b7dec39880381decebd3a85/core/dbt/clients/registry.py#L20-L23)
@jtcohen6  How does create [hidden parameter](https://github.com/dbt-labs/dbt-core/blob/4cdeff11cdeece7c6c28edb682477fd674cb9e04/core/dbt/cli/params.py#L272-L279) like what we do for `deprecated` params sounds?
User don't get it from the help text(they could use it if they really want). And it follows our existing patterns.
@ChenyuLInx Adding this as a parameter with `hidden=true` feels like a reasonable compromise. Let's do it

Resolving patch:
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -138,6 +138,7 @@ def invoke(self, args: List[str], **kwargs) -> dbtRunnerResult:
 @p.log_path
 @p.macro_debugging
 @p.partial_parse
+@p.partial_parse_file_path
 @p.populate_cache
 @p.print
 @p.printer_width
diff --git a/core/dbt/cli/params.py b/core/dbt/cli/params.py
--- a/core/dbt/cli/params.py
+++ b/core/dbt/cli/params.py
@@ -239,6 +239,15 @@
     default=True,
 )
 
+partial_parse_file_path = click.option(
+    "--partial-parse-file-path",
+    envvar="DBT_PARTIAL_PARSE_FILE_PATH",
+    help="Internal flag for path to partial_parse.manifest file.",
+    default=None,
+    hidden=True,
+    type=click.Path(exists=True, dir_okay=False, resolve_path=True),
+)
+
 populate_cache = click.option(
     "--populate-cache/--no-populate-cache",
     envvar="DBT_POPULATE_CACHE",
diff --git a/core/dbt/parser/manifest.py b/core/dbt/parser/manifest.py
--- a/core/dbt/parser/manifest.py
+++ b/core/dbt/parser/manifest.py
@@ -840,10 +840,13 @@ def skip_partial_parsing_because_of_macros(self):
         return False
 
     def read_manifest_for_partial_parse(self) -> Optional[Manifest]:
-        if not get_flags().PARTIAL_PARSE:
+        flags = get_flags()
+        if not flags.PARTIAL_PARSE:
             fire_event(PartialParsingNotEnabled())
             return None
-        path = os.path.join(self.root_project.project_target_path, PARTIAL_PARSE_FILE_NAME)
+        path = flags.PARTIAL_PARSE_FILE_PATH or os.path.join(
+            self.root_project.project_target_path, PARTIAL_PARSE_FILE_NAME
+        )
 
         reparse_reason = None
 


Test patch:
diff --git a/tests/unit/test_parse_manifest.py b/tests/unit/test_parse_manifest.py
--- a/tests/unit/test_parse_manifest.py
+++ b/tests/unit/test_parse_manifest.py
@@ -1,12 +1,16 @@
 import unittest
 from unittest import mock
-from unittest.mock import patch
+from unittest.mock import patch, MagicMock
+from argparse import Namespace
 
 from .utils import config_from_parts_or_dicts, normalize
 
 from dbt.contracts.files import SourceFile, FileHash, FilePath
 from dbt.contracts.graph.manifest import Manifest, ManifestStateCheck
 from dbt.parser import manifest
+from dbt.parser.manifest import ManifestLoader
+from dbt.config import RuntimeConfig
+from dbt.flags import set_from_args
 
 
 class MatchingHash(FileHash):
@@ -99,3 +103,21 @@ def _new_file(self, searched, name, match):
             project_root=normalize(self.root_project_config.project_root),
         )
         return SourceFile(path=path, checksum=checksum)
+
+
+class TestPartialParse(unittest.TestCase):
+    @patch("dbt.parser.manifest.ManifestLoader.build_manifest_state_check")
+    @patch("dbt.parser.manifest.os.path.exists")
+    @patch("dbt.parser.manifest.open")
+    def test_partial_parse_file_path(self, patched_open, patched_os_exist, patched_state_check):
+        mock_project = MagicMock(RuntimeConfig)
+        mock_project.project_target_path = "mock_target_path"
+        patched_os_exist.return_value = True
+        set_from_args(Namespace(), {})
+        ManifestLoader(mock_project, {})
+        # by default we use the project_target_path
+        patched_open.assert_called_with("mock_target_path/partial_parse.msgpack", "rb")
+        set_from_args(Namespace(partial_parse_file_path="specified_partial_parse_path"), {})
+        ManifestLoader(mock_project, {})
+        # if specified in flags, we use the specified path
+        patched_open.assert_called_with("specified_partial_parse_path", "rb")