internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dbt-labs/dbt-core

Issue:
[CT-3190] Pinning detective work
### Housekeeping

- [X] I am a maintainer of dbt-core

### Short description

We recently pinned `types-requests<2.31.0` because it had a dependency conflict with `urllib3` which we have pinned to `~=1.0` because of another conflict with `requests` requiring `openssl`.

This ticket is to look into if those pins are still required and clean them up if not.  

### Acceptance criteria

We have confirmed that the pins are
- required to continue to work
_or_
- not required and we have re-pinned appropriately

### Impact to Other Teams

adapters - based on the notes it seems like `urllib3` is pinned for the snowflake adapter as well so we will want to ensure changing the dependencies does not adversely affect them

### Will backports be required?

no

### Context

_No response_


Discussion:

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -61,6 +61,7 @@
         # with major versions in each new minor version of dbt-core.
         "click>=8.0.2,<9",
         "networkx>=2.3,<4",
+        "requests<3.0.0",  # should match dbt-common
         # ----
         # These packages are major-version-0. Keep upper bounds on upcoming minor versions (which could have breaking changes)
         # and check compatibility / bump in each new minor version of dbt-core.


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[Regression] External nodes not included in catalog during docs generation
### Is this a regression in a recent version of dbt-core?

- [X] I believe this is a regression in dbt-core functionality
- [X] I have searched the existing issues, and I could not find an existing issue for this regression

### Current Behavior

External nodes should be included in the catalog as a result of `dbt docs generate`. Currently, projects with external nodes are not included in the resulting catalog.json

### Expected/Previous Behavior

catalog.json includes catalog entry for external node when project contains external node.

### Steps To Reproduce

test passes 1.6.latest: https://github.com/dbt-labs/dbt-core/pull/9457
same test fails on main (and 1.7.latest, confirmed locally): https://github.com/dbt-labs/dbt-core/pull/9456

### Relevant log output

_No response_

### Environment

```markdown
- OS:
- Python:
- dbt (working version): 1.6.9
- dbt (regression version): 1.7.5, 1.8.0a2
```


### Which database adapter are you using with dbt?

_No response_

### Additional Context

_No response_


Discussion:

Resolving patch:
diff --git a/core/dbt/task/docs/generate.py b/core/dbt/task/docs/generate.py
--- a/core/dbt/task/docs/generate.py
+++ b/core/dbt/task/docs/generate.py
@@ -343,6 +343,7 @@ def get_node_selector(self) -> ResourceTypeSelector:
             manifest=self.manifest,
             previous_state=self.previous_state,
             resource_types=EXECUTABLE_NODE_TYPES,
+            include_empty_nodes=True,
         )
 
     def get_catalog_results(


Test patch:
diff --git a/tests/functional/docs/test_generate.py b/tests/functional/docs/test_generate.py
--- a/tests/functional/docs/test_generate.py
+++ b/tests/functional/docs/test_generate.py
@@ -1,5 +1,7 @@
 import pytest
+from unittest import mock
 
+from dbt.plugins.manifest import PluginNodes, ModelNodeArgs
 from dbt.tests.util import run_dbt, get_manifest
 
 sample_seed = """sample_num,sample_bool
@@ -87,6 +89,28 @@ def test_catalog_with_sources(self, project):
         assert len(catalog.sources) == 2
 
 
+class TestGenerateCatalogWithExternalNodes(TestBaseGenerate):
+    @mock.patch("dbt.plugins.get_plugin_manager")
+    def test_catalog_with_sources(self, get_plugin_manager, project):
+        project.run_sql("create table {}.external_model (id int)".format(project.test_schema))
+
+        run_dbt(["build"])
+
+        external_nodes = PluginNodes()
+        external_model_node = ModelNodeArgs(
+            name="external_model",
+            package_name="external_package",
+            identifier="external_model",
+            schema=project.test_schema,
+            database="dbt",
+        )
+        external_nodes.add_model(external_model_node)
+        get_plugin_manager.return_value.get_nodes.return_value = external_nodes
+        catalog = run_dbt(["docs", "generate"])
+
+        assert "model.external_package.external_model" in catalog.nodes
+
+
 class TestGenerateSelectSource(TestBaseGenerate):
     def test_select_source(self, project):
         run_dbt(["build"])
Repository: dbt-labs/dbt-core

Issue:
Google protobuf 5 is incompatible
### Description

We specify google.protobuf: "protobuf>=4.0.0".  Google has come out with a release candidate for version 5 which is incompatible with our code because keyword parameters to MessageToJson and MessageToDict have been renamed. There may be other incompatibilities.  We need to either change our upper limit to < 5.0.0 or support google.protobuf 5 in our code.

This was seen internally because a pipeline uses `pip install --pre`.

### Acceptance Criteria

An incompatible version of google protobuf cannot be installed.

### Will backports be required?

Yes. We need to limit the protobuf version in all versions of dbt Core that use google protobuf functions.


Discussion:
dbt-common PR: https://github.com/dbt-labs/dbt-common/pull/81

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -77,7 +77,7 @@
         # ----
         # Expect compatibility with all new versions of these packages, so lower bounds only.
         "packaging>20.9",
-        "protobuf>=4.0.0",
+        "protobuf>=4.0.0,<5",
         "pytz>=2015.7",
         "pyyaml>=6.0",
         "daff>=1.3.46",


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Standardize Dependency Definitions
### Housekeeping

- [X] I am a maintainer of dbt-core

### Short description

As OSS at the org we are standardizing on using >=,< instead of ~= for dependency definitions. Update dbt-common dependencies to follow this pattern.

### Acceptance criteria

- No uses of ~=, instead use >=,<

### Suggested Tests

None, The definitions should be equivalent.

### Impact to Other Teams

None, The definitions should be equivalent.

### Will backports be required?

No

### Context

`~=` issues; `pip` cannot resolve:

- Different "grains": `dbt-adapters` depends on `mashumaro~=3.9` and `dbt-common` depends on `mashumaro~=3.9.0`
- Different notations: `dbt-adapters` depends on `mashumaro<4.0` and `dbt-common` depends on `mashumaro~=3.9`


Discussion:

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -49,9 +49,9 @@
         # ----
         # dbt-core uses these packages deeply, throughout the codebase, and there have been breaking changes in past patch releases (even though these are major-version-one).
         # Pin to the patch or minor version, and bump in each new minor version of dbt-core.
-        "agate~=1.7.0",
+        "agate>=1.7.0,<1.8",
         "Jinja2>=3.1.3,<4",
-        "mashumaro[msgpack]~=3.9",
+        "mashumaro[msgpack]>=3.9,<4.0",
         # ----
         # Legacy: This package has not been updated since 2019, and it is unused in dbt's logging system (since v1.0)
         # The dependency here will be removed along with the removal of 'legacy logging', in a future release of dbt-core
@@ -59,8 +59,8 @@
         # ----
         # dbt-core uses these packages in standard ways. Pin to the major version, and check compatibility
         # with major versions in each new minor version of dbt-core.
-        "click>=8.0.2,<9",
-        "networkx>=2.3,<4",
+        "click>=8.0.2,<9.0",
+        "networkx>=2.3,<4.0",
         "requests<3.0.0",  # should match dbt-common
         # ----
         # These packages are major-version-0. Keep upper bounds on upcoming minor versions (which could have breaking changes)
@@ -69,11 +69,11 @@
         "sqlparse>=0.2.3,<0.5",
         # ----
         # These are major-version-0 packages also maintained by dbt-labs. Accept patches.
-        "dbt-extractor~=0.5.0",
-        "minimal-snowplow-tracker~=0.0.2",
-        "dbt-semantic-interfaces~=0.5.0a2",
-        "dbt-common~=0.1.6",
-        "dbt-adapters~=0.1.0a2",
+        "dbt-extractor>=0.5.0,<=0.6",
+        "minimal-snowplow-tracker>=0.0.2,<0.1",
+        "dbt-semantic-interfaces<1.0.0a1",
+        "dbt-common<1.0",
+        "dbt-adapters>=0.1.0a2,<1.0",
         # ----
         # Expect compatibility with all new versions of these packages, so lower bounds only.
         "packaging>20.9",


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[Regression] Retry of command with defer error: Could not find manifest in --state path
### Current Behavior

Executing a "retry" command on a command with "defer" sees error: Could not find manifest in --state path

### Expected Behavior

Retry command completes successfully

### Steps to reproduce

Do a retry of a command with a "--defer" component.

### Relevant log output

15:20:51  Encountered an error:
Runtime Error
  Could not find manifest in --state path: "<dbt.contracts.state.PreviousState object at 0x112b56170>"

### Additional context

This regression was caused by moving the "defer_state" parameter to global parameters. The existing code caused that flag to be instantiated as a "PosixPath('None')" object instead of Python None.


Discussion:

Resolving patch:
diff --git a/core/dbt/cli/flags.py b/core/dbt/cli/flags.py
--- a/core/dbt/cli/flags.py
+++ b/core/dbt/cli/flags.py
@@ -28,6 +28,7 @@
 FLAGS_DEFAULTS = {
     "INDIRECT_SELECTION": "eager",
     "TARGET_PATH": None,
+    "DEFER_STATE": None,  # necessary because of retry construction of flags
     "WARN_ERROR": None,
     # Cli args without project_flags or env var option.
     "FULL_REFRESH": False,
diff --git a/core/dbt/task/runnable.py b/core/dbt/task/runnable.py
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -677,7 +677,7 @@ def _get_previous_state(self) -> Optional[Manifest]:
             )
 
         if not state.manifest:
-            raise DbtRuntimeError(f'Could not find manifest in --state path: "{state}"')
+            raise DbtRuntimeError(f'Could not find manifest in --state path: "{state.state_path}"')
         return state.manifest
 
     def _get_deferred_manifest(self) -> Optional[Manifest]:


Test patch:
diff --git a/tests/functional/defer_state/test_defer_state.py b/tests/functional/defer_state/test_defer_state.py
--- a/tests/functional/defer_state/test_defer_state.py
+++ b/tests/functional/defer_state/test_defer_state.py
@@ -305,6 +305,9 @@ def test_defer_state_flag(self, project, unique_schema, other_schema):
             expect_pass=False,
         )
 
+        # Test that retry of a defer command works
+        run_dbt(["retry"], expect_pass=False)
+
         # this will fail because we haven't passed in --state
         with pytest.raises(
             DbtRuntimeError, match="Got a state selector method, but no comparison manifest"
diff --git a/tests/unit/test_cli_flags.py b/tests/unit/test_cli_flags.py
--- a/tests/unit/test_cli_flags.py
+++ b/tests/unit/test_cli_flags.py
@@ -394,10 +394,12 @@ def test_from_dict__build(self):
         args_dict = {
             "print": True,
             "state": "some/path",
+            "defer_state": None,
         }
         result = self._create_flags_from_dict(Command.BUILD, args_dict)
         assert result.print is True
         assert "some/path" in str(result.state)
+        assert result.defer_state is None
 
     def test_from_dict__seed(self):
         args_dict = {"use_colors": False, "exclude": ["model_three"]}
Repository: dbt-labs/dbt-core

Issue:
Move `protobuf` dependency specification section out of uncapped dependencies
### Housekeeping

- [X] I am a maintainer of dbt-core

### Short description

It [was noted](https://github.com/dbt-labs/dbt-core/issues/9759#issuecomment-2010300663) by @peter-at-work that our protobuf dependency exists in a portion of our [setup.py](https://github.com/dbt-labs/dbt-core/blob/c52a015f36a9c7d0193b6972e46e4e7102e2269b/core/setup.py#L82) that is specifically marked with `# Expect compatibility with all new versions of these packages, so lower bounds only`. We no longer expect that of `protobuf`, and have capped it, but did not move it out of that section.

### Acceptance criteria

The `protobuf` dependency has been moved out of the section in `setup.py` delineated as `# Expect compatibility with all new versions of these packages, so lower bounds only.`

### Suggested Tests

N/A

### Impact to Other Teams

None

### Will backports be required?

None

### Context

https://github.com/dbt-labs/dbt-core/issues/9759


Discussion:

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -61,6 +61,7 @@
         # with major versions in each new minor version of dbt-core.
         "click>=8.0.2,<9.0",
         "networkx>=2.3,<4.0",
+        "protobuf>=4.0.0,<5",
         "requests<3.0.0",  # should match dbt-common
         # ----
         # These packages are major-version-0. Keep upper bounds on upcoming minor versions (which could have breaking changes)
@@ -79,7 +80,6 @@
         # ----
         # Expect compatibility with all new versions of these packages, so lower bounds only.
         "packaging>20.9",
-        "protobuf>=4.0.0,<5",
         "pytz>=2015.7",
         "pyyaml>=6.0",
         "daff>=1.3.46",


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[Feature] Only create the `packages-install-path` / `dbt_packages` folder if there are packages to install
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Even if you don't have a `packages.yml` file, dbt will create a `dbt_packages` folder whenever you run a dbt command.

Rather, it should only create this directory when running `dbt deps` (and only if there is at least one package to install).

### Describe alternatives you've considered

It doesn't do much harm to have an empty `dbt_packages`, but it does feel like unnecessary cruft in the current working directory.

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

_No response_

### Anything else?

See also: https://github.com/dbt-labs/dbt-core/issues/6985

Here's an example of the extra folder in the dbt Cloud IDE:

<img width="275" alt="image" src="https://github.com/dbt-labs/dbt-core/assets/44704949/c9c1ea1f-9285-466d-b640-4be933570f5f">



Discussion:

Resolving patch:
diff --git a/core/dbt/compilation.py b/core/dbt/compilation.py
--- a/core/dbt/compilation.py
+++ b/core/dbt/compilation.py
@@ -274,7 +274,6 @@ def __init__(self, config) -> None:
 
     def initialize(self):
         make_directory(self.config.project_target_path)
-        make_directory(self.config.packages_install_path)
 
     # creates a ModelContext which is converted to
     # a dict for jinja rendering of SQL


Test patch:
diff --git a/tests/functional/list/test_list.py b/tests/functional/list/test_list.py
--- a/tests/functional/list/test_list.py
+++ b/tests/functional/list/test_list.py
@@ -37,6 +37,13 @@ def project_config_update(self):
             },
         }
 
+    def test_packages_install_path_does_not_exist(self, project):
+        run_dbt(["list"])
+        packages_install_path = "dbt_packages"
+
+        # the packages-install-path should not be created by `dbt list`
+        assert not os.path.exists(packages_install_path)
+
     def run_dbt_ls(self, args=None, expect_pass=True):
         log_manager.stdout_console()
         full_args = ["ls"]
Repository: dbt-labs/dbt-core

Issue:
Wording of TestConfigDeprecation is incorrect
### Description

The wording of this deprecation refers to 'dbt_project.yml', but this deprecation is also issued for various "tests" configs in schema.yml files, which is really confusing.

### Acceptance Criteria

We issue non-confusing messages.  There will probably be multiple places with the "tests" config, and we only want to issue one deprecation message, so specifying the file might or might not be a good idea.


Discussion:

Resolving patch:
diff --git a/core/dbt/events/types.py b/core/dbt/events/types.py
--- a/core/dbt/events/types.py
+++ b/core/dbt/events/types.py
@@ -395,7 +395,7 @@ def code(self) -> str:
     def message(self) -> str:
         description = (
             f"The `{self.deprecated_path}` config has been renamed to `{self.exp_path}`. "
-            "Please update your `dbt_project.yml` configuration to reflect this change."
+            "Please see https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax for more information."
         )
         return line_wrap_message(warning_tag(f"Deprecated functionality\n\n{description}"))
 


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[Feature] better error message when running disabled model
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

When you run a disabled model you get this error

```
21:53:02  Found 468 models, 5 analyses, 720 tests, 6 seeds, 2 operations, 149 sources, 0 exposures, 0 metrics, 954 macros, 0 groups, 0 semantic models
21:53:02  The selection criterion 'log__user_table' does not match any nodes
21:53:02  
21:53:02  Nothing to do. Try checking your model configs and model specification args
```

Would be better if this model read

```
21:53:02  Found 468 models, 5 analyses, 720 tests, 6 seeds, 2 operations, 149 sources, 0 exposures, 0 metrics, 954 macros, 0 groups, 0 semantic models
21:53:02  The selection criterion 'log__user_table' matches a disabled node
21:53:02  
21:53:02  Nothing to do. Try checking your model configs and model specification args
```

### Describe alternatives you've considered

_No response_

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_


Discussion:
Idea from @dataders -- just add the single word `enabled` to the log output:

```
21:53:02  The selection criterion 'log__user_table' does not match any enabled nodes
```

After discussing with @graciegoheen and @dataders, let's adopt this simple language.
Hi @dbeatty10, I would like to start to contribute to `dbt-core`, and this issue looks like a good opportunity to begin. Would you mind if I open a PR updating this log message?
@SamuelBFavarin here's the offending line. Check out our [`contributing.md`](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) to get started!
https://github.com/dbt-labs/dbt-core/blob/ebc22fa26c9cb26b6deef86cc0a7ceb1ee3fb642/core/dbt/events/types.py#L1102

Resolving patch:
diff --git a/core/dbt/events/types.py b/core/dbt/events/types.py
--- a/core/dbt/events/types.py
+++ b/core/dbt/events/types.py
@@ -1099,7 +1099,7 @@ def code(self) -> str:
         return "M030"
 
     def message(self) -> str:
-        return f"The selection criterion '{self.spec_raw}' does not match any nodes"
+        return f"The selection criterion '{self.spec_raw}' does not match any enabled nodes"
 
 
 class DepsLockUpdating(InfoLevel):


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Drop custom `_error_tag` and use `error_tag` provided by dbt-common
### Housekeeping

- [X] I am a maintainer of dbt-core

### Short description

In https://github.com/dbt-labs/dbt-core/pull/9886 we added a stop gap `_error_tag` to core which really should belong in dbt-common. Closing https://github.com/dbt-labs/dbt-common/issues/107 will mean it exists in dbt-common. Once dbt-common releases that feature, we should migrate to using that and drop the custom implementation in core.

### Acceptance criteria

* `_error_tag` implementation in core is delete
* things using core's `_error_tag` use dbt-common's `error_tag`

### Suggested Tests

N/A

### Impact to Other Teams

N/A

### Will backports be required?

N/A

### Context

_No response_


Discussion:

Resolving patch:
diff --git a/core/dbt/events/types.py b/core/dbt/events/types.py
--- a/core/dbt/events/types.py
+++ b/core/dbt/events/types.py
@@ -1,7 +1,7 @@
 import json
 
 from dbt.constants import MAXIMUM_SEED_SIZE_NAME, PIN_PACKAGE_URL
-from dbt_common.ui import warning_tag, line_wrap_message, green, yellow, red
+from dbt_common.ui import error_tag, warning_tag, line_wrap_message, green, yellow, red
 from dbt_common.events.base_types import EventLevel
 from dbt_common.events.format import (
     format_fancy_output_line,
@@ -11,11 +11,6 @@
 from dbt.events.base_types import WarnLevel, InfoLevel, DebugLevel, ErrorLevel, DynamicLevel
 
 
-# TODO Move this to dbt_common.ui
-def _error_tag(msg: str) -> str:
-    return f'[{red("ERROR")}]: {msg}'
-
-
 # Event codes have prefixes which follow this table
 #
 # | Code |     Description     |
@@ -430,7 +425,7 @@ def message(self) -> str:
         )
 
         if self.level == EventLevel.ERROR.value:
-            description = _error_tag(description)
+            description = error_tag(description)
         elif self.level == EventLevel.WARN.value:
             description = warning_tag(description)
 
@@ -448,7 +443,7 @@ def message(self) -> str:
             description += " Run again with `--debug` to see them all."
 
         if self.level == EventLevel.ERROR.value:
-            description = _error_tag(description)
+            description = error_tag(description)
         elif self.level == EventLevel.WARN.value:
             description = warning_tag(description)
 
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -75,7 +75,7 @@
         "minimal-snowplow-tracker>=0.0.2,<0.1",
         "dbt-semantic-interfaces>=0.5.1,<0.6",
         # Minor versions for these are expected to be backwards-compatible
-        "dbt-common<2.0",
+        "dbt-common>=1.0.1,<2.0",
         "dbt-adapters>=0.1.0a2,<2.0",
         # ----
         # Expect compatibility with all new versions of these packages, so lower bounds only.


Test patch: