internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dbt-labs/dbt-core

Issue:
[CT-2964] Backport #7630 to 1.3 and 1.4
The security fix made for #7630 in 1.5 and 1.6 to exclude some profile fields from Jinja rendering now needs to be backported to 1.3 and 1.4.


Discussion:

Resolving patch:
diff --git a/core/dbt/config/renderer.py b/core/dbt/config/renderer.py
--- a/core/dbt/config/renderer.py
+++ b/core/dbt/config/renderer.py
@@ -181,7 +181,17 @@ def render_value(self, value: Any, keypath: Optional[Keypath] = None) -> Any:
         # First, standard Jinja rendering, with special handling for 'secret' environment variables
         # "{{ env_var('DBT_SECRET_ENV_VAR') }}" -> "$$$DBT_SECRET_START$$$DBT_SECRET_ENV_{VARIABLE_NAME}$$$DBT_SECRET_END$$$"
         # This prevents Jinja manipulation of secrets via macros/filters that might leak partial/modified values in logs
-        rendered = super().render_value(value, keypath)
+
+        try:
+            rendered = super().render_value(value, keypath)
+        except Exception as ex:
+            if keypath and "password" in keypath:
+                # Passwords sometimes contain jinja-esque characters, but we
+                # don't want to render them if they aren't valid jinja.
+                rendered = value
+            else:
+                raise ex
+
         # Now, detect instances of the placeholder value ($$$DBT_SECRET_START...DBT_SECRET_END$$$)
         # and replace them with the actual secret value
         if SECRET_ENV_PREFIX in str(rendered):


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3138] [Bug] Error: Invalid value for '--warn-error-options'
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

When installing `dbt-core` into an environment where an older version of click is installed, `dbt-core` will throw a vague error `Error: Invalid value for '--warn-error-options': Cannot load YAML from type <class 'dbt.helper_types.WarnErrorOptions'>`. It seems `dbt-core` is pinning `click<9` which allows it to use older versions of click that it seems no longer compatible with. I think `dbt-core` should be pinning a minimum version requirement of click that plays well with the library.

### Expected Behavior

No error.

### Steps To Reproduce

Install `click==8.0.0`
Install the latest `dbt-core`
Attempt to use the dbt cli w/ a command `dbt ls` or something

### Relevant log output

```shell
Error: Invalid value for '--warn-error-options': Cannot load YAML from type <class 'dbt.helper_types.WarnErrorOptions'>
```


### Environment

```markdown
- OS: 
- Python: 3.9.1
- dbt-core: 1.6.3

pip freeze output
agate==1.7.1
asn1crypto==1.5.1
attrs==23.1.0
Babel==2.12.1
certifi==2023.7.22
cffi==1.15.1
charset-normalizer==3.2.0
click==8.0.0
colorama==0.4.6
cryptography==41.0.4
dbt-core==1.6.3
dbt-extractor==0.4.1
dbt-semantic-interfaces==0.2.0
filelock==3.12.4
hologram==0.0.16
idna==3.4
importlib-metadata==6.8.0
isodate==0.6.1
jaraco.classes==3.3.0
Jinja2==3.1.2
jsonschema==3.2.0
keyring==24.2.0
leather==0.3.4
Logbook==1.5.3
MarkupSafe==2.1.3
mashumaro==3.8.1
minimal-snowplow-tracker==0.0.2
more-itertools==8.14.0
msgpack==1.0.5
networkx==3.1
oscrypto==1.3.0
packaging==23.1
parsedatetime==2.6
pathspec==0.11.2
platformdirs==3.8.1
protobuf==4.24.3
pycparser==2.21
pycryptodomex==3.19.0
pydantic==1.10.12
PyJWT==2.8.0
pyOpenSSL==23.2.0
pyrsistent==0.19.3
python-dateutil==2.8.2
python-slugify==8.0.1
pytimeparse==1.1.8
pytz==2023.3.post1
PyYAML==6.0.1
requests==2.31.0
six==1.16.0
snowflake-connector-python==3.2.0
sortedcontainers==2.4.0
sqlparse==0.4.4
text-unidecode==1.3
tomlkit==0.12.1
typing_extensions==4.8.0
urllib3==1.26.16
zipp==3.17.0
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

_No response_


Discussion:
Thanks for raising this @dylan-murray !

Do you happen to know the minimum version of `click` that works?
The minimum version it appears to work with is `click==8.0.2`.

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -59,7 +59,7 @@
         # ----
         # dbt-core uses these packages in standard ways. Pin to the major version, and check compatibility
         # with major versions in each new minor version of dbt-core.
-        "click<9",
+        "click>=8.0.2,<9",
         "networkx>=2.3,<4",
         # ----
         # These packages are major-version-0. Keep upper bounds on upcoming minor versions (which could have breaking changes)


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3172] [Regression] `tag:` selection raises error if project has semantic models defined
### Is this a regression in a recent version of dbt-core?

- [X] I believe this is a regression in dbt-core functionality
- [X] I have searched the existing issues, and I could not find an existing issue for this regression

### Current Behavior

`tag:` selection raises an error if the project has `semantic_models` defined

### Expected/Previous Behavior

`tag:` selection should work

### Steps To Reproduce

1. Create a project with semantic models
2. Use `tag:` selection (even if selecting other resources)
3. See this error:
```
    if any(fnmatch(tag, selector) for tag in real_node.tags):
AttributeError: 'SemanticModel' object has no attribute 'tags'
```

### Relevant log output

_No response_

### Environment

```markdown
- OS:
- Python:
- dbt (working version):
- dbt (regression version):
```


### Which database adapter are you using with dbt?

_No response_

### Additional Context

Cause: https://github.com/dbt-labs/dbt-core/pull/8589
Backport PR: https://github.com/dbt-labs/dbt-core/pull/8724

Need tests for `tag:` selection with semantic models.


Discussion:

Resolving patch:
diff --git a/core/dbt/graph/selector_methods.py b/core/dbt/graph/selector_methods.py
--- a/core/dbt/graph/selector_methods.py
+++ b/core/dbt/graph/selector_methods.py
@@ -234,7 +234,9 @@ class TagSelectorMethod(SelectorMethod):
     def search(self, included_nodes: Set[UniqueId], selector: str) -> Iterator[UniqueId]:
         """yields nodes from included that have the specified tag"""
         for node, real_node in self.all_nodes(included_nodes):
-            if any(fnmatch(tag, selector) for tag in real_node.tags):
+            if hasattr(real_node, "tags") and any(
+                fnmatch(tag, selector) for tag in real_node.tags
+            ):
                 yield node
 
 


Test patch:
diff --git a/tests/unit/test_graph_selector_methods.py b/tests/unit/test_graph_selector_methods.py
--- a/tests/unit/test_graph_selector_methods.py
+++ b/tests/unit/test_graph_selector_methods.py
@@ -1295,6 +1295,21 @@ def test_select_semantic_model(manifest):
     assert search_manifest_using_method(manifest, method, "*omer") == {"customer"}
 
 
+def test_select_semantic_model_by_tag(manifest):
+    semantic_model = make_semantic_model(
+        "pkg",
+        "customer",
+        model="customers",
+        path="_semantic_models.yml",
+    )
+    manifest.semantic_models[semantic_model.unique_id] = semantic_model
+    methods = MethodManager(manifest, None)
+    method = methods.get_method("tag", [])
+    assert isinstance(method, TagSelectorMethod)
+    assert method.arguments == []
+    search_manifest_using_method(manifest, method, "any_tag")
+
+
 @pytest.fixture
 def previous_state(manifest):
     writable = copy.deepcopy(manifest).writable_manifest()
Repository: dbt-labs/dbt-core

Issue:
[CT-3206] Add `warning_tag` to `UnversionedBreakingChange`
_follow-up to https://github.com/dbt-labs/dbt-core/issues/8384_

I would like `UnversionedBreakingChange` to be flagged as a "warning" in CI:

<img width="971" alt="Screenshot 2023-10-11 at 16 34 46" src="https://github.com/dbt-labs/dbt-core/assets/13897643/10a3a55c-c376-4952-9b61-2df471341a7b">

Right now, that should be as simple as adding a `warning_tag` to `UnversionedBreakingChange`

For later: We are inconsistent about the application of `warning_tag` to the formatted messages of `WarnLevel` events. Probably this should be more consistent, and maybe handled within the `EventManager` (instead of each event class individually). That's for later.


Discussion:

Resolving patch:
diff --git a/core/dbt/events/types.py b/core/dbt/events/types.py
--- a/core/dbt/events/types.py
+++ b/core/dbt/events/types.py
@@ -1239,12 +1239,14 @@ def code(self) -> str:
     def message(self) -> str:
         reasons = "\n  - ".join(self.breaking_changes)
 
-        return (
+        msg = (
             f"Breaking change to contracted, unversioned model {self.model_name} ({self.model_file_path})"
             "\nWhile comparing to previous project state, dbt detected a breaking change to an unversioned model."
             f"\n  - {reasons}\n"
         )
 
+        return warning_tag(msg)
+
 
 class WarnStateTargetEqual(WarnLevel):
     def code(self):


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3213] [Bug] nitpick: incorrect docstring
should be `BaseAdapter.capabilities()` not `BaseAdapter.has_feature()`

https://github.com/dbt-labs/dbt-core/blob/1baebb423c82a9c645e59b390fc3a69089623600/core/dbt/adapters/capability.py#L7


Discussion:

Resolving patch:
diff --git a/core/dbt/adapters/capability.py b/core/dbt/adapters/capability.py
--- a/core/dbt/adapters/capability.py
+++ b/core/dbt/adapters/capability.py
@@ -4,7 +4,7 @@
 
 
 class Capability(str, Enum):
-    """Enumeration of optional adapter features which can be probed using BaseAdapter.has_feature()"""
+    """Enumeration of optional adapter features which can be probed using BaseAdapter.capabilities()"""
 
     SchemaMetadataByRelations = "SchemaMetadataByRelations"
     """Indicates efficient support for retrieving schema metadata for a list of relations, rather than always retrieving


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3287] [Bug] Fix Filtering Issue In get_filtered_catalog()
### Current Behavior

The implementation of relation filtering in #8856 was inadvertently case-sensitive, a problem caught by the dbt-snowflake automated tests, which are now failing.

### Expected Behavior

Filtering should be case-insensitive and the dbt-snowflake tests should pass.

### Impact to Other Teams

None expected.

### Will backports be required?

Yes. This should be backported to 1.7.0 for rc2.


Discussion:

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -1179,9 +1179,9 @@ def get_filtered_catalog(
             }
 
             def in_map(row: agate.Row):
-                d = _expect_row_value("table_database", row)
-                s = _expect_row_value("table_schema", row)
-                i = _expect_row_value("table_name", row)
+                d = _expect_row_value("table_database", row).casefold()
+                s = _expect_row_value("table_schema", row).casefold()
+                i = _expect_row_value("table_name", row).casefold()
                 return (d, s, i) in relation_map
 
             catalogs = catalogs.where(in_map)


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3306] [Bug] Spark Integration Failure Due in get_filtered_catalog()
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

The integration tests for dbt-spark (in CircleCI) are failing.

https://app.circleci.com/pipelines/github/dbt-labs/dbt-spark/3169/workflows/07148ef8-c908-43f9-9f61-e624918f2b97/jobs/15742

This is due to the fact that Spark does not use a database identifier.

### Impact to Other Teams

Adapters

### Backport?

Yes. Backport to 1.7.latest




Discussion:

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -1179,9 +1179,12 @@ def get_filtered_catalog(
             }
 
             def in_map(row: agate.Row):
-                d = _expect_row_value("table_database", row).casefold()
-                s = _expect_row_value("table_schema", row).casefold()
-                i = _expect_row_value("table_name", row).casefold()
+                d = _expect_row_value("table_database", row)
+                s = _expect_row_value("table_schema", row)
+                i = _expect_row_value("table_name", row)
+                d = d.casefold() if d is not None else None
+                s = s.casefold() if s is not None else None
+                i = i.casefold() if i is not None else None
                 return (d, s, i) in relation_map
 
             catalogs = catalogs.where(in_map)


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3235] [TECH-DEBT] Remove unused `documentable` class method
### Housekeeping

- [X] I am a maintainer of dbt-core

### Short description

If [`documentable`](https://github.com/dbt-labs/dbt-core/blob/efa6339e1804e07f7f47aebccb46492fd897f7f5/core/dbt/node_types.py#L67-L78) isn't used anywhere, can we 🪓 axe it?

### Acceptance criteria

1. We assess if we want to keep this class method or not
2. If we choose to, then we actually remove it


### Impact to Other Teams

Unknown impact, but assumed that there would be no impact because it is unused.

### Will backports be required?

No backports required.

### Context

It is more clear if unused code is no longer in the code base.


Discussion:

Resolving patch:
diff --git a/core/dbt/node_types.py b/core/dbt/node_types.py
--- a/core/dbt/node_types.py
+++ b/core/dbt/node_types.py
@@ -64,19 +64,6 @@ def versioned(cls) -> List["NodeType"]:
             cls.Model,
         ]
 
-    @classmethod
-    def documentable(cls) -> List["NodeType"]:
-        return [
-            cls.Model,
-            cls.Seed,
-            cls.Snapshot,
-            cls.Source,
-            cls.Macro,
-            cls.Analysis,
-            cls.Exposure,
-            cls.Metric,
-        ]
-
     def pluralize(self) -> str:
         if self is self.Analysis:
             return "analyses"


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3336] [Test Performance] Optimize the set_up_plugin_manager() function
### Short description

Profiling shows that the set_up_plugin_manager() function takes up 30% of the runtime on the main thread during integration testing. The hot path is set_up_plugin_manager() -> from_modules() -> iter_modules(). By caching the results we need after the first time iter_modules() is called, nearly all this time can be won back.

### Acceptance criteria

- The iter_modules() function is only called once in the above call tree for each test process.
- We have documented a notable improvement in performance in GitHub or via local profile.

### Impact to Other Teams

None. They may enjoy the speedier tests, though.

### Will backports be required?

No.


Discussion:

Resolving patch:
diff --git a/core/dbt/plugins/manager.py b/core/dbt/plugins/manager.py
--- a/core/dbt/plugins/manager.py
+++ b/core/dbt/plugins/manager.py
@@ -1,6 +1,8 @@
+import functools
 import importlib
 import pkgutil
-from typing import Dict, List, Callable
+from types import ModuleType
+from typing import Dict, List, Callable, Mapping
 
 from dbt.contracts.graph.manifest import Manifest
 from dbt.exceptions import DbtRuntimeError
@@ -63,6 +65,17 @@ def get_manifest_artifacts(self, manifest: Manifest) -> PluginArtifacts:
         raise NotImplementedError(f"get_manifest_artifacts hook not implemented for {self.name}")
 
 
+@functools.lru_cache(maxsize=None)
+def _get_dbt_modules() -> Mapping[str, ModuleType]:
+    # This is an expensive function, especially in the context of testing, when
+    # it is called repeatedly, so we break it out and cache the result globally.
+    return {
+        name: importlib.import_module(name)
+        for _, name, _ in pkgutil.iter_modules()
+        if name.startswith(PluginManager.PLUGIN_MODULE_PREFIX)
+    }
+
+
 class PluginManager:
     PLUGIN_MODULE_PREFIX = "dbt_"
     PLUGIN_ATTR_NAME = "plugins"
@@ -91,11 +104,7 @@ def __init__(self, plugins: List[dbtPlugin]) -> None:
 
     @classmethod
     def from_modules(cls, project_name: str) -> "PluginManager":
-        discovered_dbt_modules = {
-            name: importlib.import_module(name)
-            for _, name, _ in pkgutil.iter_modules()
-            if name.startswith(cls.PLUGIN_MODULE_PREFIX)
-        }
+        discovered_dbt_modules = _get_dbt_modules()
 
         plugins = []
         for name, module in discovered_dbt_modules.items():


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3397] [Bug] agate pinned to 1.7.1 instead of 1.7.0
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

agate pin in setup.py is `agate ~=1.7.1` on `1.7.latest`.  Other branches are pinned correctly.

### Expected Behavior

agate pin in setup.py is `agate ~=1.7.0` 

### Steps To Reproduce

n/a

### Relevant log output

```shell
n/a
```


### Environment

```markdown
- OS:
- Python:
- dbt:
```


### Which database adapter are you using with dbt?

_No response_

### Additional Context

This is due to a bug with our version bump during releases.  Ticket to fix root cause coming.


Discussion:

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -49,7 +49,7 @@
         # ----
         # dbt-core uses these packages deeply, throughout the codebase, and there have been breaking changes in past patch releases (even though these are major-version-one).
         # Pin to the patch or minor version, and bump in each new minor version of dbt-core.
-        "agate~=1.7.1",
+        "agate~=1.7.0",
         "Jinja2~=3.1.2",
         "mashumaro[msgpack]~=3.9",
         # ----


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3524] Remove usage of dbt.adapters.factory in dbt/common



Discussion:

Resolving patch:
diff --git a/core/dbt/common/contracts/config/base.py b/core/dbt/common/contracts/config/base.py
--- a/core/dbt/common/contracts/config/base.py
+++ b/core/dbt/common/contracts/config/base.py
@@ -1,7 +1,10 @@
+# necessary for annotating constructors
+from __future__ import annotations
+
 from dataclasses import dataclass, Field
 
 from itertools import chain
-from typing import Callable, Dict, Any, List, TypeVar
+from typing import Callable, Dict, Any, List, TypeVar, Type
 
 from dbt.common.contracts.config.metadata import Metadata
 from dbt.common.exceptions import CompilationError, DbtInternalError
@@ -140,21 +143,18 @@ def _merge_dicts(cls, src: Dict[str, Any], data: Dict[str, Any]) -> Dict[str, An
             )
         return result
 
-    def update_from(self: T, data: Dict[str, Any], adapter_type: str, validate: bool = True) -> T:
+    def update_from(
+        self: T, data: Dict[str, Any], config_cls: Type[BaseConfig], validate: bool = True
+    ) -> T:
         """Given a dict of keys, update the current config from them, validate
         it, and return a new config with the updated values
         """
-        # sadly, this is a circular import
-        from dbt.adapters.factory import get_config_class_by_name
-
         dct = self.to_dict(omit_none=False)
 
-        adapter_config_cls = get_config_class_by_name(adapter_type)
-
         self_merged = self._merge_dicts(dct, data)
         dct.update(self_merged)
 
-        adapter_merged = adapter_config_cls._merge_dicts(dct, data)
+        adapter_merged = config_cls._merge_dicts(dct, data)
         dct.update(adapter_merged)
 
         # any remaining fields must be "clobber"
diff --git a/core/dbt/context/context_config.py b/core/dbt/context/context_config.py
--- a/core/dbt/context/context_config.py
+++ b/core/dbt/context/context_config.py
@@ -3,6 +3,7 @@
 from dataclasses import dataclass
 from typing import List, Iterator, Dict, Any, TypeVar, Generic, Optional
 
+from dbt.adapters.factory import get_config_class_by_name
 from dbt.config import RuntimeConfig, Project, IsFQNResource
 from dbt.contracts.graph.model_config import get_config_for
 from dbt.common.contracts.config.base import BaseConfig, _listify
@@ -199,9 +200,11 @@ def initial_result(self, resource_type: NodeType, base: bool) -> C:
     def _update_from_config(self, result: C, partial: Dict[str, Any], validate: bool = False) -> C:
         translated = self._active_project.credentials.translate_aliases(partial)
         translated = self.translate_hook_names(translated)
-        updated = result.update_from(
-            translated, self._active_project.credentials.type, validate=validate
-        )
+
+        adapter_type = self._active_project.credentials.type
+        adapter_config_cls = get_config_class_by_name(adapter_type)
+
+        updated = result.update_from(translated, adapter_config_cls, validate=validate)
         return updated
 
     def translate_hook_names(self, project_dict):


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[CT-3546] Remove restrictive `pathspec` version depencency (upper bound)
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

Currently [pathspec>=0.9,<0.12](https://github.com/dbt-labs/dbt-core/blob/main/core/setup.py#L68) which conflicts with latest version of pathspec (0.12.1). I can't find any reason specifically for <0.12. It seems to be consequence of dependebot upgrading more restrictive pathspec~=0.9 [here](https://github.com/dbt-labs/dbt-core/pull/5917) and then later [here](https://github.com/dbt-labs/dbt-core/pull/6737)

### Expected Behavior

I can't find any reason to restrict pathspec <0.12. I couldn't find any major change that can break anything related to this package in [pathspec change log](https://github.com/cpburnz/python-pathspec/blob/v0.12.1/CHANGES.rst).

### Steps To Reproduce

1. `pip install dbt-core==1.7.4 -c <(echo "pathspec==0.12.1")`

### Relevant log output

```shell
The conflict is caused by:
    dbt-core 1.7.4 depends on pathspec<0.12 and >=0.9
    The user requested (constraint) pathspec==0.12.1
```


### Environment

```markdown
- OS: NA
- Python: >3.8
- dbt: >1.2.6
```


### Which database adapter are you using with dbt?

_No response_

### Additional Context

_No response_


Discussion:

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -65,7 +65,7 @@
         # ----
         # These packages are major-version-0. Keep upper bounds on upcoming minor versions (which could have breaking changes)
         # and check compatibility / bump in each new minor version of dbt-core.
-        "pathspec>=0.9,<0.12",
+        "pathspec>=0.9,<0.13",
         "sqlparse>=0.2.3,<0.5",
         # ----
         # These are major-version-0 packages also maintained by dbt-labs.


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Add the @requires.manifest decorator to the retry command
### Housekeeping

- [X] I am a maintainer of dbt-core

### Short description

The retry command is missing the `@requires.manifest` decorator.  This was simply an oversights.

### Acceptance criteria

The retry command includes the `@requires.manifest` decorator

### Suggested Tests

None

### Impact to Other Teams

None

### Will backports be required?

tbd

### Context

_No response_


Discussion:

Resolving patch:
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -602,6 +602,7 @@ def run(ctx, **kwargs):
 @requires.profile
 @requires.project
 @requires.runtime_config
+@requires.manifest
 def retry(ctx, **kwargs):
     """Retry the nodes that failed in the previous run."""
     task = RetryTask(


Test patch: