internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dbt-labs/dbt-core

Issue:
Results in sources.json missing contextual info present in run_results.json
Establish parity between `sources.json` and `run_results.json`, especially for per-result information:
- `thread_id`
- `execution_time`
- `timing`

The relevant classes here:
https://github.com/dbt-labs/dbt/blob/ab06149c81d36889cdf9dff5b5903ae0d7be92e2/core/dbt/contracts/results.py#L91-L108

https://github.com/dbt-labs/dbt/blob/ab06149c81d36889cdf9dff5b5903ae0d7be92e2/core/dbt/contracts/results.py#L255-L301

It _looks_ like the source result classes inherit from generic node results, and then the output classes do not. So it's possible that the results themselves already include these properties, and it's just a matter of adding them into the output.


Discussion:
a question for @barryaron and @jtcohen6 -- should this information keep living in `sources.json`? Or does it make more sense to embed everything into `run_results.json`? Feels kind of funny to have a different file for this particular task.

For what it's worth, we _do_ use the sources.json file in the Source Freshness view in dbt Cloud, but I don't know that that's a great reason to preserve this behavior

Resolving patch:
diff --git a/core/dbt/contracts/results.py b/core/dbt/contracts/results.py
--- a/core/dbt/contracts/results.py
+++ b/core/dbt/contracts/results.py
@@ -285,6 +285,9 @@ class SourceFreshnessOutput(dbtClassMixin):
     status: FreshnessStatus
     criteria: FreshnessThreshold
     adapter_response: Dict[str, Any]
+    timing: List[TimingInfo]
+    thread_id: str
+    execution_time: float
 
 
 @dataclass
@@ -333,7 +336,10 @@ def process_freshness_result(
         max_loaded_at_time_ago_in_s=result.age,
         status=result.status,
         criteria=criteria,
-        adapter_response=result.adapter_response
+        adapter_response=result.adapter_response,
+        timing=result.timing,
+        thread_id=result.thread_id,
+        execution_time=result.execution_time,
     )
 
 


Test patch:
diff --git a/test/integration/042_sources_test/test_sources.py b/test/integration/042_sources_test/test_sources.py
--- a/test/integration/042_sources_test/test_sources.py
+++ b/test/integration/042_sources_test/test_sources.py
@@ -272,7 +272,21 @@ def _assert_freshness_results(self, path, state):
                     'warn_after': {'count': 10, 'period': 'hour'},
                     'error_after': {'count': 18, 'period': 'hour'},
                 },
-                'adapter_response': {}
+                'adapter_response': {},
+                'thread_id': AnyStringWith('Thread-'),
+                'execution_time': AnyFloat(),
+                'timing': [
+                    {
+                        'name': 'compile',
+                        'started_at': AnyStringWith(),
+                        'completed_at': AnyStringWith(),
+                    },
+                    {
+                        'name': 'execute',
+                        'started_at': AnyStringWith(),
+                        'completed_at': AnyStringWith(),
+                    }
+                ]
             }
         ])
Repository: dbt-labs/dbt-core

Issue:
Enable deprecation: execute-macro-release
Introduced in: v0.17.2
In the last week, raised by: 0 projects

Pretty straightforward: as the deprecation [warning message](https://github.com/dbt-labs/dbt/blob/484517416f3ec7affb22f3f1be4f19b2ef825332/core/dbt/deprecations.py#L119-L121) states, we should remove the `release` arg from the `execute_macro` method. I don't think we need to worry too much about moving `text_only_columns` up one positional argument, but it may not be a bad idea to leave a placeholder in its spot.

https://github.com/dbt-labs/dbt/blob/484517416f3ec7affb22f3f1be4f19b2ef825332/core/dbt/adapters/base/impl.py#L961-L964

Then, we can remove it from the list of active deprecations.


_Originally posted by @jtcohen6 in https://github.com/dbt-labs/dbt/issues/2798#issuecomment-919998174_


Discussion:

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -17,7 +17,6 @@
     InternalException, NotImplementedException, RuntimeException,
 )
 
-from dbt import deprecations
 from dbt.adapters.protocol import (
     AdapterConfig,
     ConnectionManagerProtocol,
@@ -933,7 +932,6 @@ def execute_macro(
         project: Optional[str] = None,
         context_override: Optional[Dict[str, Any]] = None,
         kwargs: Dict[str, Any] = None,
-        release: bool = False,
         text_only_columns: Optional[Iterable[str]] = None,
     ) -> agate.Table:
         """Look macro_name up in the manifest and execute its results.
@@ -947,10 +945,8 @@ def execute_macro(
             execution context.
         :param kwargs: An optional dict of keyword args used to pass to the
             macro.
-        :param release: Ignored.
         """
-        if release is not False:
-            deprecations.warn('execute-macro-release')
+
         if kwargs is None:
             kwargs = {}
         if context_override is None:
diff --git a/core/dbt/deprecations.py b/core/dbt/deprecations.py
--- a/core/dbt/deprecations.py
+++ b/core/dbt/deprecations.py
@@ -82,15 +82,6 @@ class ModelsKeyNonModelDeprecation(DBTDeprecation):
     '''
 
 
-class ExecuteMacrosReleaseDeprecation(DBTDeprecation):
-    _name = 'execute-macro-release'
-    _description = '''\
-    The "release" argument to execute_macro is now ignored, and will be removed
-    in a future relase of dbt. At that time, providing a `release` argument
-    will result in an error.
-    '''
-
-
 class AdapterMacroDeprecation(DBTDeprecation):
     _name = 'adapter-macro'
     _description = '''\
@@ -149,7 +140,6 @@ def warn(name, *args, **kwargs):
     DispatchPackagesDeprecation(),
     NotADictionaryDeprecation(),
     ModelsKeyNonModelDeprecation(),
-    ExecuteMacrosReleaseDeprecation(),
     AdapterMacroDeprecation(),
     PackageRedirectDeprecation()
 ]


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Generate catalog for relational, non-ephemeral nodes only
Similar to what #3716 did for schema caching, we should do for cataloging, too. If we expect a node to exist as an object in the database, add its database-schema to the set of namespaces to catalog; if it's ephemeral or otherwise non-relational, ignore it.

I think this would be as simple as updating the logic in `_get_catalog_schemas` to check for `node.is_relational and not node.is_ephemeral_model`, as we do in [`_get_cache_schemas`](https://github.com/dbt-labs/dbt/blob/05dea18b625253625c135cbf37311d0206a8cc79/core/dbt/adapters/base/impl.py#L303-L314):

https://github.com/dbt-labs/dbt/blob/05dea18b625253625c135cbf37311d0206a8cc79/core/dbt/adapters/base/impl.py#L316-L337


Discussion:

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -321,7 +321,9 @@ def _get_catalog_schemas(self, manifest: Manifest) -> SchemaSearchMap:
         """
         info_schema_name_map = SchemaSearchMap()
         nodes: Iterator[CompileResultNode] = chain(
-            manifest.nodes.values(),
+            [node for node in manifest.nodes.values() if (
+                node.is_relational and not node.is_ephemeral_model
+            )],
             manifest.sources.values(),
         )
         for node in nodes:


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Nasty error with log scrubbing in v0.21.0rc2
In the dbt Cloud IDE, prefixed env var log scrubbing is failing with:
```
'AttributeError' object has no attribute 'replace'
```

The processor code is:
https://github.com/dbt-labs/dbt/blob/571beb13d91d9e0c065082f6a6d9f0293666b782/core/dbt/logger.py#L345-L348

I have _not_ been able to reproduce this locally using the RPC server. There are half a dozen or so places in dbt-core where we raise an `AttributeError`, and half a dozen more where we try to handle one. I'm not sure if it's any of those.

That said, I think we might just be able to add handling for this, to avoid tripping this particular error, by wrapping the message in `str()` first. That's exactly what we do in the other log processor where we call `.replace()`:

https://github.com/dbt-labs/dbt/blob/571beb13d91d9e0c065082f6a6d9f0293666b782/core/dbt/logger.py#L459-L464


Discussion:

Resolving patch:
diff --git a/core/dbt/logger.py b/core/dbt/logger.py
--- a/core/dbt/logger.py
+++ b/core/dbt/logger.py
@@ -345,7 +345,7 @@ def process(self, record):
 class ScrubSecrets(logbook.Processor):
     def process(self, record):
         for secret in get_secret_env():
-            record.message = record.message.replace(secret, "*****")
+            record.message = str(record.message).replace(secret, "*****")
 
 
 logger = logbook.Logger('dbt')


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Make unique_field optional, add docstring
[Slack thread](https://getdbt.slack.com/archives/C50NEBJGG/p1633699658060100): The addition of `unique_field` in v0.21.0 raised an error for adapter plugins looking to upgrade.

- Turn this from an abstract method into a standard method (i.e. optional)
- Add a docstring within the base adapter code


Discussion:

Resolving patch:
diff --git a/core/dbt/contracts/connection.py b/core/dbt/contracts/connection.py
--- a/core/dbt/contracts/connection.py
+++ b/core/dbt/contracts/connection.py
@@ -128,10 +128,14 @@ def type(self) -> str:
             'type not implemented for base credentials class'
         )
 
-    @abc.abstractproperty
+    @property
     def unique_field(self) -> str:
+        """Hashed and included in anonymous telemetry to track adapter adoption.
+        Return the field from Credentials that can uniquely identify
+        one team/organization using this adapter
+        """
         raise NotImplementedError(
-            'type not implemented for base credentials class'
+            'unique_field not implemented for base credentials class'
         )
 
     def hashed_unique_field(self) -> str:


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Failing partial parsing test: test_postgres_pp_generic_tests
Since merging of pull request #4052 we've been seeing a failing partial parsing test:

>       self.assertEqual(len(manifest.nodes), 3)
E       AssertionError: 4 != 3

test\integration\068_partial_parsing_tests\test_partial_parsing.py:535: AssertionError


Discussion:

Resolving patch:
diff --git a/core/dbt/parser/read_files.py b/core/dbt/parser/read_files.py
--- a/core/dbt/parser/read_files.py
+++ b/core/dbt/parser/read_files.py
@@ -1,3 +1,4 @@
+import pathlib
 from dbt.clients.system import load_file_contents
 from dbt.contracts.files import (
     FilePath, ParseFileType, SourceFile, FileHash, AnySourceFile, SchemaSourceFile
@@ -95,10 +96,11 @@ def get_source_files(project, paths, extension, parse_file_type, saved_files):
             fb_list.append(load_seed_source_file(fp, project.project_name))
         # singular tests live in /tests but only generic tests live
         # in /tests/generic so we want to skip those
-        elif (parse_file_type == ParseFileType.SingularTest and
-              'generic/' in fp.relative_path):
-            continue
         else:
+            if parse_file_type == ParseFileType.SingularTest:
+                path = pathlib.Path(fp.relative_path)
+                if path.parts[0] == 'generic':
+                    continue
             file = load_source_file(fp, parse_file_type, project.project_name, saved_files)
             # only append the list if it has contents. added to fix #3568
             if file:


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Failing partial parsing test: test_postgres_pp_generic_tests
Since merging of pull request #4052 we've been seeing a failing partial parsing test:

>       self.assertEqual(len(manifest.nodes), 3)
E       AssertionError: 4 != 3

test\integration\068_partial_parsing_tests\test_partial_parsing.py:535: AssertionError


Discussion:

Resolving patch:
diff --git a/core/dbt/config/project.py b/core/dbt/config/project.py
--- a/core/dbt/config/project.py
+++ b/core/dbt/config/project.py
@@ -564,6 +564,13 @@ def all_source_paths(self) -> List[str]:
             self.analysis_paths, self.macro_paths
         )
 
+    @property
+    def generic_test_paths(self):
+        generic_test_paths = []
+        for test_path in self.test_paths:
+            generic_test_paths.append(os.path.join(test_path, 'generic'))
+        return generic_test_paths
+
     def __str__(self):
         cfg = self.to_project_config(with_packages=True)
         return str(cfg)
diff --git a/core/dbt/parser/read_files.py b/core/dbt/parser/read_files.py
--- a/core/dbt/parser/read_files.py
+++ b/core/dbt/parser/read_files.py
@@ -149,8 +149,7 @@ def read_files(project, files, parser_files, saved_files):
 
     # all generic tests within /tests must be nested under a /generic subfolder
     project_files['GenericTestParser'] = read_files_for_parser(
-        project, files, ["{}{}".format(test_path, '/generic') for test_path in project.test_paths],
-        '.sql', ParseFileType.GenericTest, saved_files
+        project, files, project.generic_test_paths, '.sql', ParseFileType.GenericTest, saved_files
     )
 
     project_files['SeedParser'] = read_files_for_parser(


Test patch:
diff --git a/test/integration/068_partial_parsing_tests/test_partial_parsing.py b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
--- a/test/integration/068_partial_parsing_tests/test_partial_parsing.py
+++ b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
@@ -41,7 +41,7 @@ def setup_directories(self):
         # delete files in this directory without tests interfering with each other.
         os.mkdir(os.path.join(self.test_root_dir, 'models'))
         os.mkdir(os.path.join(self.test_root_dir, 'tests'))
-        os.mkdir(os.path.join(self.test_root_dir, 'tests/generic'))
+        os.mkdir(os.path.join(self.test_root_dir, 'tests', 'generic'))
         os.mkdir(os.path.join(self.test_root_dir, 'seeds'))
         os.mkdir(os.path.join(self.test_root_dir, 'macros'))
         os.mkdir(os.path.join(self.test_root_dir, 'analyses'))
Repository: dbt-labs/dbt-core

Issue:
Prefixed env vars not scrubbed from RPCException message
The env var value is being scrubbed from the logs part of the [`RPCException`](https://github.com/dbt-labs/dbt/blob/92a0930634b5ee3bb6592b00ef5a28fd004f2d15/core/dbt/rpc/error.py#L8), but not from the message portion, which is just the raw (un-scrubbed) error message that hasn't been passed through our log processor.

It's not common that users see raw exceptions outside the context of console/debug logging, but exceptions from the RPC server can be returned/exposed without stepping through the centralized log processor.

Options:
- Call our scrubbing method on `message` when instantiating the `RPCException` class
- Call the scrubbing method within the [`dbt_error`](https://github.com/dbt-labs/dbt/blob/92a0930634b5ee3bb6592b00ef5a28fd004f2d15/core/dbt/rpc/error.py#L84-L87) method, since that's most likely where the exception is being instantiated

Depending on the timing with #3755, we're going to need to merge/backport this one into `0.21.latest` (I'd like to include in v0.21.1), and also pull into the `dbt-rpc` repo.


Discussion:
I'm going to leave this in this repo, because we need to fix it in v0.21.1, for which the rpc code is still located here.

We should also make the same fix over in [`dbt-rpc`](https://github.com/dbt-labs/dbt-rpc)
@jtcohen6 tips on triggering this error?  I made my config schema a env_var `DBT_ENV_SECRET_SCHEMA` that is invalid and it triggers an error and my secret gets logged in the traceback as a CompilationError.  That also seems like a problem but not _this_ problem.  [Here](https://github.com/emmyoop/basic-dbt/tree/RPCServer) is the project I'm testing with. I run `dbt rpc` and then run a python script within my jaffle_shop to send requests.

Resolving patch:
diff --git a/core/dbt/exceptions.py b/core/dbt/exceptions.py
--- a/core/dbt/exceptions.py
+++ b/core/dbt/exceptions.py
@@ -3,6 +3,7 @@
 from typing import NoReturn, Optional, Mapping, Any
 
 from dbt.logger import GLOBAL_LOGGER as logger
+from dbt.logger import get_secret_env
 from dbt.node_types import NodeType
 from dbt import flags
 from dbt.ui import line_wrap_message
@@ -390,6 +391,8 @@ def __init__(self, cwd, cmd, message='Error running command'):
         super().__init__(message)
         self.cwd = cwd
         self.cmd = cmd
+        for secret in get_secret_env():
+            self.cmd = str(self.cmd).replace(secret, "*****")
         self.args = (cwd, cmd, message)
 
     def __str__(self):


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Wasted time spent in `incorporate_indirect_nodes` during node selection
While reviewing #4104, I noticed a lot of wasted time spent in `incorporate_indirect_nodes`.

Here's 3s spent at runtime in the performance testing project:
<img width="1393" alt="Screenshot 2021-11-05 at 08 38 36" src="https://user-images.githubusercontent.com/13897643/140476038-a882dcab-9862-41f0-82b4-925a42e1fc48.png">

This method is frequently unnecessary, _especially_ with the change to the default indirect selection mode in that PR. This method only takes effect when there are nodes selected "indirectly" but not "directly," and we want to incorporate them on the basis of their parents' direct selection.

This one is completely on me: I added this way back in #3235, a change we're mostly turning off by default in v1. I think it should only take a few lines to skip this method when it isn't needed.


Discussion:

Resolving patch:
diff --git a/core/dbt/graph/selector.py b/core/dbt/graph/selector.py
--- a/core/dbt/graph/selector.py
+++ b/core/dbt/graph/selector.py
@@ -255,6 +255,10 @@ def incorporate_indirect_nodes(
         # Check tests previously selected indirectly to see if ALL their
         # parents are now present.
 
+        # performance: if identical, skip the processing below
+        if set(direct_nodes) == set(indirect_nodes):
+            return direct_nodes
+
         selected = set(direct_nodes)
 
         for unique_id in indirect_nodes:


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Recalculate metric `depends_on` in partial parsing
In partial parsing, when a metric's `model` is updated, the `depends_on` for metrics is not updated, even though `refs` is updated.

### reproduction steps

Create any models `model_a` + `model_b`, then create a metric like:
```yml
# models/whatever.yml
version: 2
metrics:
  - model: ref('model_a')
    name: my_metric
    label: "My Metric"
    type: count
    sql: "*"
    timestamp: my_date
    time_grains: [day]
```
Good on the first go:
```
$ dbt --partial-parse ls --resource-type metric --output json --output-keys name,refs,depends_on
...
{"name": "my_metric", "depends_on": {"macros": [], "nodes": ["model.testy.model_a"]}, "refs": [["model_a"]]}
```
Then, edit the `model` property to be `ref('model_b')` instead.
```
$ dbt --partial-parse ls --resource-type metric --output json --output-keys name,refs,depends_on
...
{"name": "my_metric", "depends_on": {"macros": [], "nodes": []}, "refs": [["model_b"]]}
```
```
$ dbt --no-partial-parse ls --resource-type metric --output json --output-keys name,refs,depends_on
{"name": "my_metric", "depends_on": {"macros": [], "nodes": ["model.testy.model_b"]}, "refs": [["model_b"]]}
```


Discussion:

Resolving patch:
diff --git a/core/dbt/contracts/graph/parsed.py b/core/dbt/contracts/graph/parsed.py
--- a/core/dbt/contracts/graph/parsed.py
+++ b/core/dbt/contracts/graph/parsed.py
@@ -800,7 +800,7 @@ class ParsedMetric(UnparsedBaseNode, HasUniqueID, HasFqn):
     sources: List[List[str]] = field(default_factory=list)
     depends_on: DependsOn = field(default_factory=DependsOn)
     refs: List[List[str]] = field(default_factory=list)
-    created_at: int = field(default_factory=lambda: int(time.time()))
+    created_at: float = field(default_factory=lambda: time.time())
 
     @property
     def depends_on_nodes(self):


Test patch:
diff --git a/test/integration/068_partial_parsing_tests/test_pp_metrics.py b/test/integration/068_partial_parsing_tests/test_pp_metrics.py
--- a/test/integration/068_partial_parsing_tests/test_pp_metrics.py
+++ b/test/integration/068_partial_parsing_tests/test_pp_metrics.py
@@ -74,6 +74,8 @@ def test_postgres_env_vars_models(self):
         self.assertEqual(metric_people.meta, expected_meta)
         self.assertEqual(metric_people.refs, [['people']])
         self.assertEqual(metric_tenure.refs, [['people']])
+        expected_depends_on_nodes = ['model.test.people']
+        self.assertEqual(metric_people.depends_on.nodes, expected_depends_on_nodes)
 
         # Change metrics yaml files
         self.copy_file('test-files/people_metrics2.yml', 'models/people_metrics.yml')
@@ -83,3 +85,6 @@ def test_postgres_env_vars_models(self):
         metric_people = manifest.metrics[metric_people_id]
         expected_meta = {'my_meta': 'replaced'}
         self.assertEqual(metric_people.meta, expected_meta)
+        expected_depends_on_nodes = ['model.test.people']
+        self.assertEqual(metric_people.depends_on.nodes, expected_depends_on_nodes)
+
Repository: dbt-labs/dbt-core

Issue:
[Refactor] Use `cls` in classmethod instead of referencing the class explicitely


### Is there an existing feature request for this?

- [X] I have searched the existing issues

### Describe the ~Feature~ refactor

In [`ManifestLoader.get_full_manifest`](https://github.com/dbt-labs/dbt-core/blob/a9b4316346e3c25b82fc59fa1f090e14b57a65db/core/dbt/parser/manifest.py#L178) the [`ManifestLoader` is initialized explicitly](https://github.com/dbt-labs/dbt-core/blob/a9b4316346e3c25b82fc59fa1f090e14b57a65db/core/dbt/parser/manifest.py#L198) instead of using the `cls` variable given by the `classmethod` decorator.

### Describe alternatives you've considered

NA, it's a refactor

### Who will this benefit?

Those who read dbt's source code

### Are you interested in contributing this feature?

Yes

### Anything else?

It's not too important, this is part of my process of understanding dbt's internals. In this way I hope to contribute back to the dbt project.


Discussion:

Resolving patch:
diff --git a/core/dbt/parser/manifest.py b/core/dbt/parser/manifest.py
--- a/core/dbt/parser/manifest.py
+++ b/core/dbt/parser/manifest.py
@@ -195,7 +195,7 @@ def get_full_manifest(
             start_load_all = time.perf_counter()
 
             projects = config.load_dependencies()
-            loader = ManifestLoader(config, projects, macro_hook)
+            loader = cls(config, projects, macro_hook)
 
             manifest = loader.load()
 


Test patch:
Repository: dbt-labs/dbt-core

Issue:
typo in UnparsedSourceDefinition.__post_serialize__
Haven't actually run into a bug from this yet but was just perusing the source code and noticed this:

https://github.com/dbt-labs/dbt/blob/85627aafcd7fcd1a2a1c61b13348ced1b52571d4/core/dbt/contracts/graph/unparsed.py#L271

Have to imagine `freshnewss` is a typo


Discussion:
@abuckenheimer I have to imagine you're right :)

Care to make that one-line fix?
Still there!

https://github.com/dbt-labs/dbt-core/blob/ba23395c8edfaf11fccfb136db920fd276b1d063/core/dbt/contracts/graph/unparsed.py#L288-L290

Going to mark this a `good first issue`

Resolving patch:
diff --git a/core/dbt/contracts/graph/unparsed.py b/core/dbt/contracts/graph/unparsed.py
--- a/core/dbt/contracts/graph/unparsed.py
+++ b/core/dbt/contracts/graph/unparsed.py
@@ -285,7 +285,7 @@ def yaml_key(self) -> 'str':
 
     def __post_serialize__(self, dct):
         dct = super().__post_serialize__(dct)
-        if 'freshnewss' not in dct and self.freshness is None:
+        if 'freshness' not in dct and self.freshness is None:
             dct['freshness'] = None
         return dct
 


Test patch: