internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dbt-labs/dbt-core

Issue:
Raise a better error message if dbt parses the same file twice due to misconfiguration
### Describe the feature
If a macro is addressed twice in a dbt project, then dbt will fail to compile with:
```
Compilation Error
  dbt found two macros named "<macro name>" in the project "projectname".
   To fix this error, rename or remove one of the following macros:
      - dbt_modules/package/macros/my_macro.sql
      - dbt_modules/package/macros/my_macro.sql
```

This is confusing because there is actually on a single file and the macro is _not_ defined twice, dbt just doesn't recognize that it has already parsed the file.

The root cause of an error like this is a misconfiguration like:
```yml
macro-paths: [macros, dbt_modules/package/macros]
```

Instead, dbt should raise the following error if a file is parsed twice in parsing:

```
Compilation Error
  The file <filepath> was parsed multiple times by dbt.
  This error happens when a path is duplicated in a dbt_project.yml configuration.
  Check your `dbt_project.yml` file path configurations and remove any
  duplicated paths to fix this error.
```

### Describe alternatives you've considered
We could optionally _allow_ the duplicate paths, raise a warning, and continue on. I'm amenable to that, but I don't think there's any real merit to supporting it!

### Who will this benefit?
Clearer error messages are good for everyone


Discussion:

Resolving patch:
diff --git a/core/dbt/parser/results.py b/core/dbt/parser/results.py
--- a/core/dbt/parser/results.py
+++ b/core/dbt/parser/results.py
@@ -125,7 +125,7 @@ def add_macro(self, source_file: SourceFile, macro: ParsedMacro):
             # subtract 2 for the "Compilation Error" indent
             # note that the line wrap eats newlines, so if you want newlines,
             # this is the result :(
-            msg = line_wrap_message(
+            dup_macro_msg = line_wrap_message(
                 f'''\
                 dbt found two macros named "{macro.name}" in the project
                 "{macro.package_name}".
@@ -140,6 +140,25 @@ def add_macro(self, source_file: SourceFile, macro: ParsedMacro):
                 ''',
                 subtract=2
             )
+            dup_path_msg = line_wrap_message(
+                f"""\
+                The macro {macro.name} in file {macro.original_file_path}
+                was parsed multiple times by dbt. This error happens when a
+                macro is defined multiple times in the same file, or when a
+                path is duplicated in a dbt_project.yml configuration.
+                To fix this error, check:
+
+                    - {macro.original_file_path}
+
+                    - The `macro-paths` configurations in dbt_project.yml
+                """,
+                subtract=2,
+            )
+            msg = (
+                dup_path_msg
+                if macro.original_file_path == other_path
+                else dup_macro_msg
+            )
             raise_compiler_error(msg)
 
         self.macros[macro.unique_id] = macro


Test patch:
Repository: dbt-labs/dbt-core

Issue:
git.bat not picked up on dbt deps
ref. https://github.com/fishtown-analytics/dbt/pull/1110

```
> dbt deps
Encountered an error:
Could not find command, ensure it is in the user's PATH: "git"
```

I have reasons to wrap my executables. On Windows I wrap my executables in `.bat` scripts (to get a similar effect to `.sh`). But then[ `Popen( ['git'] )`](https://github.com/fishtown-analytics/dbt/blob/42a85ac39f34b058678fd0c03ff8e8d2835d2808/core/dbt/clients/system.py#L419) doesn't work. `Popen( ['git.bat'])` works. 

So the platform-independent way to get `git` to work is to use  `shutil.which` to get the full path to the exe. `Popen([ which('git') ], )` works.


(explicitly) find the executable before running run_cmd
resolves #3035



### Checklist
 - [x] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
 - [x] I have run this code in development and it appears to resolve the stated issue
 - [x] This PR includes tests, or tests are not required/relevant for this PR
   ran test_system_client.py
 - [x] I have updated the `CHANGELOG.md` and added information about my change to the "dbt next" section.
 



Discussion:
@majidaldo Just to clarify, you're encountering an error when you run `dbt deps`?
> @majidaldo Just to clarify, you're encountering an error when you run `dbt deps`?

ah sorry. yes.
Ok! And if I understand your original message, are you saying that you're able to resolve the `dbt deps` error by changing line 420 below from `cmd` to `shutil.which(cmd)`?
https://github.com/fishtown-analytics/dbt/blob/42a85ac39f34b058678fd0c03ff8e8d2835d2808/core/dbt/clients/system.py#L419-L424
...well, depend on how you guys want to structure subprocess calls. The way you have it is that `cmd` is a list of strings (shell=False). So you'd only need to change the first element in the list: `cmd` --> `[which(cmd[0])] + list(cmd[1:])`.
FYI, with `shell=True`, your wouldn't have to use `which`. You'd have to consider security implications but I think it's safe here.
Ok, this one is a little bit beyond me. @kwigley When you have a moment, could I ask for your sense-check here?
bump. this should be straightforward.
@majidaldo Is this a fix you'd be interested in contributing? I'd welcome a PR for it, if you don't mind explaining the system/security implications of the proposed changes.


Resolving patch:
diff --git a/core/dbt/clients/system.py b/core/dbt/clients/system.py
--- a/core/dbt/clients/system.py
+++ b/core/dbt/clients/system.py
@@ -416,6 +416,9 @@ def run_cmd(
         full_env.update(env)
 
     try:
+        exe_pth = shutil.which(cmd[0])
+        if exe_pth:
+            cmd = [os.path.abspath(exe_pth)] + list(cmd[1:])
         proc = subprocess.Popen(
             cmd,
             cwd=cwd,


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Generate run_results.json even when no nodes are selected
### Describe the feature
I'd like the file `target/run_results.json` to be generated with an empty results list even when the selector selected no nodes. The current behavior is to not generate a new `run_results.json` when the selector returned no nodes.

A test run with no nodes selected is a legitimate occurence when using the `state:modified` selector.

### Describe alternatives you've considered
Parsing `dbt test` output for the string `WARNING: Nothing to do. Try checking your model configs and model specification args`. This is a bit hacky and unreliable though.

### Additional context
Using `dbt test` with the `state:modified` selector in my CI pipeline. I parse `run_results,json` to get the test outcome, but that file doesn't get generated when there are no model differences between the branch and PROD. However, the absence of `run_results.json` could also mean the test process somehow failed...

### Who will this benefit?
Anyone who depends on `run_results.json` - for example CI pipeline devs.

### Are you interested in contributing this feature?
I'll browse the source code and give it a shot, but no promises.



Discussion:

Resolving patch:
diff --git a/core/dbt/task/runnable.py b/core/dbt/task/runnable.py
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -413,7 +413,7 @@ def run(self):
         if len(self._flattened_nodes) == 0:
             logger.warning("WARNING: Nothing to do. Try checking your model "
                            "configs and model specification args")
-            return self.get_result(
+            result = self.get_result(
                 results=[],
                 generated_at=datetime.utcnow(),
                 elapsed_time=0.0,
@@ -421,9 +421,8 @@ def run(self):
         else:
             with TextOnly():
                 logger.info("")
-
-        selected_uids = frozenset(n.unique_id for n in self._flattened_nodes)
-        result = self.execute_with_hooks(selected_uids)
+            selected_uids = frozenset(n.unique_id for n in self._flattened_nodes)
+            result = self.execute_with_hooks(selected_uids)
 
         if flags.WRITE_JSON:
             self.write_manifest()


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Missing package named `packaging`
### Describe the bug
#3267 introduced a new python dependency named `packaging` (https://pypi.org/project/packaging/). We need to add it to the `install_requires` list in `core/setup.py`.

### Steps To Reproduce
This issue only affects 0.20.0b1 at the moment. Install dbt v0.20.0b1 in a bare virtualenv and run a command that utilizes the missing packaging (`dbt --version`)

```bash
$ dbt --version
Traceback (most recent call last):
  File "/Users/kwigley/.virtualenvs/tempenv-0d661869220df/bin/dbt", line 5, in <module>
    from dbt.main import main
  File "/Users/kwigley/.virtualenvs/tempenv-0d661869220df/lib/python3.9/site-packages/dbt/main.py", line 17, in <module>
    import dbt.task.deps as deps_task
  File "/Users/kwigley/.virtualenvs/tempenv-0d661869220df/lib/python3.9/site-packages/dbt/task/deps.py", line 9, in <module>
    from dbt.deps.resolver import resolve_packages
  File "/Users/kwigley/.virtualenvs/tempenv-0d661869220df/lib/python3.9/site-packages/dbt/deps/resolver.py", line 11, in <module>
    from dbt.deps.git import GitUnpinnedPackage
  File "/Users/kwigley/.virtualenvs/tempenv-0d661869220df/lib/python3.9/site-packages/dbt/deps/git.py", line 5, in <module>
    from dbt.clients import git, system
  File "/Users/kwigley/.virtualenvs/tempenv-0d661869220df/lib/python3.9/site-packages/dbt/clients/git.py", line 7, in <module>
    from packaging import version
ModuleNotFoundError: No module named 'packaging'
```

### Additional context
This wasn't caught during testing since `packaging` is installed because it is an upstream dependency of our testing tools (`tox` and `pytest`). Our current testing setup is not configured to catch this. 



Discussion:

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -61,19 +61,20 @@ def read(fname):
     install_requires=[
         'Jinja2==2.11.2',
         'PyYAML>=3.11',
-        'sqlparse>=0.2.3,<0.4',
-        'networkx>=2.3,<3',
-        'minimal-snowplow-tracker==0.0.2',
-        'colorama>=0.3.9,<0.4.5',
         'agate>=1.6,<1.6.2',
-        'isodate>=0.6,<0.7',
-        'json-rpc>=1.12,<2',
-        'werkzeug>=0.15,<2.0',
+        'colorama>=0.3.9,<0.4.5',
         'dataclasses>=0.6,<0.9;python_version<"3.7"',
         'hologram==0.0.14',
+        'isodate>=0.6,<0.7',
+        'json-rpc>=1.12,<2',
         'logbook>=1.5,<1.6',
         'mashumaro==2.0',
+        'minimal-snowplow-tracker==0.0.2',
+        'networkx>=2.3,<3',
+        'packaging~=20.9',
+        'sqlparse>=0.2.3,<0.4',
         'typing-extensions>=3.7.4,<3.8',
+        'werkzeug>=0.15,<2.0',
         # the following are all to match snowflake-connector-python
         'requests<3.0.0',
         'idna>=2.5,<3',


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Default arg values ignored in test block signature
### Describe the bug

Previously, when schema tests were macros prefixed `test_`, it was possible to set default values for arguments within the macro signature. Now that ~schema~ generic tests are special test blocks, it would be ideal to still set default values in the signature. Worst case, there is a way of working around it via `kwargs.get`, but this represents a more serious breaking change and more time-consuming upgrade path for existing packages + projects.

### Steps To Reproduce

```sql
{% test my_signature_test(model, default_arg = 'default value') %}
    select '{{ default_arg }}'
{% endtest %}

{% test my_kwargy_test(model) %}
    {% set default_arg = kwargs.get('default_arg', 'default value') %}
    select '{{ default_arg }}'
{% endtest %}
```

```
version: 2

models:
  - name: my_model
    tests:
      - my_signature_test
      - my_kwargy_test
      - my_signature_test:
          default_arg: override value
      - my_kwargy_test:
          default_arg: override value
```

When I run `dbt test`, the test output looks like (in order of the above):
```
select ''
select 'default value'
select 'override value'
select 'override value'
```

### Expected behavior

The default values for arguments in the `test` block signature should be used if no user-supplied argument is provided.

### System information

**The output of `dbt --version`:**
```
$ dbt --version
installed version: 0.20.0-b1
   latest version: 0.19.1

Your version of dbt is ahead of the latest release!

Plugins:
  - bigquery: 0.20.0b1
  - snowflake: 0.20.0b1
  - redshift: 0.20.0b1
  - postgres: 0.20.0b1
```


Discussion:

Resolving patch:
diff --git a/core/dbt/clients/jinja.py b/core/dbt/clients/jinja.py
--- a/core/dbt/clients/jinja.py
+++ b/core/dbt/clients/jinja.py
@@ -416,7 +416,6 @@ def parse(self, parser):
         test_name = parser.parse_assign_target(name_only=True).name
 
         parser.parse_signature(node)
-        node.defaults = []
         node.name = get_test_macro_name(test_name)
         node.body = parser.parse_statements(('name:endtest',),
                                             drop_needle=True)


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Avoid caching schemas for tests when store_failures is not enabled
In dbt v0.20.0, tests are now configured by default to use a schema named `dbt_test__audit`, which is useful for the "store failures" feature. On some databases, this is causing dbt to run an extra metadata query when `--store-failures` is _not_ enabled.

On Snowflake, because of how we list objects to populate the relational cache, such a metadata query often appears in the query history as a failed statement (`show terse objects in database.dbt_test__audit`), leading to user confusion.

https://github.com/dbt-labs/dbt/blob/43642956a21571e4589d3f7b9a7ad69439c6d4cc/plugins/snowflake/dbt/adapters/snowflake/impl.py#L135-L141

I view this as a low-priority cosmetic issue, with a simple fix: We could change the logic in `_get_cache_schemas` to check for _relational_ nodes (ones that map to database objects) instead of _executable_ nodes (ones dbt knows how to compile/execute):

https://github.com/dbt-labs/dbt/blob/43642956a21571e4589d3f7b9a7ad69439c6d4cc/core/dbt/adapters/base/impl.py#L304-L316

```python
            for node in manifest.nodes.values()
            if (
                node.is_relational and not node.is_ephemeral_model
            )
```

The `is_relational` method already includes logic around whether to consider tests relational:
https://github.com/dbt-labs/dbt/blob/43642956a21571e4589d3f7b9a7ad69439c6d4cc/core/dbt/contracts/graph/parsed.py#L127-L133

And this is the same method we now use inside runnable tasks, to determine which schemas need creating:
https://github.com/dbt-labs/dbt/blob/43642956a21571e4589d3f7b9a7ad69439c6d4cc/core/dbt/task/runnable.py#L460-L462

This would remove `Analysis`, `Operation`, `Documentation`, and `RPCCall` from the schema-caching list, but frankly it doesn't make much sense that they were there in the first place!


Discussion:

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -31,7 +31,6 @@
 from dbt.contracts.graph.manifest import Manifest, MacroManifest
 from dbt.contracts.graph.parsed import ParsedSeedNode
 from dbt.exceptions import warn_or_error
-from dbt.node_types import NodeType
 from dbt.logger import GLOBAL_LOGGER as logger
 from dbt.utils import filter_null_values, executor
 
@@ -310,8 +309,7 @@ def _get_cache_schemas(self, manifest: Manifest) -> Set[BaseRelation]:
             self.Relation.create_from(self.config, node).without_identifier()
             for node in manifest.nodes.values()
             if (
-                node.resource_type in NodeType.executable() and
-                not node.is_ephemeral_model
+                node.is_relational and not node.is_ephemeral_model
             )
         }
 


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Unexpected occurrences of partial parsing version mismatch
From dbt community slack:

JC, I'm now using the 0.20.1 actual release and I'm still seeing occasions of Unable to do partial parsing because of a dbt version mismatch on runs.  It's not happening on every run, but it's definitely happening with regularity.
I'm having trouble generating a reproducible case (which is why I haven't opened an issue), but it's happening (with this project anyway) at least 1 in 3 runs.  I can't see anything that seems relevant in the logs, but I wanted to call it out in case (a) you hear of this elsewhere and (b) there's anything I can do to give you more context to understand why this might be happening.

Two consecutive commands were shared, both reporting the same version, with the second one claiming a version mismatch.


Discussion:

Resolving patch:
diff --git a/core/dbt/parser/manifest.py b/core/dbt/parser/manifest.py
--- a/core/dbt/parser/manifest.py
+++ b/core/dbt/parser/manifest.py
@@ -478,6 +478,12 @@ def write_manifest_for_partial_parse(self):
         path = os.path.join(self.root_project.target_path,
                             PARTIAL_PARSE_FILE_NAME)
         try:
+            # This shouldn't be necessary, but we have gotten bug reports (#3757) of the
+            # saved manifest not matching the code version.
+            if self.manifest.metadata.dbt_version != __version__:
+                logger.debug("Manifest metadata did not contain correct version. "
+                             f"Contained '{self.manifest.metadata.dbt_version}' instead.")
+                self.manifest.metadata.dbt_version = __version__
             manifest_msgpack = self.manifest.to_msgpack()
             make_directory(os.path.dirname(path))
             with open(path, 'wb') as fp:
@@ -493,7 +499,10 @@ def is_partial_parsable(self, manifest: Manifest) -> Tuple[bool, Optional[str]]:
         reparse_reason = None
 
         if manifest.metadata.dbt_version != __version__:
-            logger.info("Unable to do partial parsing because of a dbt version mismatch")
+            # #3757 log both versions because of reports of invalid cases of mismatch.
+            logger.info("Unable to do partial parsing because of a dbt version mismatch. "
+                        f"Saved manifest version: {manifest.metadata.dbt_version}. "
+                        f"Current version: {__version__}.")
             # If the version is wrong, the other checks might not work
             return False, ReparseReason.version_mismatch
         if self.manifest.state_check.vars_hash != manifest.state_check.vars_hash:


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Update log levels for partial parsing
When we turn on partial parsing by default (https://github.com/dbt-labs/dbt/issues/3867), I think we should update log levels for partial parsing messages so that:
- When partial parsing works, those messages (`Partial parsing enabled`) are `debug` level
- When partial parsing does not work, keep those messages at `info` level
- Reword the "cold parse" message to make it clearer what's going on: `Unable to do partial parsing because target/partial_parse.msgpack not found` &rarr; `Partial parse save file not found. Starting full parse` (or something like that), to avoid user confusion [e.g. Slack thread](https://getdbt.slack.com/archives/C2JRRQDTL/p1626289854180300)


Discussion:

Resolving patch:
diff --git a/core/dbt/parser/manifest.py b/core/dbt/parser/manifest.py
--- a/core/dbt/parser/manifest.py
+++ b/core/dbt/parser/manifest.py
@@ -265,7 +265,7 @@ def load(self):
             self.manifest._parsing_info = ParsingInfo()
 
         if skip_parsing:
-            logger.info("Partial parsing enabled, no changes found, skipping parsing")
+            logger.debug("Partial parsing enabled, no changes found, skipping parsing")
         else:
             # Load Macros
             # We need to parse the macros first, so they're resolvable when
@@ -577,7 +577,7 @@ def read_manifest_for_partial_parse(self) -> Optional[Manifest]:
                 )
                 reparse_reason = ReparseReason.load_file_failure
         else:
-            logger.info(f"Unable to do partial parsing because {path} not found")
+            logger.info("Partial parse save file not found. Starting full parse.")
             reparse_reason = ReparseReason.file_not_found
 
         # this event is only fired if a full reparse is needed
diff --git a/core/dbt/parser/partial.py b/core/dbt/parser/partial.py
--- a/core/dbt/parser/partial.py
+++ b/core/dbt/parser/partial.py
@@ -105,10 +105,10 @@ def build_file_diff(self):
         }
         if changed_or_deleted_macro_file:
             self.macro_child_map = self.saved_manifest.build_macro_child_map()
-        logger.info(f"Partial parsing enabled: "
-                    f"{len(deleted) + len(deleted_schema_files)} files deleted, "
-                    f"{len(added)} files added, "
-                    f"{len(changed) + len(changed_schema_files)} files changed.")
+        logger.debug(f"Partial parsing enabled: "
+                     f"{len(deleted) + len(deleted_schema_files)} files deleted, "
+                     f"{len(added)} files added, "
+                     f"{len(changed) + len(changed_schema_files)} files changed.")
         self.file_diff = file_diff
 
     # generate the list of files that need parsing


Test patch: