internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: iterative/dvc

Issue:
Exponential memory allocation caused by YAML parameter files with reused anchors
# Bug Report

YAML parameter files with enough [reused anchors](https://yaml.org/spec/1.2.2/#alias-nodes) will cause DVC to exhaust the system resources by exponentially appending duplicate elements to `list` objects.

## Minimal reproducible example

```console
$ python <<< "from dvc.repo import Repo; print(Repo().params.show())"
```

> [!NOTE]
> If you prefer [porcelain](https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain), feel free to run `dvc params diff` or call [`dvc.api.params_show()`](https://dvc.org/doc/api-reference/params_show) instead.

### Sample project files

#### `dvc.yaml`
```yaml
stages:
  example:
    foreach: [0, 1, 2, 3]
    do:
      cmd: test
      params:
      - custom.yaml: [references]
      - custom.yaml: [references]
      - custom.yaml: [references]
      - custom.yaml: [references]
```

#### `custom.yaml`

```yaml
values: &anchor [0, 1, 2, 3]

references:
  first: *anchor
  second: *anchor
  third: *anchor
  fourth: *anchor
```

#### `.dvc/config`
```toml
[core]
```

## Environment information

```console
$ dvc doctor
DVC version: 3.33.4 (pip)
-------------------------
Platform: Python 3.10.8 on Linux-6.2.0-1018-azure-x86_64-with-glibc2.31
Subprojects:
        dvc_data = 2.24.0
        dvc_objects = 2.0.1
        dvc_render = 1.0.0
        dvc_task = 0.3.0
        scmrepo = 1.6.0
Supports:
        azure (adlfs = 2023.10.0, knack = 0.11.0, azure-identity = 1.15.0),
        gdrive (pydrive2 = 1.18.0),
        gs (gcsfs = 2023.12.2.post1),
        hdfs (fsspec = 2023.12.2, pyarrow = 14.0.1),
        http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
        oss (ossfs = 2023.12.0),
        s3 (s3fs = 2023.12.2, boto3 = 1.33.13),
        ssh (sshfs = 2023.10.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.12.2)
Config:
        Global: /home/codespace/.config/dvc
        System: /etc/xdg/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/loop3
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/847c03a095ad0b42b23e20ceb6c386f0
```


Discussion:

Resolving patch:
diff --git a/dvc/dependency/param.py b/dvc/dependency/param.py
--- a/dvc/dependency/param.py
+++ b/dvc/dependency/param.py
@@ -57,8 +57,9 @@ def read_param_file(
     from copy import deepcopy
 
     from dpath import merge
+    from funcy import distinct
 
-    for key_path in key_paths:
+    for key_path in distinct(key_paths):
         merge(
             ret,
             deepcopy(dpath.search(config, key_path, separator=".")),


Test patch:
Repository: iterative/dvc

Issue:
Cannot set azure timeout: config file error: expected int for dictionary value @ data['remote']['dvc-remote']['timeout']
## Bug Report

Cannot set azure timeout

## Description

I want to set longer azure timeouts to avoid some problems connected with poor network condition but when I try to do that I get msg:
 ERROR: configuration error - config file error: expected int for dictionary value @ data['remote']['dvc-remote']['timeout']

## Reproduce

dvc remote modify dvc-remote timeout 600

### Expected

Adding timeout parameter to dvc .config file



Discussion:
@efiop It looks like in https://github.com/iterative/dvc/pull/10027, we should be using `Coerce(int)`:

https://github.com/iterative/dvc/blob/5c31189a919f6f7208db9ca29eef940113fe5155/dvc/config_schema.py#L263-L265

Do we have tests for the config schema to catch these kind of issues?
Thanks for fast response, I agree that value from config should be mapped to int.  

Resolving patch:
diff --git a/dvc/config_schema.py b/dvc/config_schema.py
--- a/dvc/config_schema.py
+++ b/dvc/config_schema.py
@@ -260,9 +260,9 @@ def __call__(self, data):
                     "exclude_shared_token_cache_credential": Bool,
                     "exclude_managed_identity_credential": Bool,
                     Optional("verify", default=False): Bool,
-                    "timeout": int,
-                    "read_timeout": int,
-                    "connection_timeout": int,
+                    "timeout": Coerce(int),
+                    "read_timeout": Coerce(int),
+                    "connection_timeout": Coerce(int),
                     **REMOTE_COMMON,
                 },
                 "oss": {


Test patch:
Repository: iterative/dvc

Issue:
Unable to extract dvc Stage Name
Currently there is no way to extract dvc Stage Name for the current stage that is being run by dvc. It will be a good idea to create a environment variable that can extract dvc stage name.


Discussion:

Resolving patch:
diff --git a/dvc/env.py b/dvc/env.py
--- a/dvc/env.py
+++ b/dvc/env.py
@@ -19,3 +19,4 @@
 DVC_SQLALCHEMY_ECHO = "DVC_SQLALCHEMY_ECHO"
 DVC_SYSTEM_CONFIG_DIR = "DVC_SYSTEM_CONFIG_DIR"
 DVC_UPDATER_ENDPOINT = "DVC_UPDATER_ENDPOINT"
+DVC_STAGE = "DVC_STAGE"
diff --git a/dvc/stage/run.py b/dvc/stage/run.py
--- a/dvc/stage/run.py
+++ b/dvc/stage/run.py
@@ -39,7 +39,7 @@ def _enforce_cmd_list(cmd):
 
 
 def prepare_kwargs(stage, run_env=None):
-    from dvc.env import DVC_ROOT
+    from dvc.env import DVC_ROOT, DVC_STAGE
 
     kwargs = {"cwd": stage.wdir, "env": fix_env(None), "close_fds": True}
 
@@ -48,6 +48,9 @@ def prepare_kwargs(stage, run_env=None):
     if DVC_ROOT not in kwargs["env"]:
         kwargs["env"][DVC_ROOT] = stage.repo.root_dir
 
+    # Create DVC_STAGE env variable for every command
+    kwargs["env"][DVC_STAGE] = stage.addressing
+
     # NOTE: when you specify `shell=True`, `Popen` [1] will default to
     # `/bin/sh` on *nix and will add ["/bin/sh", "-c"] to your command.
     # But we actually want to run the same shell that we are running


Test patch:
Repository: iterative/dvc

Issue:
Ability to configure the replication factor for the HDFS file system
Based on this [PR](https://github.com/fsspec/filesystem_spec/pull/1486), we need to be able to configure replication argument in the config file.


Discussion:
@samssh Would you be able to contribute the PR?
@dberenbaum Yes, I'm working on it.

@dberenbaum Could you take a look at the PR?

Resolving patch:
diff --git a/dvc/config_schema.py b/dvc/config_schema.py
--- a/dvc/config_schema.py
+++ b/dvc/config_schema.py
@@ -232,7 +232,12 @@ def __call__(self, data):
                     Optional("verify", default=False): Bool,
                     **REMOTE_COMMON,
                 },
-                "hdfs": {"user": str, "kerb_ticket": str, **REMOTE_COMMON},
+                "hdfs": {
+                    "user": str,
+                    "kerb_ticket": str,
+                    "replication": int,
+                    **REMOTE_COMMON,
+                },
                 "webhdfs": {
                     "kerberos": Bool,
                     "kerberos_principal": str,


Test patch:
Repository: iterative/dvc

Issue:
Feature proposal: `dvc artifacts get --show-url`
DVC currently supports `dvc get --show-url` as a way to retrieve just the URL of a DVC-versioned object as opposed to the object itself.

However, there is no equivalent for `dvc artifacts get`. This came as a customer request (to allow easier sharing of results even to people who are not DVC/DVC Studio users). It also has advantages e.g. in model deployment to Sagemaker (which requires the artifact URL on S3).


Discussion:

Resolving patch:
diff --git a/dvc/commands/artifacts.py b/dvc/commands/artifacts.py
--- a/dvc/commands/artifacts.py
+++ b/dvc/commands/artifacts.py
@@ -13,6 +13,9 @@ def run(self):
         from dvc.scm import CloneError
         from dvc.ui import ui
 
+        if self.args.show_url:
+            return self._show_url()
+
         try:
             count, out = Artifacts.get(
                 self.args.url,
@@ -36,6 +39,28 @@ def run(self):
             )
             return 1
 
+    def _show_url(self):
+        from dvc.api import artifacts_show, get_url
+        from dvc.ui import ui
+
+        artifact = artifacts_show(
+            self.args.name,
+            version=self.args.rev,
+            stage=self.args.stage,
+            repo=self.args.url,
+        )
+
+        url = get_url(
+            artifact["path"],
+            repo=self.args.url,
+            rev=artifact["rev"],
+            remote=self.args.remote,
+            remote_config=self.args.remote_config,
+        )
+        ui.write(url, force=True)
+
+        return 0
+
 
 def add_parser(subparsers, parent_parser):
     ARTIFACTS_HELP = "DVC model registry artifact commands."
@@ -84,6 +109,14 @@ def add_parser(subparsers, parent_parser):
         help="Destination path to download artifact to",
         metavar="<path>",
     ).complete = completion.DIR
+    get_parser.add_argument(
+        "--show-url",
+        action="store_true",
+        help=(
+            "Print the storage location (URL) the target data would be "
+            "downloaded from, and exit."
+        ),
+    )
     get_parser.add_argument(
         "-j",
         "--jobs",


Test patch:
Repository: iterative/dvc

Issue:
Config setting studio.url is not respected
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description

<!--
A clear and concise description of what the bug is.
-->
The `studio.url` config setting is ignored by dvc, and the login link is always to studio.iterative.ai.

### Reproduce

<!--
Step list of how to reproduce the bug
-->

```console
$ dvc init
$ dvc config studio.url https://example.com
$ dvc config studio.url
https://example.com
$ dvc studio login
Opening link for login at https://studio.iterative.ai/auth/device-login?code=Z7JDECGR

Once you've logged in, return here and you'll be ready to start the experiments.

Failed to open a web browser. Open the above url to continue in your web browser.
```

### Expected

<!--
A clear and concise description of what you expect to happen.
-->
The login command should produce a link to the server specified by `studio.url`, as described in the [docs](https://dvc.org/doc/user-guide/project-structure/configuration#studio).

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor
DVC version: 3.50.1 (pip)
-------------------------
Platform: Python 3.10.12 on Linux-6.5.0-1022-oem-x86_64-with-glibc2.35
Subprojects:
	dvc_data = 3.15.1
	dvc_objects = 5.1.0
	dvc_render = 1.0.2
	dvc_task = 0.4.0
	scmrepo = 3.3.2
Supports:
	http (aiohttp = 3.9.5, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.5, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2024.3.1, boto3 = 1.34.69)
Config:
	Global: /home/george/.config/dvc
	System: /etc/xdg/xdg-ubuntu/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/vgubuntu-root
Caches: local
Remotes: local, s3
Workspace directory: ext4 on /dev/mapper/vgubuntu-root
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/b380dbe5c5355f17c427aac92d8f5759
```


Discussion:

Resolving patch:
diff --git a/dvc/commands/studio.py b/dvc/commands/studio.py
--- a/dvc/commands/studio.py
+++ b/dvc/commands/studio.py
@@ -18,11 +18,16 @@ def run(self):
         from dvc.utils.studio import STUDIO_URL
         from dvc_studio_client.auth import StudioAuthError, get_access_token
 
+        studio = self.config.get("studio", {})
         name = self.args.name
-        hostname = self.args.hostname or os.environ.get(DVC_STUDIO_URL) or STUDIO_URL
+        hostname = (
+            self.args.hostname
+            or os.environ.get(DVC_STUDIO_URL)
+            or studio.get("url")
+            or STUDIO_URL
+        )
         scopes = self.args.scopes
 
-        studio = self.config.get("studio", {})
         if studio.get("url", hostname) == hostname and "token" in studio:
             raise DvcException(
                 "Token already exists. "


Test patch:
Repository: opensearch-project/opensearch-build

Issue:
Script to build each component from manifest and assemble bundle.
This script should read a manifest and output all artifacts ready for upload.
Example.
/build/opensearch-dashboards-min-1.0.0-SNAPSHOT-linux-arm64-198.tar.gz <- min bundle
/build/opensearch-dashboards-1.0.0-SNAPSHOT-linux-arm64-198.tar.gz. <-- full bundle
/build/opensearch-sql-1.0.0.0-rc1.zip
/build/org/opensearch... <- maven artifacts

Input: to the script should be a manifest file location - format defined here #111 
Output: all required artifacts are written to ./build

- [x] Clone each component repository defined in the manifest
- [x] Build each component. This includes plugin zips and maven artifacts and place under a new folder with build id.  Note: We need to know if the version of a particular component is already published to maven central.  If this is the case we do not need to rebuild and include that artifact.
- [ ] Assemble the bundle itself and add to the /build directory. This is dependent on being able to write manifests - #134 

To make assembling maven artifacts easier, each repo can be published to maven local and copied from that location into /build.  All artifacts will be under org/opensearch.



Discussion:

Resolving patch:
diff --git a/tools/bundle-build/lib/component.py b/tools/bundle-build/lib/component.py
--- a/tools/bundle-build/lib/component.py
+++ b/tools/bundle-build/lib/component.py
@@ -21,6 +21,9 @@ def git_repository(self):
     def ref(self):
         return self._ref
 
+    def artifacts(self):
+        return self._artifacts
+
     def checkout(self):
         self._git_repository = GitRepository(self.repository(), self.ref())
 
@@ -57,13 +60,29 @@ def export(self, dest):
         if os.path.exists(artifacts_path):
             print(f'Publishing artifacts from {artifacts_path} into {dest} ...')
             self.git_repository().execute(f'cp -r "{artifacts_path}/"* "{dest}"')
+            self.set_artifacts()
         else:
             print(f'No artifacts found in {artifacts_path}, skipping.')
 
+    def set_artifacts(self):
+       self._artifacts = {key: self.file_paths(key) for key in ["maven", "plugins", "bundle", "libs"] if self.file_paths(key)}
+
+    def file_paths(self, dir_name):
+      artifacts_path = self.artifacts_path()
+      sub_dir = os.path.join(artifacts_path, dir_name)
+      file_paths = []
+      if os.path.exists(sub_dir):
+        for dir, dirs, files in os.walk(sub_dir):
+          for file_name in files:
+            path = os.path.relpath(os.path.join(dir, file_name), artifacts_path)
+            file_paths.append(path)
+      return file_paths
+
     def dict(self):
         return {
             'name': self.name(),
             'repository': self.repository(),
             'ref': self.ref(),
-            'sha': self.git_repository().sha()
+            'sha': self.git_repository().sha(),
+            'artifacts': self.artifacts()
         }


Test patch:
Repository: opensearch-project/opensearch-build

Issue:
Bundle system - copy opensearch-tar-install script into bundle
The [opensearch-tar-install](https://github.com/opensearch-project/opensearch-build/blob/main/release/tar/linux/opensearch-tar-install.sh) script needs to be included with assembled bundles similar to [this](https://github.com/opensearch-project/opensearch-build/blob/main/release/tar/linux/opensearch-tar-build.sh#L156) shell script.




Discussion:

Resolving patch:
diff --git a/bundle-workflow/python/assemble.py b/bundle-workflow/python/assemble.py
--- a/bundle-workflow/python/assemble.py
+++ b/bundle-workflow/python/assemble.py
@@ -6,6 +6,7 @@
 import os
 import tempfile
 import argparse
+import shutil
 from assemble_workflow.bundle import Bundle
 from assemble_workflow.bundle_recorder import BundleRecorder
 from manifests.build_manifest import BuildManifest
@@ -14,6 +15,12 @@
 parser.add_argument('manifest', type = argparse.FileType('r'), help="Manifest file.")
 args = parser.parse_args()
 
+tarball_installation_script = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../../release/tar/linux/opensearch-tar-install.sh')
+
+if not os.path.isfile(tarball_installation_script):
+    print(f'No installation script found at path: {tarball_installation_script}')
+    exit(1)
+
 build_manifest = BuildManifest.from_file(args.manifest)
 build = build_manifest.build
 artifacts_dir = os.path.dirname(os.path.realpath(args.manifest.name))
@@ -31,6 +38,9 @@
     bundle.install_plugins()
     print(f'Installed plugins: {bundle.installed_plugins}')
 
+    # Copy the tar installation script into the bundle
+    shutil.copyfile(tarball_installation_script, os.path.join(bundle.archive_path, os.path.basename(tarball_installation_script)))
+
     #  Save a copy of the manifest inside of the tar
     bundle_recorder.write_manifest(bundle.archive_path)
     bundle.build_tar(output_dir)


Test patch:
Repository: opensearch-project/opensearch-build

Issue:
Add Core plugins zips to published artifacts.
The plugins built with the core repo should be bundled as zips and hosted at artifacts.opensearch.org.

This task involves updating Opensearch's build.sh script to build and include these artifacts with the component's artifact list.

Right now we also have brittle logic that assumes any component with "plugins" in their artifacts list is a plugin repository.  This should be updated to identify the min bundle component in another way.  Perhaps with a separate artifact folder for "min-bundle"


Discussion:

Resolving patch:
diff --git a/bundle-workflow/python/build_workflow/builder.py b/bundle-workflow/python/build_workflow/builder.py
--- a/bundle-workflow/python/build_workflow/builder.py
+++ b/bundle-workflow/python/build_workflow/builder.py
@@ -32,7 +32,7 @@ def build(self, version, arch, snapshot):
 
     def export_artifacts(self):
         artifacts_dir = os.path.realpath(os.path.join(self.git_repo.dir, self.output_path))
-        for artifact_type in ["maven", "bundle", "plugins", "libs"]:
+        for artifact_type in ["maven", "bundle", "plugins", "libs", "core-plugins"]:
             for dir, dirs, files in os.walk(os.path.join(artifacts_dir, artifact_type)):
                 for file_name in files:
                     absolute_path = os.path.join(dir, file_name)


Test patch:
Repository: opensearch-project/opensearch-build

Issue:
Signer client does not support files with '.' in filenames
Many of our artifacts contain '.' in the artifact names.

`client-benchmarks-1.0.0-javadoc.jar`

These files need to be signed properly.


Discussion:

Resolving patch:
diff --git a/bundle-workflow/src/sign.py b/bundle-workflow/src/sign.py
--- a/bundle-workflow/src/sign.py
+++ b/bundle-workflow/src/sign.py
@@ -7,6 +7,7 @@
 # compatible open source license.
 
 import argparse
+import os
 
 from manifests.build_manifest import BuildManifest
 from signing_workflow.signer import Signer
@@ -20,6 +21,7 @@
 args = parser.parse_args()
 
 manifest = BuildManifest.from_file(args.manifest)
+basepath = os.path.dirname(os.path.abspath(manifest.name))
 signer = Signer()
 
 for component in manifest.components:
@@ -34,6 +36,6 @@
         if args.type and args.type != artifact_type:
             continue
 
-        signer.sign(component.artifacts[artifact_type])
+        signer.sign_artifacts(component.artifacts[artifact_type], basepath)
 
 print("Done.")
diff --git a/bundle-workflow/src/signing_workflow/signer.py b/bundle-workflow/src/signing_workflow/signer.py
--- a/bundle-workflow/src/signing_workflow/signer.py
+++ b/bundle-workflow/src/signing_workflow/signer.py
@@ -28,17 +28,15 @@ def __init__(self):
 
     def sign_artifacts(self, artifacts, basepath):
         for artifact in artifacts:
-            if self.is_invalid_file_type(artifact):
+            if not self.is_valid_file_type(artifact):
                 print(f"Skipping signing of file ${artifact}")
                 continue
             location = os.path.join(basepath, artifact)
             self.sign(location)
             self.verify(location + ".asc")
 
-    def is_invalid_file_type(self, file_name):
-        return (
-            "".join(pathlib.Path(file_name).suffixes) not in Signer.ACCEPTED_FILE_TYPES
-        )
+    def is_valid_file_type(self, file_name):
+        return any(x in [pathlib.Path(file_name).suffix, "".join(pathlib.Path(file_name).suffixes)] for x in Signer.ACCEPTED_FILE_TYPES)
 
     def get_repo_url(self):
         if "GITHUB_TOKEN" in os.environ:


Test patch:
diff --git a/bundle-workflow/tests/tests_signing_workflow/test_sign.py b/bundle-workflow/tests/tests_signing_workflow/test_signer.py
similarity index 91%
rename from bundle-workflow/tests/tests_signing_workflow/test_sign.py
rename to bundle-workflow/tests/tests_signing_workflow/test_signer.py
--- a/bundle-workflow/tests/tests_signing_workflow/test_sign.py
+++ b/bundle-workflow/tests/tests_signing_workflow/test_signer.py
@@ -17,6 +17,7 @@ def test_accepted_file_types(self, git_repo):
             "the-module.module",
             "the-tar.tar.gz",
             "random-file.txt",
+            "something-1.0.0.0.jar"
         ]
         expected = [
             call("/path/the-jar.jar"),
@@ -25,6 +26,7 @@ def test_accepted_file_types(self, git_repo):
             call("/path/the-pom.pom"),
             call("/path/the-module.module"),
             call("/path/the-tar.tar.gz"),
+            call("/path/something-1.0.0.0.jar"),
         ]
         signer = Signer()
         signer.sign = MagicMock()