internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: iterative/dvc

Issue:
pull: failed to pull large file with reporting 'InconsistentError: IncompleteRead from source' with aliyun-oss
# Bug Report
pull: failed to download with reporting 'InconsistentError: IncompleteRead from source' with aliyun-oss

## Description
If any file pushed onto aliyun-oss needs more than 60s to download in dvc pull, it will throw error with:
```
ERROR: failed to download 'oss://<bucket>/<path>/<cache_dir>' to '.dvc/cache/<cache_dir>' - {'status': -3, 'x-oss-request-id': '<RANDOM-REQUEST-ID>', 'details': 'InconsistentError: IncompleteRead from source'}
```

### Reproduce
1. dvc init
2. configure an aliyun oss remote as `remote add` document
3. dvc add <any single large file that needs more than 60s to download in dvc pull>
4. dvc push
5. commit git repo

6. clone git repo
7. git pull
8. dvc pull


### Expected
It should pull success with files fetched return.

```console
$ dvc version
DVC version: 1.10.2 (brew)
---------------------------------
Platform: Python 3.9.0 on macOS-11.0.1-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Caches: local
Remotes: oss
Repo: dvc, git
```

**Additional Information (if any):**
https://github.com/iterative/dvc/blob/c4f92314dbb0dd0f0686b6117942154c58a7e41f/dvc/tree/oss.py#L131

The download function here should be `oss2.resumable_download` instead of `get_object_to_file`, otherwise the download timeout will exceed connection_timeout and the connection will be force closed.


Discussion:
> The download function here should be oss2.resumable_download instead of get_object_to_file, otherwise the download timeout will exceed connection_timeout and the connection will be force closed.

@9173860, is there any way to increase that session timeout? And, does this also affect the `push`/`upload`?

I could not find a way to increase it, but the documentation for `resumable_download` is [here](https://translate.google.com/translate?hl=&sl=zh-CN&tl=en&u=https%3A%2F%2Faliyun-oss-python-sdk.readthedocs.io%2Fen%2Flatest%2Feasy.html%23id3). 




Resolving patch:
diff --git a/dvc/tree/oss.py b/dvc/tree/oss.py
--- a/dvc/tree/oss.py
+++ b/dvc/tree/oss.py
@@ -128,6 +128,11 @@ def _download(
         self, from_info, to_file, name=None, no_progress_bar=False, **_kwargs
     ):
         with Tqdm(desc=name, disable=no_progress_bar, bytes=True) as pbar:
-            self.oss_service.get_object_to_file(
-                from_info.path, to_file, progress_callback=pbar.update_to
+            import oss2
+
+            oss2.resumable_download(
+                self.oss_service,
+                from_info.path,
+                to_file,
+                progress_callback=pbar.update_to,
             )


Test patch:
Repository: iterative/dvc

Issue:
dvc version: does not follow symlinks
# Bug Report

## Description

This is the `dvc version` output, where it says the cache directory is `nfs4 on storage:/home` and cache type is `symlink`.

```
DVC version: 1.10.2 (pip)
---------------------------------
Platform: Python 3.8.3 on Linux-5.4.0-54-generic-x86_64-with-glibc2.10
Supports: All remotes
Cache types: symlink
Cache directory: nfs4 on storage:/home
Caches: local
Remotes: s3
Workspace directory: nfs4 on storage:/home
Repo: dvc, git
```

However, I do have a `~/.config/dvc/config` file that overrides this:

```
[core]
    experiments = true
[cache]
    type = "reflink,symlink,copy"
    protected = true
    dir = /home/jc/ssd_cache/dvc_cache
[feature]
    parametrization = true
```

And the actual cache dir is `/home/jc/ssd_cache/dvc_cache` as I've specified instead of `nfs4 on storage:/home`


Discussion:
@johnnychen94 `storage:/home` is not a path to the cache directory, but the partition that it belongs to. Makes it easy for us to tell if your cache and workspace are on the same partition and so on the same filesystem. From docs https://dvc.org/doc/command-reference/version

> Filesystem type (e.g. ext4, FAT, etc.) and drive on which the cache directory is mounted

@johnnychen94 To show current cache dir, run `dvc cache dir`
Oh, I see the issue here.

`/home/jc/ssd_cache` is a symlink to `/mnt/cache/home/jc`, where `/mnt/cache/` is a local ssd raid with ext4 format.

Perhaps `dvc version` needs to follow the symlinks?

---

By modifying it to `/mnt/cache/home/jc/dvc_cache`, the results show up normally

```
DVC version: 1.10.2 (pip)
---------------------------------
Platform: Python 3.8.3 on Linux-5.4.0-54-generic-x86_64-with-glibc2.10
Supports: All remotes
Cache types: symlink
Cache directory: ext4 on /dev/sda3
Caches: local
Remotes: s3
Workspace directory: nfs4 on storage:/home
Repo: dvc, git
```
@johnnychen94 Ah, yeah, makes sense! I think we just need to change https://github.com/iterative/dvc/blob/a7d665157cd838a9406e30c249065111d93a9e4a/dvc/info.py#L145 
to `path = pathlib.Path(path).resolve()` and that should make it work as expected.
Yes, just verified that this change works.
@johnnychen94 Mind sending a PR? :slightly_smiling_face: :pray: 

Resolving patch:
diff --git a/dvc/info.py b/dvc/info.py
--- a/dvc/info.py
+++ b/dvc/info.py
@@ -142,7 +142,8 @@ def get_fs_type(path):
         for part in psutil.disk_partitions(all=True)
     }
 
-    path = pathlib.Path(path)
+    # need to follow the symlink: https://github.com/iterative/dvc/issues/5065
+    path = pathlib.Path(path).resolve()
 
     for parent in itertools.chain([path], path.parents):
         if parent in partition:


Test patch:
Repository: iterative/dvc

Issue:
remove one of many stage outputs: misleading error message
Users cannot remove a data file if it is only one output of many for a stage. It is a valid behavior that needs a better error message:

```
$ dvc remove model.h5
ERROR: failed to remove 'model.h5' - "Stage 'model.h5' not found inside 'dvc.yaml' file"
```

The pipeline:
```
$ cat dvc.yaml
stages:
  train:
    cmd: python train.py
    deps:
    - users.csv
    - train.py
    - mymodel.py
    params:
    - epochs
    - opt
    - log_file
    - dropout
    outs:
    - logs
    - model.h5       # <-- trying to remove this one
    metrics:
    - summary.json:
        cache: false
    plots:
    - logs.csv:
        cache: false
```


Discussion:
Also, formatting issues:
1. The prefix ` failed to remove 'model.h5' -` is not needed. Too obvious.
2. Quoted error message `"Stage 'model.h5' not ..."`

Resolving patch:
diff --git a/dvc/command/remove.py b/dvc/command/remove.py
--- a/dvc/command/remove.py
+++ b/dvc/command/remove.py
@@ -13,8 +13,8 @@ def run(self):
         for target in self.args.targets:
             try:
                 self.repo.remove(target, outs=self.args.outs)
-            except DvcException:
-                logger.exception(f"failed to remove '{target}'")
+            except DvcException as exc:
+                logger.error(exc)
                 return 1
         return 0
 
diff --git a/dvc/stage/exceptions.py b/dvc/stage/exceptions.py
--- a/dvc/stage/exceptions.py
+++ b/dvc/stage/exceptions.py
@@ -81,6 +81,8 @@ def __init__(self, missing_files):
 
 
 class StageNotFound(KeyError, DvcException):
+    __str__ = DvcException.__str__
+
     def __init__(self, file, name):
         self.file = file.relpath
         self.name = name


Test patch:
Repository: iterative/dvc

Issue:
ssh remote: no progress bar on push/pull
Not much to add to the details. When we download a file(s) it does not show progress (only rate).



Discussion:

Resolving patch:
diff --git a/dvc/tree/ssh/connection.py b/dvc/tree/ssh/connection.py
--- a/dvc/tree/ssh/connection.py
+++ b/dvc/tree/ssh/connection.py
@@ -189,6 +189,7 @@ def download(self, src, dest, no_progress_bar=False, progress_title=None):
             desc=progress_title or os.path.basename(src),
             disable=no_progress_bar,
             bytes=True,
+            total=self.getsize(src),
         ) as pbar:
             self.sftp.get(src, dest, callback=pbar.update_to)
 
@@ -221,7 +222,10 @@ def upload(self, src, dest, no_progress_bar=False, progress_title=None):
             progress_title = posixpath.basename(dest)
 
         with Tqdm(
-            desc=progress_title, disable=no_progress_bar, bytes=True
+            desc=progress_title,
+            disable=no_progress_bar,
+            bytes=True,
+            total=os.path.getsize(src),
         ) as pbar:
             self.sftp.put(src, tmp_file, callback=pbar.update_to)
 


Test patch:
Repository: iterative/dvc

Issue:
dvc exp gc: can't run experiments after garbage collection
# Bug Report

## Description

Experiments fail after garbage collection via `dvc exp gc`.

### Reproduce

1. dvc init
2. dvc stage add -n foo -M foo.yaml 'echo "score: 1" > foo.yaml'
3. git add .
4. git commit -m "add stage"
5. dvc exp run
6. dvc exp gc -fT
7. dvc exp run

```console
Stage 'foo' didn't change, skipping
ERROR: Failed to set 'refs/exps/d1/5cee9fd7a4fc20c5d7167671df4ed78b3cfb7a/exp-985c5'
ERROR: Failed to set 'refs/exps/d1/5cee9fd7a4fc20c5d7167671df4ed78b3cfb7a/exp-985c5'
```

### Expected

`dvc exp run` to successfully run experiment.

### Environment information

**Output of `dvc version`:**

```console
$ dvc version
DVC version: 2.0.0a0+adedd1
---------------------------------
Platform: Python 3.9.1 on Linux-5.8.0-38-generic-x86_64-with-glibc2.31
Supports: http, https
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/mapper/vgubuntu-root
Repo: dvc, git
```

**Additional Information (if any):**

<!--
If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
-->


Discussion:

Resolving patch:
diff --git a/dvc/repo/experiments/gc.py b/dvc/repo/experiments/gc.py
--- a/dvc/repo/experiments/gc.py
+++ b/dvc/repo/experiments/gc.py
@@ -3,6 +3,7 @@
 
 from dvc.repo import locked
 
+from .base import EXEC_APPLY, EXEC_BRANCH, EXEC_CHECKPOINT
 from .utils import exp_refs
 
 logger = logging.getLogger(__name__)
@@ -31,9 +32,20 @@ def gc(
     if not keep_revs:
         return 0
 
+    exec_branch = repo.scm.get_ref(EXEC_BRANCH, follow=False)
+    exec_apply = repo.scm.get_ref(EXEC_APPLY)
+    exec_checkpoint = repo.scm.get_ref(EXEC_CHECKPOINT)
+
     removed = 0
     for ref_info in exp_refs(repo.scm):
         if ref_info.baseline_sha not in keep_revs:
+            ref = repo.scm.get_ref(str(ref_info))
+            if exec_branch and str(ref_info):
+                repo.scm.remove_ref(EXEC_BRANCH)
+            if exec_apply and exec_apply == ref:
+                repo.scm.remove_ref(EXEC_APPLY)
+            if exec_checkpoint and exec_checkpoint == ref:
+                repo.scm.remove_ref(EXEC_CHECKPOINT)
             repo.scm.remove_ref(str(ref_info))
             removed += 1
 


Test patch:
Repository: iterative/dvc

Issue:
gs: test failures
Some gs tests started failing after https://github.com/iterative/dvc/pull/5275 


Discussion:

Resolving patch:
diff --git a/dvc/tree/gs.py b/dvc/tree/gs.py
--- a/dvc/tree/gs.py
+++ b/dvc/tree/gs.py
@@ -147,16 +147,19 @@ def walk_files(self, path_info, **kwargs):
     def ls(
         self, path_info, detail=False, recursive=False
     ):  # pylint: disable=arguments-differ
+        import base64
+
         assert recursive
 
         for blob in self.gs.bucket(path_info.bucket).list_blobs(
             prefix=path_info.path
         ):
             if detail:
+                md5_hash = base64.b64decode(blob.md5_hash)
                 yield {
                     "type": "file",
                     "name": blob.name,
-                    "md5": blob.md5_hash.encode().hex(),
+                    "md5": md5_hash.hex(),
                     "size": blob.size,
                 }
             else:


Test patch:
Repository: iterative/dvc

Issue:
tests: exp executor teardown is flaky on windows
Looks like there is some race condition on windows that sometimes happens between cleaning up the test `tmp_dir` and cleaning up the experiments executor temp directory (which is placed in `tmp_dir/.dvc/tmp/...`). May be better to go back to running experiments in system `$TEMP` instead of `.dvc/tmp` (for win tests only)?


Discussion:

Resolving patch:
diff --git a/dvc/repo/experiments/executor/local.py b/dvc/repo/experiments/executor/local.py
--- a/dvc/repo/experiments/executor/local.py
+++ b/dvc/repo/experiments/executor/local.py
@@ -1,6 +1,5 @@
 import logging
 import os
-import sys
 from tempfile import TemporaryDirectory
 from typing import Optional
 
@@ -56,11 +55,4 @@ def _config(self, cache_dir):
     def cleanup(self):
         super().cleanup()
         logger.debug("Removing tmpdir '%s'", self._tmp_dir)
-        try:
-            self._tmp_dir.cleanup()
-        except PermissionError:
-            if os.name == "nt" and sys.version_info < (3, 8):
-                # see https://bugs.python.org/issue26660
-                remove(self._tmp_dir.name)
-                return
-            raise
+        remove(self._tmp_dir.name)


Test patch:
Repository: iterative/dvc

Issue:
decide on `${ var }` vs `${{ var }}`
We have to decide on single braces vs double braces syntax. Both of them work right now, but the former is most tested internally.

1. Single braces
- For: Easier on eyes
- Against: Might affect existing users using environment variables

2. Double braces
- For: compatible with 1.0 users that were using environment variables
- Against: looks ugly

cc @dmpetrov @shcheklein 


Discussion:

Resolving patch:
diff --git a/dvc/parsing/interpolate.py b/dvc/parsing/interpolate.py
--- a/dvc/parsing/interpolate.py
+++ b/dvc/parsing/interpolate.py
@@ -22,11 +22,10 @@
 PERIOD = "."
 KEYCRE = re.compile(
     r"""
-    (?<!\\)                   # escape \${} or ${{}}
-    \$                        # starts with $
-    (?:({{)|({))              # either starts with double braces or single
-    (.*?)                     # match every char inside
-    (?(1)}})(?(2)})           # end with same kinds of braces it opened with
+    (?<!\\)                            # escape \${}
+    \${                                # starts with ${
+    (?P<inner>.*?)                     # match every char inside
+    }                                  # end with {
 """,
     re.VERBOSE,
 )
@@ -137,6 +136,7 @@ def parse_expr(s: str):
         result = get_parser().parseString(s, parseAll=True)
     except ParseException as exc:
         format_and_raise_parse_error(exc)
+        raise AssertionError("unreachable")
 
     joined = result.asList()
     assert len(joined) == 1
@@ -144,7 +144,7 @@ def parse_expr(s: str):
 
 
 def get_expression(match: "Match", skip_checks: bool = False):
-    _, _, inner = match.groups()
+    inner = match["inner"]
     return inner if skip_checks else parse_expr(inner)
 
 


Test patch:
diff --git a/tests/unit/test_interpolate.py b/tests/unit/test_interpolate.py
--- a/tests/unit/test_interpolate.py
+++ b/tests/unit/test_interpolate.py
@@ -6,13 +6,7 @@
 
 
 @pytest.mark.parametrize(
-    "template, var",
-    [
-        ("${value}", "value"),
-        ("${{item}}", "item"),
-        ("${ item }", "item"),
-        ("${{ value }}", "value"),
-    ],
+    "template, var", [("${value}", "value"), ("${ item }", "item")],
 )
 @pytest.mark.parametrize(
     "data", [True, 12, pi, None, False, 0, "0", "123", "Foobar", "", inf, 3e4]
@@ -26,10 +20,7 @@ def test_resolve_primitive_values(data, template, var):
     "template, expected",
     [
         (r"\${value}", "${value}"),
-        (r"\${{value}}", "${{value}}"),
         (r"\${ value }", "${ value }"),
-        (r"\${{ value }}", "${{ value }}"),
-        (r"\${{ value }} days", "${{ value }} days"),
         (r"\${ value } days", "${ value } days"),
         (r"Month of \${value}", "Month of ${value}"),
         (r"May the \${value} be with you", "May the ${value} be with you"),
@@ -39,7 +30,7 @@ def test_resolve_primitive_values(data, template, var):
         ),
     ],
 )
-def test_escape(template, expected, mocker):
+def test_escape(template, expected):
     context = Context({"value": "value"})
     assert context.resolve(template) == expected
Repository: iterative/dvc

Issue:
repo/run: Gitignores with exclusion and sub-gitignore produce duplicates
# Bug Report

## Description

I've identified that the bug is coming from dulwich, you'll find more information in https://github.com/dulwich/dulwich/issues/836.
See the steps below to reproduce it with dvc.

I've created this bug report to make sure to update dulwich when the bug will be solved.

### Reproduce


```bash
dvc init
mkdir -p foo/bar
echo "/*" > foo/.gitignore
echo "\!/bar" >> foo/.gitignore
echo "/should_be_ignored" > foo/bar/.gitignore
dvc run -n test -o foo/bar/should_be_ignored echo "ignored" \> foo/bar/should_be_ignored
```

In `foo/bar/.gitignore` we have:
```text
/should_be_ignored
/should_be_ignored
```

### Expected

In `foo/bar/.gitignore` we should have:
```
/should_be_ignored
```

### Environment information


**Output of `dvc version`:**

```console
DVC version: 1.11.15 (pip)
---------------------------------
Platform: Python 3.7.6 on Darwin-20.2.0-x86_64-i386-64bit
Supports: http, https
Cache types: reflink, hardlink, symlink
Caches: local
Remotes: None
Repo: dvc, git
```


Discussion:
It's not necessarily a bug on our side, though it's still a bug, so marking it as one. The best way to solve it would be to contribute to dulwich and update the version in our setup.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -50,7 +50,7 @@ def run(self):
     "colorama>=0.3.9",
     "configobj>=5.0.6",
     "gitpython>3",
-    "dulwich>=0.20.18",
+    "dulwich>=0.20.19",
     "pygit2>=1.4.0",
     "setuptools>=34.0.0",
     "nanotime>=0.5.2",


Test patch:
Repository: iterative/dvc

Issue:
repo/run: Gitignores with exclusion and sub-gitignore produce duplicates
# Bug Report

## Description

I've identified that the bug is coming from dulwich, you'll find more information in https://github.com/dulwich/dulwich/issues/836.
See the steps below to reproduce it with dvc.

I've created this bug report to make sure to update dulwich when the bug will be solved.

### Reproduce


```bash
dvc init
mkdir -p foo/bar
echo "/*" > foo/.gitignore
echo "\!/bar" >> foo/.gitignore
echo "/should_be_ignored" > foo/bar/.gitignore
dvc run -n test -o foo/bar/should_be_ignored echo "ignored" \> foo/bar/should_be_ignored
```

In `foo/bar/.gitignore` we have:
```text
/should_be_ignored
/should_be_ignored
```

### Expected

In `foo/bar/.gitignore` we should have:
```
/should_be_ignored
```

### Environment information


**Output of `dvc version`:**

```console
DVC version: 1.11.15 (pip)
---------------------------------
Platform: Python 3.7.6 on Darwin-20.2.0-x86_64-i386-64bit
Supports: http, https
Cache types: reflink, hardlink, symlink
Caches: local
Remotes: None
Repo: dvc, git
```


Discussion:
It's not necessarily a bug on our side, though it's still a bug, so marking it as one. The best way to solve it would be to contribute to dulwich and update the version in our setup.
A new dulwich version has been released with the fix, I've linked the PR to make sure to use the right dulwich version.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -50,7 +50,7 @@ def run(self):
     "colorama>=0.3.9",
     "configobj>=5.0.6",
     "gitpython>3",
-    "dulwich>=0.20.14",
+    "dulwich>=0.20.19",
     "setuptools>=34.0.0",
     "nanotime>=0.5.2",
     "pyasn1>=0.4.1",
@@ -143,6 +143,7 @@ def run(self):
     "rangehttpserver==1.2.0",
     "beautifulsoup4==4.4.0",
     "pylint==2.5.3",
+    "astroid<2.5",  # temporary required for pylint, see #5469
     "pylint-pytest>=0.3.0",
     "pylint-plugin-utils",
     "wget",


Test patch:
Repository: iterative/dvc

Issue:
exp show: malformed output for modified/branched checkpoint runs
# Bug Report

If table contains experiments with a modified checkpoint run, returned commits from before the branch point do not include any params/metrics data. To reproduce, run some checkpoints, then resume with a modified param (`exp run -S ...` without `--reset`)

See `a9829d6` through `ebe8fc2` in the example output:

```
$ dvc exp show --no-pager
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┓
┃ Experiment            ┃ Created      ┃ epoch ┃ start ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━┩
│ workspace             │ -            │     9 │ 100   │
│ main                  │ Jan 22, 2021 │     - │ 0     │
│ │ ╓ exp-2e6e3         │ 01:54 PM     │     9 │ 100   │
│ │ ╟ 49876a5           │ 01:54 PM     │     8 │ 100   │
│ │ ╟ 0108609           │ 01:54 PM     │     7 │ 100   │
│ │ ╟ 230af0e (ebe8fc2) │ 01:54 PM     │     6 │ 100   │
│ │ ╓ ebe8fc2           │ -            │     - │ -     │
│ │ ╟ 5e76872           │ -            │     - │ -     │
│ │ ╟ 4b99deb           │ -            │     - │ -     │
│ │ ╟ 0ed55ae           │ -            │     - │ -     │
│ │ ╟ 8449716           │ -            │     - │ -     │
│ ├─╨ a9829d6           │ -            │     - │ -     │
│ │ ╓ exp-d4d58         │ Feb 04, 2021 │    15 │ 0     │
│ ├─╨ 2eb95e4           │ Feb 04, 2021 │    14 │ 0     │
...
```


Discussion:

Resolving patch:
diff --git a/dvc/repo/experiments/show.py b/dvc/repo/experiments/show.py
--- a/dvc/repo/experiments/show.py
+++ b/dvc/repo/experiments/show.py
@@ -66,7 +66,8 @@ def _collect_experiment_branch(res, repo, branch, baseline, **kwargs):
                 exp.update(collected_exp)
         else:
             exp = collected_exp
-        res[rev] = exp
+        if rev not in res:
+            res[rev] = exp
         prev = rev
     if len(revs) > 1:
         res[prev]["checkpoint_parent"] = baseline


Test patch: