internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: iterative/dvc

Issue:
Analytics errors out on SCM creation
After https://github.com/iterative/dvc/pull/3257

`dvc init --no-scm` throws errors which seem to be related to the analytics module.


Discussion:

Resolving patch:
diff --git a/dvc/analytics.py b/dvc/analytics.py
--- a/dvc/analytics.py
+++ b/dvc/analytics.py
@@ -15,7 +15,8 @@
 from dvc.exceptions import NotDvcRepoError
 from dvc.lock import Lock, LockError
 from dvc.repo import Repo
-from dvc.scm import SCM
+from dvc.scm import SCM, NoSCM
+from dvc.scm.base import SCMError
 from dvc.utils import env2bool, is_binary
 from dvc.utils.fs import makedirs
 
@@ -85,6 +86,8 @@ def _scm_in_use():
     try:
         scm = SCM(root_dir=Repo.find_root())
         return type(scm).__name__
+    except SCMError:
+        return NoSCM.__name__
     except NotDvcRepoError:
         pass
 


Test patch:
diff --git a/tests/func/test_analytics.py b/tests/func/test_analytics.py
--- a/tests/func/test_analytics.py
+++ b/tests/func/test_analytics.py
@@ -1,7 +1,9 @@
 import mock
 
+from dvc.analytics import _scm_in_use
 from dvc.main import main
 from dvc.compat import fspath
+from dvc.repo import Repo
 
 
 @mock.patch("dvc.analytics.send")
@@ -19,3 +21,24 @@ def test_main_analytics(mock_is_enabled, mock_report, tmp_dir, dvc):
     assert 0 == main(["add", "foo"])
     assert mock_is_enabled.called
     assert mock_report.called
+
+
+def test_scm_dvc_only(tmp_dir, dvc):
+    scm = _scm_in_use()
+    assert scm == "NoSCM"
+
+
+def test_scm_git(tmp_dir, scm, dvc):
+    scm = _scm_in_use()
+    assert scm == "Git"
+
+
+def test_scm_subrepo(tmp_dir, scm):
+    subdir = tmp_dir / "subdir"
+    subdir.mkdir()
+
+    with subdir.chdir():
+        Repo.init(subdir=True)
+        scm = _scm_in_use()
+
+    assert scm == "Git"
Repository: iterative/dvc

Issue:
pyfastcopy causes BlockingIOError
**Please provide information about your setup**

> DVC version: 0.83.0
> Python version: 3.6.9
> Platform: Linux-4.12.14-150.38-default-x86_64-with-glibc2.9
> Binary: False
> Package: pip
> Cache: reflink - not supported, hardlink - supported, symlink - supported
> Filesystem type (cache directory): ('gpfs', 'tlhome2')
> Filesystem type (workspace): ('gpfs', 'tlhome2')

-------------------------------------------

I am getting the error `BlockingIOError: [Errno 11] Resource temporarily unavailable` when running `dvc checkout`. This has been solved by uninstalling pyfastcopy, but for future reference I am submitting it as a bug report.




Discussion:
[dvc_debug.txt](https://github.com/iterative/dvc/files/4246099/dvc_debug.txt)
See output of `dvc checkout -v` for full details.
Hi @murnanedaniel !

Thanks for the report! We've removed pyfastcopy from deps for now, will release a new version soon.

Need to revisit pyfastcopy in general, not liking these BlockingIOError errors popping up.

CC @rxxg 
Unfortunately I don't have access to a Linux setup to test that module, I just put in the code for symmetry with the Windows version.  If nobody is complaining about Linux network perf you can probably just leave it out and wait for the transition to Python 3.8.

Otherwise it might be worth a bug report to the upstream project, but with no possibility to test it I'll have to let someone else handle that.

Resolving patch:
diff --git a/dvc/system.py b/dvc/system.py
--- a/dvc/system.py
+++ b/dvc/system.py
@@ -16,15 +16,6 @@
         speedcopy.patch_copyfile()
     except ImportError:
         pass
-else:
-    import sys
-
-    if sys.version_info < (3, 8):
-        try:
-            # Importing the module monkey-patches shutil.copyfile
-            import pyfastcopy  # noqa: F401
-        except ImportError:
-            pass
 
 
 class System(object):
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -74,7 +74,7 @@ def run(self):
     "pywin32>=225; sys_platform == 'win32'",
     "networkx>=2.1,<2.4",
     "pydot>=1.2.4",
-    "speedcopy>=2.0.1",
+    "speedcopy>=2.0.1; sys_platform == 'win32'",
     "flatten_json>=0.1.6",
     "texttable>=0.5.2",
 ]


Test patch:
Repository: iterative/dvc

Issue:
ERROR: unexpected error - No module named 'jsonpath_ng.bin'; 'jsonpath_ng'
Suddenly I have this error

```
ERROR: unexpected error - No module named 'jsonpath_ng.bin'; 'jsonpath_ng' is not a package
    at ChildProcess.exithandler (child_process.js:295:12)
    at ChildProcess.emit (events.js:210:5)
    at maybeClose (internal/child_process.js:1021:16)
    at Socket.<anonymous> (internal/child_process.js:430:11)
    at Socket.emit (events.js:210:5)
    at Pipe.<anonymous> (net.js:659:12) {
  killed: false,
  code: 255,
  signal: null,
  cmd: 'dvc metrics diff  --show-json origin/master 42e3fcbefe1f4b68fd9462ef1f79bcdeed0fd3fc',
  stdout: '\n' +
    'Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!\n',
  stderr: "ERROR: unexpected error - No module named 'jsonpath_ng.bin'; 'jsonpath_ng' is not a package\n"
```

**Please provide information about your setup**
DVC version: 0.87.0
Python version: 3.6.10
Platform: Linux-5.0.0-1032-azure-x86_64-with-debian-buster-sid
Binary: False
Package: pip




Discussion:
Caused by https://github.com/h2non/jsonpath-ng/issues/36.

We can either pin it to `1.4.3` or upgrade to `1.5.1` which is already released. Or, just disallow `1.5.0`.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -53,7 +53,7 @@ def run(self):
     "nanotime>=0.5.2",
     "pyasn1>=0.4.1",
     "voluptuous>=0.11.7",
-    "jsonpath-ng>=1.4.3",
+    "jsonpath-ng>=1.5.1",
     "requests>=2.22.0",
     "grandalf==0.6",
     "distro>=1.3.0",


Test patch:
Repository: iterative/dvc

Issue:
pipeline show: in outputs (--outs) mode prints dependencies
Version:

```
DVC version: 0.88.0+79aa61
Python version: 3.7.6
Platform: Darwin-18.2.0-x86_64-i386-64bit
Binary: False
Package: None
Filesystem type (workspace): ('apfs', '/dev/disk1s1')
```

Issue:

See here:

https://github.com/iterative/dvc.org/pull/1065

Actual output is the one we have in the docs now.




Discussion:

Resolving patch:
diff --git a/dvc/command/pipeline.py b/dvc/command/pipeline.py
--- a/dvc/command/pipeline.py
+++ b/dvc/command/pipeline.py
@@ -48,26 +48,21 @@ def _build_graph(self, target, commands, outs):
             elif outs:
                 for out in stage.outs:
                     nodes.add(str(out))
-                for dep in stage.deps:
-                    nodes.add(str(dep))
             else:
                 nodes.add(stage.relpath)
 
         edges = []
-
-        if outs:
-            for stage in networkx.dfs_preorder_nodes(G, target_stage):
-                for dep in stage.deps:
-                    for out in stage.outs:
-                        edges.append((str(out), str(dep)))
-        else:
-            for from_stage, to_stage in networkx.edge_dfs(G, target_stage):
-                if commands:
-                    if to_stage.cmd is None:
-                        continue
-                    edges.append((from_stage.cmd, to_stage.cmd))
-                else:
-                    edges.append((from_stage.relpath, to_stage.relpath))
+        for from_stage, to_stage in networkx.edge_dfs(G, target_stage):
+            if commands:
+                if to_stage.cmd is None:
+                    continue
+                edges.append((from_stage.cmd, to_stage.cmd))
+            elif outs:
+                for from_out in from_stage.outs:
+                    for to_out in to_stage.outs:
+                        edges.append((str(from_out), str(to_out)))
+            else:
+                edges.append((from_stage.relpath, to_stage.relpath))
 
         return list(nodes), edges, networkx.is_tree(G)
 


Test patch:
diff --git a/tests/func/test_pipeline.py b/tests/func/test_pipeline.py
--- a/tests/func/test_pipeline.py
+++ b/tests/func/test_pipeline.py
@@ -239,15 +239,18 @@ def locked_stage(self):
         self.assertEqual(len(pipelines), 0)
 
 
-def test_split_pipeline(tmp_dir, dvc):
+def test_split_pipeline(tmp_dir, scm, dvc):
+    tmp_dir.scm_gen("git_dep1", "git_dep1")
+    tmp_dir.scm_gen("git_dep2", "git_dep2")
+
     tmp_dir.dvc_gen("data", "source file content")
     dvc.run(
-        deps=["data"],
+        deps=["git_dep1", "data"],
         outs=["data_train", "data_valid"],
         cmd="echo train >> data_train && echo valid >> data_valid",
     )
     stage = dvc.run(
-        deps=["data_train", "data_valid"],
+        deps=["git_dep2", "data_train", "data_valid"],
         outs=["result"],
         cmd="echo result >> result",
     )
Repository: iterative/dvc

Issue:
repro: change 'already exists' msg
https://discuss.dvc.org/t/tutorial-improvement-other-suggestions/79


Discussion:
I'd like to take this one up.
@SRITANU sounds good!
I am extremely busy with my work. If anyone wants to take up the issue, please feel free to do that. If not, I am gonna get back to this issue when I get time.

Resolving patch:
diff --git a/dvc/stage.py b/dvc/stage.py
--- a/dvc/stage.py
+++ b/dvc/stage.py
@@ -55,7 +55,7 @@ def __init__(self, fname):
 
 class StageFileAlreadyExistsError(DvcException):
     def __init__(self, relpath):
-        msg = "stage '{}' already exists".format(relpath)
+        msg = "not overwriting '{}'".format(relpath)
         super().__init__(msg)
 
 


Test patch:
Repository: iterative/dvc

Issue:
PyYaml version is pinned to <5.2 despite vulnerability
DVC still requires PyYAML < 5.2 (https://github.com/iterative/dvc/blob/master/setup.py#L67) for aws-cli compatibility, despite the fact that that aws-cli supports version < 5.4 for python >= 3.4 (https://github.com/aws/aws-cli/blob/develop/setup.cfg). 

PyYAML versions < 5.3.1 might contain a vulnerability (https://nvd.nist.gov/vuln/detail/CVE-2020-1747). 

Could it be possible to drop support for Python 3.4 in order to allow PyYAML upgrade? I can do the PR if that's easier for you.

(In my use-case, I'm not even using AWS backend and this requirement is pinning my PyYAML version for my whole project, so it is a major issue). 


Discussion:
Hi @SdgJlbl ! We don't support 3.4 for more than a year now. I think that dependency limitation might be a leftover. We would definitely appreciate a PR :pray: 
Great, I'll do that

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -64,7 +64,7 @@ def run(self):
     "treelib>=1.5.5",
     "inflect>=2.1.0,<4",
     "humanize>=0.5.1",
-    "PyYAML>=5.1.2,<5.2",  # Compatibility with awscli
+    "PyYAML>=5.1.2,<5.4",  # Compatibility with awscli
     "ruamel.yaml>=0.16.1",
     "funcy>=1.14",
     "pathspec>=0.6.0",


Test patch:
Repository: iterative/dvc

Issue:
ssh remote: Pass `ProxyCommand` to paramiko
If you want to set a ssh remote to a private server you should be able to do so by doing a proxy jump using the `ProxyCommand` option in your ssh config file, e.g:
```
Host <ON-PREMISES-SERVER>
        User <USER>
        ProxyCommand ssh -W %h:%p <PUBLIC-SERVER>

Host <PUBLIC-SERVER>
        HostName <PUBLIC-IP>
        User <USER>
        Port <PORT>
```
Currently the `ProxyCommand` is not passed to paramiko, so the on-premises server name is not found, e.g.:
```bash
$ dvc remote add -d <ON-PREMISES-SERVER> ssh://<ON-PREMISES-SERVER>/path/to/directory/
$ dvc push
ERROR: unexpected error - [Errno -2] Name or service not known
```


Discussion:
Discord context https://discordapp.com/channels/485586884165107732/485596304961962003/694489311164235854 

Resolving patch:
diff --git a/dvc/remote/ssh/__init__.py b/dvc/remote/ssh/__init__.py
--- a/dvc/remote/ssh/__init__.py
+++ b/dvc/remote/ssh/__init__.py
@@ -85,6 +85,13 @@ def __init__(self, repo, config):
         self.password = config.get("password", None)
         self.ask_password = config.get("ask_password", False)
         self.gss_auth = config.get("gss_auth", False)
+        proxy_command = user_ssh_config.get("proxycommand", False)
+        if proxy_command:
+            import paramiko
+
+            self.sock = paramiko.ProxyCommand(proxy_command)
+        else:
+            self.sock = None
 
     @staticmethod
     def ssh_config_filename():
@@ -136,6 +143,7 @@ def ssh(self, path_info):
             timeout=self.timeout,
             password=self.password,
             gss_auth=self.gss_auth,
+            sock=self.sock,
         )
 
     def exists(self, path_info):


Test patch:
Repository: iterative/dvc

Issue:
CopyFile2 not supported in Windows 7
A user reported on [discord](https://discordapp.com/channels/485586884165107732/485596304961962003/672482282522673162) that `dvc add` was throwing the following error:

```
ERROR: unexpected error - function 'CopyFile2' not found
```

Here's the output of `dvc version`:
```
DVC version: 0.82.6
Python version: 3.6.3
Platform: Windows-7-6.1.7601-SP1
Binary: False
Package: pip
Cache: reflink - not supported, hardlink - supported, symlink - not supported
```

We are using `CopyFile2` on Windows to increase the performance of copy (see [speedcopy](https://github.com/antirotor/speedcopy)), but that [system call was introduced in Windows 8](
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfile2).

The solution was to `pip uninstall speedcopy` and let DVC fallback to Python's `copy` implementation.

**What should we do?**
* Catch `AttributeError: function 'CopyFile2' not found` and output an actionable error message to fix it.
* State in README / docs that we don't support `< Windows 8`?
* Nothing?


Discussion:
Looks like Windows 7 EOL is already here? Considering that the whole Windows users share for DVC is about 2% or something not sure we should spend too much time on it?

We definitely should introduce some user-friendly message though and change docs is needed.
@shcheklein , totally, looks like it already happened:


[EOL: Windows 7, service pack 1*, (mainstream) January 13, 2015 - (extended) January 14, 2020](https://support.microsoft.com/en-us/help/13853/windows-lifecycle-fact-sheet)




Resolving patch:
diff --git a/dvc/system.py b/dvc/system.py
--- a/dvc/system.py
+++ b/dvc/system.py
@@ -10,7 +10,11 @@
 
 logger = logging.getLogger(__name__)
 
-if platform.system() == "Windows" and sys.version_info < (3, 8):
+if (
+    platform.system() == "Windows"
+    and sys.version_info < (3, 8)
+    and sys.getwindowsversion() >= (6, 2)
+):
     try:
         import speedcopy
 


Test patch:
Repository: iterative/dvc

Issue:
Relax pip dependency versions constraints for python-dateutil
Description of the problem:
ERROR: dvc 0.93.0 has requirement python-dateutil **<2.8.1** ,>=2.1, but you'll have python-dateutil 2.8.1 which is incompatible.

It'd be good to remove or relax upper constraints on python-dateutil, to include 2.8.1 and 2.8.2.  Is there a good reason why these are not included, or something is broken after 2.8.1 ?


Discussion:
Hi @dchichkov ! 

Mind submitting a PR to remove it? https://github.com/iterative/dvc/blob/master/setup.py#L50 It is an old sync requirement that is probably no longer needed. We'll see in the PR CI tests.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -47,7 +47,6 @@ def run(self):
 
 
 install_requires = [
-    "python-dateutil<2.8.1,>=2.1",  # Consolidates azure-blob-storage and boto3
     "ply>=3.9",  # See https://github.com/pyinstaller/pyinstaller/issues/1945
     "colorama>=0.3.9",
     "configobj>=5.0.6",


Test patch:
Repository: iterative/dvc

Issue:
remote: reduce traverse weight multiplier
In `remote.cache_exists()`, the traverse/full remote listing method is weighted to account for some performance hits that make it inherently slower than querying individual objects

https://github.com/iterative/dvc/blob/18e8f075728bcb6f0d4bbff1b8a100a45ca1b88c/dvc/remote/base.py#L923-L928

The initial weight multiplier value (`20`) can be lowered now, due to the subsequent remote improvements that have come since the initial traverse/no traverse optimization PR


Discussion:

Resolving patch:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -95,7 +95,7 @@ class BaseRemote(object):
     DEFAULT_CACHE_TYPES = ["copy"]
     DEFAULT_VERIFY = False
     LIST_OBJECT_PAGE_SIZE = 1000
-    TRAVERSE_WEIGHT_MULTIPLIER = 20
+    TRAVERSE_WEIGHT_MULTIPLIER = 5
     TRAVERSE_PREFIX_LEN = 3
     TRAVERSE_THRESHOLD_SIZE = 500000
     CAN_TRAVERSE = True


Test patch:
Repository: iterative/dvc

Issue:
Importing ABC from collections has been deprecated and will be removed in Python 3.10
Use collections,abc instead since the supported versions are python >= 3.5

```
dvc/stage/loader.py
3:from collections import Mapping, defaultdict
```


Discussion:

Resolving patch:
diff --git a/dvc/stage/loader.py b/dvc/stage/loader.py
--- a/dvc/stage/loader.py
+++ b/dvc/stage/loader.py
@@ -1,6 +1,7 @@
 import logging
 import os
-from collections import Mapping, defaultdict
+from collections import defaultdict
+from collections.abc import Mapping
 from copy import deepcopy
 from itertools import chain
 


Test patch:
Repository: iterative/dvc

Issue:
`dvc import -o` fails  when source is location within same project
env: dvc v0.94, pip, Ubuntu

When I run the following dvc command:

    dvc import -o my/target/loc/ . some/long/path/to/dir

I get this error message: 

```
ERROR: failed to import 'some/long/path/to/dir' from '.'. - The path '../../../some/long/path/to/dir' does not exist in the target repository '.' neither as a DVC output nor\
 as a Git-tracked file
```
I tracked the issue down to [this line](https://github.com/iterative/dvc/blob/ce946d11609597f3b1c6c05e61f2701ca8f43f5c/dvc/repo/imp_url.py#L21), where a relative prefix is incorrectly being appended to the `url`.

```python
# NOTE: when user is importing something from within his own repository
if os.path.exists(url) and path_isin(os.path.abspath(url), self.root_dir):
    url = relpath(url, wdir)
```


I'll try to put together a PR
`dvc import -o` fails  when source is location within same project
env: dvc v0.94, pip, Ubuntu

When I run the following dvc command:

    dvc import -o my/target/loc/ . some/long/path/to/dir

I get this error message: 

```
ERROR: failed to import 'some/long/path/to/dir' from '.'. - The path '../../../some/long/path/to/dir' does not exist in the target repository '.' neither as a DVC output nor\
 as a Git-tracked file
```
I tracked the issue down to [this line](https://github.com/iterative/dvc/blob/ce946d11609597f3b1c6c05e61f2701ca8f43f5c/dvc/repo/imp_url.py#L21), where a relative prefix is incorrectly being appended to the `url`.

```python
# NOTE: when user is importing something from within his own repository
if os.path.exists(url) and path_isin(os.path.abspath(url), self.root_dir):
    url = relpath(url, wdir)
```


I'll try to put together a PR


Discussion:

Resolving patch:
diff --git a/dvc/repo/imp_url.py b/dvc/repo/imp_url.py
--- a/dvc/repo/imp_url.py
+++ b/dvc/repo/imp_url.py
@@ -17,8 +17,12 @@ def imp_url(self, url, out=None, fname=None, erepo=None, locked=True):
     out = resolve_output(url, out)
     path, wdir, out = resolve_paths(self, out)
 
-    # NOTE: when user is importing something from within his own repository
-    if os.path.exists(url) and path_isin(os.path.abspath(url), self.root_dir):
+    # NOTE: when user is importing something from within their own repository
+    if (
+        erepo is None
+        and os.path.exists(url)
+        and path_isin(os.path.abspath(url), self.root_dir)
+    ):
         url = relpath(url, wdir)
 
     stage = create_stage(


Test patch:
diff --git a/tests/func/test_import.py b/tests/func/test_import.py
--- a/tests/func/test_import.py
+++ b/tests/func/test_import.py
@@ -342,3 +342,9 @@ def test_import_pipeline_tracked_outs(
             "file:///{}".format(fspath(tmp_dir)), "bar", out="baz"
         )
         assert (erepo_dir / "baz").read_text() == "foo"
+
+
+def test_local_import(tmp_dir, dvc, scm):
+    tmp_dir.dvc_gen("foo", "foo", commit="init")
+    (tmp_dir / "outdir").mkdir()
+    dvc.imp(".", "foo", out="outdir")
Repository: iterative/dvc

Issue:
End of file fixer
I am using an [end-of-file fixer in the pre-commit hook](https://pre-commit.com/hooks.html). It checks that the file ends with an empty new line.

It looks like files

```
	modified:   .dvc/plots/confusion.json
	modified:   .dvc/plots/default.json
	modified:   .dvc/plots/scatter.json

```

That are automatically created by `dvc init` do not have an empty line at the end of the file.


Discussion:
@ternaus Thanks! Feel free to contribute the PR, we'll really appreciate it! :pray: 

Resolving patch:
diff --git a/dvc/repo/plots/template.py b/dvc/repo/plots/template.py
--- a/dvc/repo/plots/template.py
+++ b/dvc/repo/plots/template.py
@@ -59,6 +59,7 @@ def dump(self):
                 indent=self.INDENT,
                 separators=self.SEPARATORS,
             )
+            fobj.write("\n")
 
     @staticmethod
     def get_data_anchor(template_content):


Test patch:
Repository: iterative/dvc

Issue:
Plot: link to docs
Link to docs as per @jorgeorpinel review under https://github.com/iterative/dvc/pull/3577, after plot documentation is merged from https://github.com/iterative/dvc.org/pull/1186.


Discussion:
Thanks. I did start doing this in #3759 and mentioned it in https://github.com/iterative/dvc.org/pull/1186#pullrequestreview-407626236 but having the issue also doesn't hurt 🙂 

Resolving patch:
diff --git a/dvc/command/plots.py b/dvc/command/plots.py
--- a/dvc/command/plots.py
+++ b/dvc/command/plots.py
@@ -4,6 +4,7 @@
 
 from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
 from dvc.exceptions import DvcException
+from dvc.utils import format_link
 
 logger = logging.getLogger(__name__)
 
@@ -123,7 +124,12 @@ def add_parser(subparsers, parent_parser):
         "--template",
         nargs="?",
         default=None,
-        help="File to be injected with data.",
+        help=(
+            "Special JSON or HTML schema file to inject with the data. "
+            "See {}".format(
+                format_link("https://man.dvc.org/plots#plot-templates")
+            )
+        ),
     )
     plots_show_parser.add_argument(
         "-o", "--out", default=None, help="Destination path to save plots to.",
@@ -182,7 +188,12 @@ def add_parser(subparsers, parent_parser):
         "--template",
         nargs="?",
         default=None,
-        help="File to be injected with data.",
+        help=(
+            "Special JSON or HTML schema to inject with the data. "
+            "See {}".format(
+                format_link("https://man.dvc.org/plots#plot-templates")
+            )
+        ),
     )
     plots_diff_parser.add_argument(
         "--targets",


Test patch: