internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: iterative/dvc

Issue:
Deduplicate files in `git add` command suggestion
0.23.2

When running a `dvc run` with multiple outputs it prints `git add` with `.gitignore` repeated multiple times.

![ss](https://user-images.githubusercontent.com/3659196/50811811-f4b67900-12c4-11e9-824b-0df8905824ab.png)





Discussion:

Resolving patch:
diff --git a/dvc/repo/scm_context.py b/dvc/repo/scm_context.py
--- a/dvc/repo/scm_context.py
+++ b/dvc/repo/scm_context.py
@@ -4,6 +4,7 @@ def run(repo, *args, **kw):
             result = method(repo, *args, **kw)
             repo.scm.reset_ignores()
             repo.scm.remind_to_track()
+            repo.scm.reset_tracked_files()
             return result
         except Exception:
             repo.scm.cleanup_ignores()
diff --git a/dvc/scm/base.py b/dvc/scm/base.py
--- a/dvc/scm/base.py
+++ b/dvc/scm/base.py
@@ -126,6 +126,11 @@ def reset_ignores(self):
         Method to reset in-memory ignore storing mechanism.
         """
 
+    def reset_tracked_files(self):
+        """
+        Method to reset in-memory tracked files storing mechanism.
+        """
+
     def remind_to_track(self):
         """
         Method to remind user to track newly created files handled by scm
diff --git a/dvc/scm/git/__init__.py b/dvc/scm/git/__init__.py
--- a/dvc/scm/git/__init__.py
+++ b/dvc/scm/git/__init__.py
@@ -54,7 +54,7 @@ def __init__(self, root_dir=os.curdir, repo=None):
         self.git.git.update_environment(LD_LIBRARY_PATH=libpath)
 
         self.ignored_paths = []
-        self.files_to_track = []
+        self.files_to_track = set()
 
     @staticmethod
     def is_repo(root_dir):
@@ -245,6 +245,9 @@ def cleanup_ignores(self):
     def reset_ignores(self):
         self.ignored_paths = []
 
+    def reset_tracked_files(self):
+        self.files_to_track = set()
+
     def remind_to_track(self):
         if not self.files_to_track:
             return
@@ -257,7 +260,7 @@ def remind_to_track(self):
         )
 
     def track_file(self, path):
-        self.files_to_track.append(path)
+        self.files_to_track.add(path)
 
     def belongs_to_scm(self, path):
         basename = os.path.basename(path)


Test patch:
Repository: iterative/dvc

Issue:
Metrics show xpath does not override dvcfile xpath

I do have a Dvcfile with metrics, and for some period of time (a few commits) I defined a specific xpath in that file; Now, when I run a different xpath on `dvc metrics show`, the old xpath (and hense, different metric) shows up for the commits where dvcfile specified it:

```
>>> dvc metric show -x pe20

working tree:
        data/rentals/nyc/metrics.json: [88.15]
bin_bedroom:
        data/rentals/nyc/metrics.json: [88.25]
first-iteration:
        data/rentals/nyc/metrics.json: [88.13]
nhds:
        data/rentals/nyc/metrics.json: [7.45]
roosevelt:
        data/rentals/nyc/metrics.json: [7.11]
``` 

obviosly pe20 was > 88 for all the commits, it's just showing median error for the `nhds` and `roosevelt`



Discussion:
Thanks for reporting @Casyfill ! Indeed, we need to override it. We'll take a look.
You probably can instead just check for the xpath in the current Dvcfile, and override it with the explicit path, if needed? 
@Casyfill Yes, the explicitly specified xpath should override the one specified in a dvc file.
@efiop @pared metric type should probably behave the same way.

also, what about multiple metric files? or a single file fetched from different tags? now, you can have them with different settings, and explicitly specified xpath and type are essentially defaults if one of the files is not specifying them. That will change and we'll be applying same settings to all files.

I can actually imagine both options are valuable.
it looks like right now if you specify type + xpath or just type they will override setting in the DVC-file. xpath alone is not enough.
@efiop @pared reopening to fix comments, logic if needed and fix the docs appropriately.

Resolving patch:
diff --git a/dvc/repo/metrics/show.py b/dvc/repo/metrics/show.py
--- a/dvc/repo/metrics/show.py
+++ b/dvc/repo/metrics/show.py
@@ -188,16 +188,15 @@ def _collect_metrics(repo, path, recursive, typ, xpath, branch):
         if not o.metric:
             continue
 
-        # NOTE this case assumes that metric has been modified with
-        # `dvc metric modify -t {} -x {}`, so stage file should contain
-        # info what type metric is and what is its xpath
+        # NOTE this case assumes that typ has not been provided in CLI call
+        # and one of the following cases:
+        # - stage file contains metric type
+        # - typ will be read from file extension later
         if not typ and isinstance(o.metric, dict):
             t = o.metric.get(o.PARAM_METRIC_TYPE, typ)
             # NOTE user might want to check different xpath, hence xpath first
             x = xpath or o.metric.get(o.PARAM_METRIC_XPATH)
         else:
-            # NOTE if metric has not been modified, user needs to provide
-            # both metric type and xpath
             t = typ
             x = xpath
 


Test patch:
Repository: iterative/dvc

Issue:
status: change nothing to reproduce message
If I use DVC only to version data/models and don't care about pipelines, this message:

`Pipelines are up to date. Nothing to reproduce.` 

looks really strange.

Let's change it to something more generic:

`Data and pipelines are up to date.` 

or something similar



Discussion:

Resolving patch:
diff --git a/dvc/command/status.py b/dvc/command/status.py
--- a/dvc/command/status.py
+++ b/dvc/command/status.py
@@ -12,7 +12,7 @@
 class CmdDataStatus(CmdDataBase):
     STATUS_LEN = 20
     STATUS_INDENT = "\t"
-    UP_TO_DATE_MSG = "Pipelines are up to date. Nothing to reproduce."
+    UP_TO_DATE_MSG = "Data and pipelines are up to date."
 
     def _normalize(self, s):
         s += ":"


Test patch:
Repository: iterative/dvc

Issue:
update `url` argument description in `get` and `import` help output
to `URL of Git repository with DVC project to download from.`

per https://github.com/iterative/dvc.org/pull/464#pullrequestreview-259776737


Discussion:

Resolving patch:
diff --git a/dvc/command/get.py b/dvc/command/get.py
--- a/dvc/command/get.py
+++ b/dvc/command/get.py
@@ -40,7 +40,7 @@ def add_parser(subparsers, parent_parser):
         formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     get_parser.add_argument(
-        "url", help="DVC repository URL to download data from."
+        "url", help="URL of Git repository with DVC project to download from."
     )
     get_parser.add_argument("path", help="Path to data within DVC repository.")
     get_parser.add_argument(
diff --git a/dvc/command/imp.py b/dvc/command/imp.py
--- a/dvc/command/imp.py
+++ b/dvc/command/imp.py
@@ -41,7 +41,9 @@ def add_parser(subparsers, parent_parser):
         help=IMPORT_HELP,
         formatter_class=argparse.RawTextHelpFormatter,
     )
-    import_parser.add_argument("url", help="DVC repository URL.")
+    import_parser.add_argument(
+        "url", help="URL of Git repository with DVC project to download from."
+    )
     import_parser.add_argument(
         "path", help="Path to data within DVC repository."
     )


Test patch:
Repository: iterative/dvc

Issue:
Update azure-storage-blob package to version 2.1.0
Mainly that's required to support conda-forge distribution channel in anaconda package manager.
Currently used version 2.0.1 of azure-storage-blob is missed in conda-forge https://anaconda.org/conda-forge/azure-storage-blob


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -72,7 +72,7 @@ def run(self):
 # Extra dependencies for remote integrations
 gs = ["google-cloud-storage==1.13.0"]
 s3 = ["boto3==1.9.115"]
-azure = ["azure-storage-blob==2.0.1"]
+azure = ["azure-storage-blob==2.1.0"]
 oss = ["oss2==2.6.1"]
 ssh = ["paramiko>=2.5.0"]
 hdfs = ["pyarrow==0.14.0"]


Test patch:
Repository: iterative/dvc

Issue:
git checkout some-file causes dvc to try to remove a dvc-managed file
When I `git checkout some-file` to clean away my local changes to that file, the dvc hook butts in and says:

> ERROR: unexpected error - unable to remove 'data/qix/1a285cea-7b71-4492-9b4d-87092bfb9869.json.gz' without a confirmation from the user. Use '-f' to force.

But I did not request removal of that file, and I don't understand why DVC is trying to do that.

Note that I'm _not_ switching to a different branch/tag/commit, because I specified a path to `git checkout`. So I'm only modifying files in my working tree. I don't think that should trigger DVC to [try to] remove files.

Platform: macOS, Python 3.7.4 virtualenv installed using `pyenv`, and:
```bash
$ dvc --version
0.54.1+5e4d26
```


Discussion:
@gthb Thanks for reporting this! Looks like we are not using post-checkout parameters that are being passed to us by git and are indicating if we are checking out a specific file or a branch/tag/etc. https://git-scm.com/docs/githooks#_post_checkout We need to adjust our hook accordingly.
@gthb Looks like the third parameter is just what we need. You could try modifying your git hook by hand to see if that will do the job for you. If it does, please let us know, and maybe consider submitting a PR, we really appreciate your contributions :slightly_smiling_face: If it doesn't work, please ping us as well.

Resolving patch:
diff --git a/dvc/scm/git/__init__.py b/dvc/scm/git/__init__.py
--- a/dvc/scm/git/__init__.py
+++ b/dvc/scm/git/__init__.py
@@ -216,7 +216,11 @@ def list_tags(self):
         return [t.name for t in self.repo.tags]
 
     def _install_hook(self, name, cmd):
-        command = '[ -z "$(git ls-files .dvc)" ] || exec dvc {}'.format(cmd)
+        command = (
+            '[ "$3" == "0" ]'
+            ' || [ -z "$(git ls-files .dvc)" ]'
+            " || exec dvc {}".format(cmd)
+        )
 
         hook = os.path.join(self.root_dir, self.GIT_DIR, "hooks", name)
 


Test patch:
diff --git a/tests/func/test_install.py b/tests/func/test_install.py
--- a/tests/func/test_install.py
+++ b/tests/func/test_install.py
@@ -39,7 +39,9 @@ def test_should_append_hooks_if_file_already_exists(self, git, dvc_repo):
         expected_script = (
             "#!/bin/sh\n"
             "echo hello\n"
-            '[ -z "$(git ls-files .dvc)" ] || exec dvc checkout\n'
+            '[ "$3" == "0" ]'
+            ' || [ -z "$(git ls-files .dvc)" ]'
+            " || exec dvc checkout\n"
         )
 
         with open(self._hook("post-checkout"), "r") as fobj:
@@ -51,7 +53,9 @@ def test_should_be_idempotent(self, git, dvc_repo):
 
         expected_script = (
             "#!/bin/sh\n"
-            '[ -z "$(git ls-files .dvc)" ] || exec dvc checkout\n'
+            '[ "$3" == "0" ]'
+            ' || [ -z "$(git ls-files .dvc)" ]'
+            " || exec dvc checkout\n"
         )
 
         with open(self._hook("post-checkout"), "r") as fobj:
Repository: iterative/dvc

Issue:
allow setting ACL for s3
we work across two AWS accounts, one that stores the data and one where I run dvc. I need to set `--acl bucket-owner-full-control` when writing to S3

PR incoming.

thread: https://discordapp.com/channels/485586884165107732/485596304961962003/613152718755987488


Discussion:

Resolving patch:
diff --git a/dvc/config.py b/dvc/config.py
--- a/dvc/config.py
+++ b/dvc/config.py
@@ -210,6 +210,7 @@ class Config(object):  # pylint: disable=too-many-instance-attributes
     SECTION_AWS_PROFILE = "profile"
     SECTION_AWS_USE_SSL = "use_ssl"
     SECTION_AWS_SSE = "sse"
+    SECTION_AWS_ACL = "acl"
     SECTION_AWS_SCHEMA = {
         SECTION_AWS_STORAGEPATH: str,
         Optional(SECTION_AWS_REGION): str,
@@ -219,6 +220,7 @@ class Config(object):  # pylint: disable=too-many-instance-attributes
         Optional(SECTION_AWS_LIST_OBJECTS, default=False): BOOL_SCHEMA,
         Optional(SECTION_AWS_USE_SSL, default=True): BOOL_SCHEMA,
         Optional(SECTION_AWS_SSE): str,
+        Optional(SECTION_AWS_ACL): str,
     }
 
     # backward compatibility
@@ -261,6 +263,7 @@ class Config(object):  # pylint: disable=too-many-instance-attributes
         Optional(SECTION_AWS_LIST_OBJECTS, default=False): BOOL_SCHEMA,
         Optional(SECTION_AWS_USE_SSL, default=True): BOOL_SCHEMA,
         Optional(SECTION_AWS_SSE): str,
+        Optional(SECTION_AWS_ACL): str,
         Optional(SECTION_GCP_PROJECTNAME): str,
         Optional(SECTION_CACHE_TYPE): SECTION_CACHE_TYPE_SCHEMA,
         Optional(SECTION_CACHE_PROTECTED, default=False): BOOL_SCHEMA,
diff --git a/dvc/remote/s3.py b/dvc/remote/s3.py
--- a/dvc/remote/s3.py
+++ b/dvc/remote/s3.py
@@ -65,6 +65,10 @@ def __init__(self, repo, config):
         if self.sse:
             self.extra_args["ServerSideEncryption"] = self.sse
 
+        self.acl = config.get(Config.SECTION_AWS_ACL, "")
+        if self.acl:
+            self.extra_args["ACL"] = self.acl
+
         shared_creds = config.get(Config.SECTION_AWS_CREDENTIALPATH)
         if shared_creds:
             os.environ.setdefault("AWS_SHARED_CREDENTIALS_FILE", shared_creds)


Test patch:
Repository: iterative/dvc

Issue:
Doing `commit` after `move`
Hi, after using `dvc move` to move versioned data files, when doing `dvc status` one gets `changed checksum` so these files need to be `dvc commit`. I suggest combining `move+commit` because this is obviously what we want to do. 


Discussion:
It looks reasonable to me. @efiop do you think there is some use case of move that does not require commiting?  
@pared I'm actually surprised that it doesn't, this is probably a bug.
I'll do it. I'll add a `commit` step by default with a possible flag `--no-commit`. 
@danielfleischer I think we are simply missing `stage.save()` in https://github.com/iterative/dvc/blob/master/dvc/repo/move.py#L68

Resolving patch:
diff --git a/dvc/repo/move.py b/dvc/repo/move.py
--- a/dvc/repo/move.py
+++ b/dvc/repo/move.py
@@ -65,5 +65,6 @@ def move(self, from_path, to_path):
 
     with self.state:
         out.move(to_out)
+        stage.save()
 
     stage.dump()


Test patch:
diff --git a/tests/func/test_move.py b/tests/func/test_move.py
--- a/tests/func/test_move.py
+++ b/tests/func/test_move.py
@@ -192,3 +192,11 @@ def test(self):
         dvc_fullpath = os.path.join(self.DATA_DIR, "data.txt.dvc")
         self.assertTrue(os.path.exists(data_fullpath))
         self.assertTrue(os.path.exists(dvc_fullpath))
+
+
+def test_move_should_save_stage_info(dvc_repo, repo_dir):
+    dvc_repo.add(repo_dir.DATA_DIR)
+
+    dvc_repo.move(repo_dir.DATA_DIR, "new_name")
+
+    assert dvc_repo.status() == {}
Repository: iterative/dvc

Issue:
dvc remove CLI documentation inconsistency
`dvc remove` (without `targets`) prints help which states that `targets` are optional, and if not specified will remove all DVC-files. Clearly not the case.

```bash
$ dvc remove
[...]
  targets        DVC-files to remove. Optional. (Finds all DVC-files in the
                 workspace by default.)
```


Discussion:
@jorgeorpinel This slipped-in in 936f941bf and will require doc modification. Could you take a look, please?

Resolving patch:
diff --git a/dvc/command/remove.py b/dvc/command/remove.py
--- a/dvc/command/remove.py
+++ b/dvc/command/remove.py
@@ -74,9 +74,6 @@ def add_parser(subparsers, parent_parser):
         help="Force purge.",
     )
     remove_parser.add_argument(
-        "targets",
-        nargs="+",
-        help="DVC-files to remove. Optional. "
-        "(Finds all DVC-files in the workspace by default.)",
+        "targets", nargs="+", help="DVC-files to remove."
     )
     remove_parser.set_defaults(func=CmdRemove)


Test patch:
Repository: iterative/dvc

Issue:
Dvc add data unprotect files for each use
My DVC version 
![image](https://user-images.githubusercontent.com/20395242/59399222-dac61880-8dc5-11e9-88db-c5fc4e232d56.png)

This issue comes from **Share NAS data in serve**r in the [discuss post 11#](https://discuss.dvc.org/t/share-nas-data-in-server/180/11?u=jimmy15923)

When adding new data, dvc will unprotect all data which cost lots of time (about 20 mins) which seems inevitable.

![image](https://user-images.githubusercontent.com/20395242/59399331-2aa4df80-8dc6-11e9-9aa2-f0c2ef9d4b01.png)

Dvc add data unprotect files for each use
My DVC version 
![image](https://user-images.githubusercontent.com/20395242/59399222-dac61880-8dc5-11e9-88db-c5fc4e232d56.png)

This issue comes from **Share NAS data in serve**r in the [discuss post 11#](https://discuss.dvc.org/t/share-nas-data-in-server/180/11?u=jimmy15923)

When adding new data, dvc will unprotect all data which cost lots of time (about 20 mins) which seems inevitable.

![image](https://user-images.githubusercontent.com/20395242/59399331-2aa4df80-8dc6-11e9-9aa2-f0c2ef9d4b01.png)



Discussion:
+1

I unprotected and changed a single file in a directory. dvc add for this directory makes unnecessary unprotection and copy of other files.
+1

I unprotected and changed a single file in a directory. dvc add for this directory makes unnecessary unprotection and copy of other files.

Resolving patch:
diff --git a/dvc/stage.py b/dvc/stage.py
--- a/dvc/stage.py
+++ b/dvc/stage.py
@@ -505,8 +505,6 @@ def create(repo, **kwargs):
             stage.remove_outs(ignore_remove=False)
             logger.warning("Build cache is ignored when using --remove-outs.")
             ignore_build_cache = True
-        else:
-            stage.unprotect_outs()
 
         if os.path.exists(path) and any(out.persist for out in stage.outs):
             logger.warning("Build cache is ignored when persisting outputs.")


Test patch:
diff --git a/tests/func/test_add.py b/tests/func/test_add.py
--- a/tests/func/test_add.py
+++ b/tests/func/test_add.py
@@ -12,6 +12,7 @@
 import logging
 import colorama
 
+from dvc.remote import RemoteLOCAL
 from dvc.system import System
 from mock import patch
 
@@ -535,3 +536,20 @@ def test_windows_should_add_when_cache_on_different_drive(
 
     ret = main(["add", repo_dir.DATA])
     assert ret == 0
+
+
+def test_readding_dir_should_not_unprotect_all(dvc_repo, repo_dir):
+    dvc_repo.cache.local.cache_types = ["symlink"]
+    dvc_repo.cache.local.protected = True
+
+    dvc_repo.add(repo_dir.DATA_DIR)
+    new_file = os.path.join(repo_dir.DATA_DIR, "new_file")
+
+    repo_dir.create(new_file, "new_content")
+
+    unprotect_spy = spy(RemoteLOCAL.unprotect)
+    with patch.object(RemoteLOCAL, "unprotect", unprotect_spy):
+        dvc_repo.add(repo_dir.DATA_DIR)
+
+    assert not unprotect_spy.mock.called
+    assert System.is_symlink(new_file)
Repository: iterative/dvc

Issue:
dvc get: confusing message if file already exists
v 0.58.1

```
$ mkdir tmp
$ cd tmp
$ dvc get https://github.com/iterative/example-get-started model.pkl
$ dvc get https://github.com/iterative/example-get-started model.pkl
ERROR: unexpected error - 'NoneType' object is not callable

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
```


Discussion:

Resolving patch:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -771,7 +771,10 @@ def checkout(
             skip = True
 
         if skip:
-            progress_callback(str(path_info), self.get_files_number(checksum))
+            if progress_callback:
+                progress_callback(
+                    str(path_info), self.get_files_number(checksum)
+                )
             return
 
         msg = "Checking out '{}' with cache '{}'."


Test patch:
Repository: iterative/dvc

Issue:
get: error while fetching DVC-file
`dvc get` is to download data artifacts, but if the user hasn't this clear, they might try to `get` a DVC-file and the error is not that clear.

```
❯ dvc get https://github.com/iterative/dvc scripts/innosetup/dvc.ico.dvc
ERROR: failed to get 'scripts/innosetup/dvc.ico.dvc' from 'https://github.com/iterative/dvc' - unable to find DVC-file with output '../../../tmp/tmpbzwhrmqcdvc-repo/scripts/innosetup/dvc.ico.dvc'
```

I suggest showing a message to guard from this behavior:
```
ERROR: failed to get 'scripts/innosetup/dvc.ico.dvc' from 'https://github.com/iterative/dvc' - this is a DVC-file, you must specify the path of an output
```


Discussion:

Resolving patch:
diff --git a/dvc/exceptions.py b/dvc/exceptions.py
--- a/dvc/exceptions.py
+++ b/dvc/exceptions.py
@@ -286,3 +286,11 @@ def __init__(self, url):
         super(UrlNotDvcRepoError, self).__init__(
             "URL '{}' is not a dvc repository.".format(url)
         )
+
+
+class GetDVCFileError(DvcException):
+    def __init__(self):
+        super(GetDVCFileError, self).__init__(
+            "the given path is a DVC-file, you must specify a data file "
+            "or a directory"
+        )
diff --git a/dvc/repo/get.py b/dvc/repo/get.py
--- a/dvc/repo/get.py
+++ b/dvc/repo/get.py
@@ -2,8 +2,9 @@
 import os
 import shortuuid
 
+from dvc.stage import Stage
 from dvc.config import Config
-from dvc.exceptions import NotDvcRepoError, UrlNotDvcRepoError
+from dvc.exceptions import NotDvcRepoError, UrlNotDvcRepoError, GetDVCFileError
 from dvc.path_info import PathInfo
 from dvc.external_repo import external_repo
 from dvc.state import StateNoop
@@ -17,6 +18,9 @@
 def get(url, path, out=None, rev=None):
     out = out or os.path.basename(urlparse(path).path)
 
+    if Stage.is_valid_filename(out):
+        raise GetDVCFileError()
+
     # Creating a directory right beside the output to make sure that they
     # are on the same filesystem, so we could take the advantage of
     # reflink and/or hardlink. Not using tempfile.TemporaryDirectory


Test patch:
diff --git a/tests/func/test_get.py b/tests/func/test_get.py
--- a/tests/func/test_get.py
+++ b/tests/func/test_get.py
@@ -3,7 +3,7 @@
 
 import pytest
 
-from dvc.exceptions import UrlNotDvcRepoError
+from dvc.exceptions import UrlNotDvcRepoError, GetDVCFileError
 from dvc.repo import Repo
 
 from tests.utils import trees_equal
@@ -46,3 +46,8 @@ def test_get_repo_rev(repo_dir, erepo):
 def test_get_from_non_dvc_repo(git_erepo):
     with pytest.raises(UrlNotDvcRepoError):
         Repo.get(git_erepo.root_dir, "some_file.zip")
+
+
+def test_get_a_dvc_file(repo_dir, erepo):
+    with pytest.raises(GetDVCFileError):
+        Repo.get(erepo.root_dir, "some_file.dvc")