internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: iterative/dvc

Issue:
'dvc add' fails on files encoded in Latin-1
Running 'dvc add TEXTFILE' fails if TEXTFILE is encoded in Latin-1. The following error message is shown: 

Unexpected error: 'utf8' codec can't decode byte 0xa3 in position 12204: invalid start byte

The error disapears if TEXTFILE is saved with encoding UTF-8 before running 'dvc add'. So does dvc only support files encoded in UTF-8?


Discussion:
Hi @noeddl !

Which dvc version are you using?
Nevermind, I was able to reproduce on the recent versions as well. Working on the fix right now. Thanks for the report!
Great, thanks for the quick reaction!

Resolving patch:
diff --git a/dvc/utils.py b/dvc/utils.py
--- a/dvc/utils.py
+++ b/dvc/utils.py
@@ -20,7 +20,7 @@
 
 
 def dos2unix(data):
-    return '\n'.join(data.split('\r\n'))
+    return data.replace(b'\r\n', b'\n')
 
 
 def file_md5(fname):
@@ -35,12 +35,7 @@ def file_md5(fname):
         else:
             binary = is_binary(fname)
 
-        if binary:
-            mode = "rb"
-        else:
-            mode = "r"
-
-        with open(fname, mode) as fobj:
+        with open(fname, 'rb') as fobj:
             while True:
                 data = fobj.read(LOCAL_CHUNK_SIZE)
                 if not data:
@@ -49,9 +44,7 @@ def file_md5(fname):
                 if binary:
                     chunk = data
                 else:
-                    if sys.version_info[0] == 2:
-                        data = data.decode('utf-8')
-                    chunk = dos2unix(data).encode('utf-8')
+                    chunk = dos2unix(data)
 
                 hash_md5.update(chunk)
 


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -37,3 +37,11 @@ def f(target):
 
         for n in n_threads:
             utils.map_progress(f, targets, n)
+
+    def test_file_md5_crlf(self):
+        with open('cr', 'wb+') as fd:
+            fd.write(b'a\nb\nc')
+        with open('crlf', 'wb+') as fd:
+            fd.write(b'a\r\nb\r\nc')
+
+        self.assertEqual(utils.file_md5('cr')[0], utils.file_md5('crlf')[0])
Repository: iterative/dvc

Issue:
Add option for `dvc remote add` to set default remote
Currently when you setup first remote for your repo, you need to manually set it as a default one in a config:
```
dvc remote add myremote s3://mybucket/
dvc config core.remote myremote
```
Lets just do it automatically by providing -d|--default option for `dvc remote add`.


Discussion:

Resolving patch:
diff --git a/dvc/cli.py b/dvc/cli.py
--- a/dvc/cli.py
+++ b/dvc/cli.py
@@ -332,6 +332,11 @@ def parse_args(argv=None):
                         action='store_true',
                         default=False,
                         help='Use local config')
+    remote_add_parser.add_argument('-d',
+                        '--default',
+                        action='store_true',
+                        default=False,
+                        help='Set as default remote')
     remote_add_parser.set_defaults(func=CmdRemoteAdd)
 
 
diff --git a/dvc/command/remote.py b/dvc/command/remote.py
--- a/dvc/command/remote.py
+++ b/dvc/command/remote.py
@@ -8,8 +8,15 @@
 class CmdRemoteAdd(CmdConfig):
     def run(self):
         section = Config.SECTION_REMOTE_FMT.format(self.args.name)
-        return self.set(section, Config.SECTION_REMOTE_URL, self.args.url)
+        ret = self.set(section, Config.SECTION_REMOTE_URL, self.args.url)
+        if ret != 0:
+            return ret
 
+        if self.args.default:
+            Logger.info('Setting \'{}\' as a default remote.'.format(self.args.name))
+            ret = self.set(Config.SECTION_CORE, Config.SECTION_CORE_REMOTE, self.args.name)
+
+        return ret
 
 class CmdRemoteRemove(CmdConfig):
     def run(self):


Test patch:
diff --git a/tests/test_remote.py b/tests/test_remote.py
--- a/tests/test_remote.py
+++ b/tests/test_remote.py
@@ -11,7 +11,7 @@ def test(self):
         self.assertNotEqual(main(['remote', 'remove', remotes[0]]), 0)
 
         for r in remotes:
-            self.assertEqual(main(['remote', 'add', r, 's3://bucket/name']), 0)
+            self.assertEqual(main(['remote', 'add', '--default', r, 's3://bucket/name']), 0)
 
         self.assertEqual(main(['remote', 'list']), 0)
Repository: iterative/dvc

Issue:
Use dvc to store images for windows installer in our repo
We currently have 3 images stored in git in scripts/innosetup. We should start using dvc for them. Required for https://github.com/iterative/dvc/issues/735 . Depends on https://github.com/iterative/dvc/issues/785 .


Discussion:

Resolving patch:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -1,6 +1,7 @@
 import os
 import re
 import tempfile
+import posixpath
 
 from dvc.config import Config
 from dvc.logger import Logger
@@ -66,7 +67,7 @@ def cache_file_key(self, fname):
         """ Key of a file within the bucket """
         relpath = os.path.relpath(fname, self.project.cache.local.cache_dir)
         relpath = relpath.replace('\\', '/')
-        return '{}/{}'.format(self.prefix, relpath).strip('/')
+        return posixpath.join(self.prefix, relpath).strip('/')
 
     def cache_key_name(self, path):
         relpath = os.path.relpath(path, self.project.cache.local.cache_dir)


Test patch:
Repository: iterative/dvc

Issue:
cloud: collect: keep downloaded dir cache file
Currently we download it every time, which results in longer `dvc status -c/push/pull/fetch` work time.


Discussion:
Related to #758 

Resolving patch:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -96,24 +96,22 @@ def collect(self, arg):
 
         path, local = arg
         ret = [path]
+        md5 = self.project.cache.local.path_to_md5(path)
 
         if not RemoteLOCAL.is_dir_cache(path):
             return ret
 
-        if local:
-            if not os.path.isfile(path):
-                return ret
-            dir_path = path
-        else:
+        if not local and self.project.cache.local.changed(md5):
             path_info = self._get_path_info(path)
             if not path_info:
                 Logger.debug("File '{}' does not exist in the cloud".format(path))
                 return ret
-            tmp = os.path.join(tempfile.mkdtemp(), os.path.basename(path))
-            self.download(path_info, tmp, no_progress_bar=True)
-            dir_path = tmp
+            self.download(path_info, path, no_progress_bar=True)
+
+        if self.project.cache.local.changed(md5):
+            return ret
 
-        for relpath, md5 in RemoteLOCAL.get_dir_cache(dir_path).items():
+        for relpath, md5 in RemoteLOCAL.get_dir_cache(path).items():
             cache = self.project.cache.local.get(md5)
             ret.append(cache)
 


Test patch:
Repository: iterative/dvc

Issue:
Checkout single file when pulling single file
Hi guys, I noticed `self.checkout()` of whole project is performed even when pulling single target, is this expected behavior? I would expect `self.checkout(target=target)`.

Cheers,
David


Discussion:
I'd say that's probably a harmless absence of optimization. Is there a scenario where that gives you trouble?
Hi @prihoda ! Great catch! You are right, we should not checkout the whole repo when only one target is pulled. I will prepare a fix for it soon. Meanwhile, you can work around it by using:
```
dvc fetch target.dvc
dvc checkout target.dvc
```

Thank you!

Resolving patch:
diff --git a/dvc/project.py b/dvc/project.py
--- a/dvc/project.py
+++ b/dvc/project.py
@@ -295,7 +295,7 @@ def fetch(self, target=None, jobs=1, remote=None, all_branches=False):
 
     def pull(self, target=None, jobs=1, remote=None, all_branches=False):
         self.fetch(target, jobs, remote=remote, all_branches=all_branches)
-        self.checkout()
+        self.checkout(target=target)
 
     def _local_status(self, target=None):
         status = {}


Test patch:
Repository: iterative/dvc

Issue:
Allow specifying ordered list of link methods in config's cache.type
I.e. ['reflink', 'copy'] or smth like that.


Discussion:

Resolving patch:
diff --git a/dvc/config.py b/dvc/config.py
--- a/dvc/config.py
+++ b/dvc/config.py
@@ -19,6 +19,15 @@ def supported_url(url):
     return supported(url)
 
 
+def supported_cache_type(types):
+    if isinstance(types, str):
+        types = [t.strip() for t in types.split(',')]
+    for t in types:
+        if t not in ['reflink', 'hardlink', 'symlink', 'copy']:
+            return False
+    return True
+
+
 class Config(object):
     CONFIG = 'config'
     CONFIG_LOCAL = 'config.local'
@@ -31,7 +40,7 @@ class Config(object):
     SECTION_CACHE = 'cache'
     SECTION_CACHE_DIR = 'dir'
     SECTION_CACHE_TYPE = 'type'
-    SECTION_CACHE_TYPE_SCHEMA = schema.And(schema.Use(str.lower), lambda t: t in ('reflink', 'hardlink', 'symlink', 'copy'))
+    SECTION_CACHE_TYPE_SCHEMA = supported_cache_type
     SECTION_CACHE_LOCAL = 'local'
     SECTION_CACHE_S3 = 's3'
     SECTION_CACHE_GS = 'gs'
diff --git a/dvc/remote/local.py b/dvc/remote/local.py
--- a/dvc/remote/local.py
+++ b/dvc/remote/local.py
@@ -10,6 +10,7 @@
 from dvc.logger import Logger
 from dvc.utils import remove, move, copyfile, file_md5
 from dvc.config import Config
+from dvc.exceptions import DvcException
 
 
 class RemoteLOCAL(RemoteBase):
@@ -30,7 +31,14 @@ def __init__(self, project, config):
         self.link_state = project.link_state
         storagepath = config.get(Config.SECTION_AWS_STORAGEPATH, None)
         self.cache_dir = config.get(Config.SECTION_REMOTE_URL, storagepath)
-        self.cache_type = config.get(Config.SECTION_CACHE_TYPE, None)
+
+        types = config.get(Config.SECTION_CACHE_TYPE, None)
+        if types:
+            if isinstance(types, str):
+                types = [t.strip() for t in types.split(',')]
+            self.cache_types = types
+        else:
+            self.cache_types = self.CACHE_TYPES
 
         if self.cache_dir != None and not os.path.exists(self.cache_dir):
             os.mkdir(self.cache_dir)
@@ -79,8 +87,8 @@ def link(self, src, link, dump=True):
         if not os.path.exists(dname):
             os.makedirs(dname)
 
-        if self.cache_type != None:
-            types = [self.cache_type]
+        if self.cache_types != None:
+            types = self.cache_types
         else:
             types = self.CACHE_TYPES
 
@@ -95,7 +103,6 @@ def link(self, src, link, dump=True):
                 msg = 'Cache type \'{}\' is not supported'.format(typ)
                 Logger.debug(msg)
                 if typ == types[-1]:
-                    raise
                     raise DvcException(msg, cause=exc)
 
     @staticmethod


Test patch:
Repository: iterative/dvc

Issue:
dvc checkout: create empty files if no cache
This is an open question.


Discussion:
I don't think we should do that, sounds very counter intuitive to be honest. Unless I'm missing something. 
I agree. I just like to raise the question - we should be clear about this behavior.

`dvc checkout` outputs this correctly. But it looks like we need a better error message. And again, we probably need a special message for data sources.

```
$ dvc checkout
'data/matrix-train.p': cache file not found
'data/matrix-test.p': cache file not found
...
'data/Posts.xml.tgz': cache file not found
```

How about:
```
$ dvc checkout
'data/matrix-train.p': cache file not found. The file won't be created.
'data/matrix-test.p': cache file not found. The file won't be created.
...
'data/Posts.xml.tgz': data source is missing. Data source update is needed.
```


Resolving patch:
diff --git a/dvc/remote/local.py b/dvc/remote/local.py
--- a/dvc/remote/local.py
+++ b/dvc/remote/local.py
@@ -158,8 +158,8 @@ def checkout(self, path_info, checksum_info):
 
         if not cache or not os.path.exists(cache) or self.changed(md5):
             if cache:
-                Logger.warn(u'\'{}({})\': cache file not found'.format(os.path.relpath(cache),
-                                                                       os.path.relpath(path)))
+                msg = u'Cache \'{}\' not found. File \'{}\' won\'t be created.'
+                Logger.warn(msg.format(md5, os.path.relpath(path)))
             remove(path)
             return
 


Test patch:
Repository: iterative/dvc

Issue:
implement 'dvc destroy'
Can you please share an example of DVC remove. I could be missing something in its command.


Discussion:
`dvc remove` removes outputs of the specified dvc file. Example:
```
echo foo > file
dvc add file
dvc remove file.dvc # Equivalent to `rm file` in this particular case
```
The usefulness in more obvious when your stage has multiple outputs.
Thanks I was hoping that if we could have something like dvc destroy which would be opposite to dvc init. Currently if i were to clean up my project of dvc, I have to remove all files manually.
Ah, I see. Not going to lie, didn't even think of such feature until now :smiley: Turning this issue into feature request.

Resolving patch:
diff --git a/dvc/cli.py b/dvc/cli.py
--- a/dvc/cli.py
+++ b/dvc/cli.py
@@ -7,6 +7,7 @@
 
 from dvc.logger import Logger
 from dvc.command.init import CmdInit
+from dvc.command.destroy import CmdDestroy
 from dvc.command.remove import CmdRemove
 from dvc.command.move import CmdMove
 from dvc.command.run import CmdRun
@@ -87,6 +88,15 @@ def parse_args(argv=None):
                         help="Initiate dvc in directory that is not tracked by any scm tool(e.g. git)")
     init_parser.set_defaults(func=CmdInit)
 
+
+    # Destroy
+    destroy_parser = subparsers.add_parser(
+                        'destroy',
+                        parents=[parent_parser],
+                        help='Destroy dvc')
+    destroy_parser.set_defaults(func=CmdDestroy)
+
+
     # Add
     add_parser = subparsers.add_parser(
                         'add',
diff --git a/dvc/command/destroy.py b/dvc/command/destroy.py
new file mode 100644
--- /dev/null
+++ b/dvc/command/destroy.py
@@ -0,0 +1,11 @@
+from dvc.command.common.base import CmdBase
+
+
+class CmdDestroy(CmdBase):
+    def run(self):
+        try:
+            self.project.destroy()
+        except Exception as exc:
+            self.project.logger.error('Failed to destroy DVC', exc)
+            return 1
+        return 0
diff --git a/dvc/project.py b/dvc/project.py
--- a/dvc/project.py
+++ b/dvc/project.py
@@ -2,6 +2,7 @@
 import csv
 import stat
 import json
+import shutil
 import networkx as nx
 from jsonpath_rw import parse
 
@@ -96,6 +97,12 @@ def init(root_dir=os.curdir, no_scm=False):
 
         return proj
 
+    def destroy(self):
+        for stage in self.stages():
+            stage.remove()
+
+        shutil.rmtree(self.dvc_dir)
+
     def _ignore(self):
         l = [self.state.state_file,
              self.state._lock_file.lock_file,


Test patch:
Repository: iterative/dvc

Issue:
add '-a' alias for '--all-branches'



Discussion:

Resolving patch:
diff --git a/dvc/cli.py b/dvc/cli.py
--- a/dvc/cli.py
+++ b/dvc/cli.py
@@ -207,7 +207,7 @@ def parse_args(argv=None):
     pull_parser.add_argument('-r',
                         '--remote',
                         help='Remote repository to pull from')
-    pull_parser.add_argument(
+    pull_parser.add_argument('-a',
                         '--all-branches',
                         action='store_true',
                         default=False,
@@ -222,7 +222,7 @@ def parse_args(argv=None):
     push_parser.add_argument('-r',
                         '--remote',
                         help='Remote repository to push to')
-    push_parser.add_argument(
+    push_parser.add_argument('-a',
                         '--all-branches',
                         action='store_true',
                         default=False,
@@ -237,7 +237,7 @@ def parse_args(argv=None):
     fetch_parser.add_argument('-r',
                         '--remote',
                         help='Remote repository to fetch from')
-    fetch_parser.add_argument(
+    fetch_parser.add_argument('-a',
                         '--all-branches',
                         action='store_true',
                         default=False,
@@ -329,7 +329,7 @@ def parse_args(argv=None):
                         'gc',
                         parents=[parent_parser],
                         help='Collect garbage')
-    gc_parser.add_argument(
+    gc_parser.add_argument('-a',
                         '--all-branches',
                         action='store_true',
                         default=False,
@@ -493,7 +493,7 @@ def parse_args(argv=None):
     metrics_show_parser_group.add_argument(
                         '--hcsv-path',
                         help='Headed CSV path \'row,column\'(e.g. \'Name,3\'')
-    metrics_show_parser.add_argument(
+    metrics_show_parser.add_argument('-a',
                         '--all-branches',
                         action='store_true',
                         default=False,


Test patch:
Repository: iterative/dvc

Issue:
Locked warning is shown for all files in dvc status
Hi there, the *locked file* warning seems to be shown for all files in dvc status:

```
git init
dvc init
git add .dvc 

# Create example folder
mkdir example
cd example

# Create hello.txt and world.txt
dvc run -o hello.txt 'yes "Hello" | head -10 > hello.txt'
dvc run -o world.txt 'yes "World" | head -10 > world.txt'

# Merge into hello_world.txt
dvc run -d hello.txt -d world.txt -o hello_world.txt 'paste hello.txt world.txt > hello_world.txt'

## Running command:
## 	yes "Hello" | head -10 > hello.txt
## Running command:
## 	yes "World" | head -10 > world.txt
## Running command:
## 	paste hello.txt world.txt > hello_world.txt

dvc status
## DVC file 'hello.txt.dvc' is locked. Its dependecies are not going to be shown in status output.
## DVC file 'world.txt.dvc' is locked. Its dependecies are not going to be shown in status output.
## DVC file 'hello_world.txt.dvc' is locked. Its dependecies are not going to be shown in status output.
## hello.txt.dvc
## world.txt.dvc

cat hello_world.txt.dvc 
## cmd: paste hello.txt world.txt > hello_world.txt
## deps:
## - md5: 2b360db687b5870bd328adbb56d3bf8f
##   path: hello.txt
## - md5: 53d23c8121564867f8ee222841a83811
##   path: world.txt
## md5: e1e60249c4f84c11cda133706d649e31
## outs:
## - cache: true
##   md5: 1abddce536be836e612dedc0a272b86f
##   metric: false
##   path: hello_world.txt
```


Discussion:
Actually, it's shown for the non-locked files 😄 

The bug is in project.status:
```
        for stage in stages:
            if not stage.locked:
```
Oops, sorry :slightly_smiling_face: I'll fix it ASAP, will be included into 0.14.1 release today.

Thank you!

Resolving patch:
diff --git a/dvc/project.py b/dvc/project.py
--- a/dvc/project.py
+++ b/dvc/project.py
@@ -362,7 +362,7 @@ def _local_status(self, target=None):
             stages = self.active_stages()
 
         for stage in stages:
-            if not stage.locked:
+            if stage.locked:
                 msg = 'DVC file \'{}\' is locked. Its dependecies are not ' \
                       'going to be shown in status output.'
                 self.logger.warn(msg.format(stage.relpath))


Test patch:
Repository: iterative/dvc

Issue:
updater: print "Checking for updates"
As @shcheklein noticed, updater takes quite a noticeable time to check for updates and it would be great if we could print some msg to let people know what dvc is doing.


Discussion:

Resolving patch:
diff --git a/dvc/updater.py b/dvc/updater.py
--- a/dvc/updater.py
+++ b/dvc/updater.py
@@ -9,6 +9,7 @@ class Updater(object):
     URL = 'https://4ki8820rsf.execute-api.us-east-2.amazonaws.com/prod/latest-version'
     UPDATER_FILE = 'updater'
     TIMEOUT = 7 * 24 * 60 * 60 #every week
+    TIMEOUT_GET = 10
 
     def __init__(self, dvc_dir):
         self.dvc_dir = dvc_dir
@@ -33,8 +34,10 @@ def check(self):
 
             os.unlink(self.updater_file)
 
+        Logger.info('Checking for updates...')
+
         try:
-            r = requests.get(self.URL)
+            r = requests.get(self.URL, timeout=self.TIMEOUT_GET)
             j = r.json()
             latest = j['version']
             open(self.updater_file, 'w+').close()


Test patch:
Repository: iterative/dvc

Issue:
Travis CI: Flake8 spots some missing imports
flake8 testing of https://github.com/iterative/dvc on Python 3

$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
```
./dvc/scm.py:86:16: F821 undefined name 'InvalidGitRepositoryError'
        except InvalidGitRepositoryError:
               ^
./dvc/command/destroy.py:16:36: F821 undefined name 'err'
                raise DvcException(err)
                                   ^
./dvc/remote/gs.py:63:19: F821 undefined name 'DvcException'
            raise DvcException('{} doesn\'t exist in the cloud'.format(from_info['key']))
                  ^
3     F821 undefined name 'err'
3
```


Discussion:

Resolving patch:
diff --git a/dvc/command/destroy.py b/dvc/command/destroy.py
--- a/dvc/command/destroy.py
+++ b/dvc/command/destroy.py
@@ -13,7 +13,7 @@ def run_cmd(self):
             if not self.args.force and not prompt(msg, False):
                 msg = u'Cannot destroy without a confirmation from the user. ' \
                       u'Use \'-f\' to force.'
-                raise DvcException(err)
+                raise DvcException(msg)
 
             self.project.destroy()
         except Exception as exc:
diff --git a/dvc/remote/gs.py b/dvc/remote/gs.py
--- a/dvc/remote/gs.py
+++ b/dvc/remote/gs.py
@@ -16,6 +16,7 @@
 from dvc.remote.local import RemoteLOCAL
 from dvc.config import Config
 from dvc.progress import progress
+from dvc.exceptions import DvcException
 
 
 class RemoteGS(RemoteBase):
diff --git a/dvc/scm.py b/dvc/scm.py
--- a/dvc/scm.py
+++ b/dvc/scm.py
@@ -53,6 +53,7 @@ def brancher(self, branches=None, all_branches=False):
         if not branches:
             branches = self.list_branches() if all_branches else [saved]
         for branch in branches:
+            print("OLOLO " + branch)
             self.checkout(branch)
             yield branch
         self.checkout(saved)
@@ -81,6 +82,7 @@ def __init__(self, root_dir=os.curdir):
         super(Git, self).__init__(root_dir)
 
         import git
+        from git.exc import InvalidGitRepositoryError
         try:
             self.repo = git.Repo(root_dir)
         except InvalidGitRepositoryError:


Test patch:
Repository: iterative/dvc

Issue:
Testing against Python 3.6 and 3.7
https://github.com/pyinstaller/pyinstaller#main-advantages says that pyinstaller supports these versions of Python but https://github.com/iterative/dvc/blob/master/.travis.yml#L9 says the opposite.


Discussion:
Hi @cclauss !

Looks like Pyinstaller now supports 3.6 and 3.7. It wasn't the case when I wrote those tests. I'll enable tests for those versions shortly. Thank you!

--Ruslan
https://github.com/travis-ci/travis-ci/issues/9069 is important with Python 3.7...

TL;DR is that the Travis CI Python 3.7 job needs __dist: xenial__ and __sudo: required__.

Resolving patch:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -1,5 +1,6 @@
 import os
 import re
+import errno
 
 from dvc.config import Config
 from dvc.logger import Logger
@@ -84,10 +85,11 @@ def move(self, path_info):
 
     def _makedirs(self, fname):
         dname = os.path.dirname(fname)
+
         try:
             os.makedirs(dname)
         except OSError as e:
-            if e.errno != os.errno.EEXIST:
+            if e.errno != errno.EEXIST:
                 raise
 
     def md5s_to_path_infos(self, md5s):


Test patch: