internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: iterative/dvc

Issue:
dynamic version bug
Even though package versions are correct, if you call 'dvc --version' it will add sha from the current repository that you are in, which is wrong.


Discussion:

Resolving patch:
diff --git a/dvc/__init__.py b/dvc/__init__.py
--- a/dvc/__init__.py
+++ b/dvc/__init__.py
@@ -5,18 +5,43 @@
 """
 import os
 
-VERSION = '0.9.5'
+__version__ = '0.9.5'
 
-if os.getenv('APPVEYOR_REPO_TAG', '').lower() != 'true' and os.getenv('TRAVIS_TAG', '') == '':
-    # Dynamically update version
+PACKAGEPATH = os.path.abspath(os.path.dirname(__file__))
+HOMEPATH = os.path.dirname(PACKAGEPATH)
+VERSIONPATH = os.path.join(PACKAGEPATH, 'version.py')
+
+if os.path.exists(os.path.join(HOMEPATH, 'setup.py')):
+    # dvc is run directly from source without installation or
+    # __version__ is called from setup.py
+    if os.getenv('APPVEYOR_REPO_TAG', '').lower() != 'true' and os.getenv('TRAVIS_TAG', '') == '':
+        # Dynamically update version
+        try:
+            import git
+            repo = git.Repo(HOMEPATH)
+            sha = repo.head.object.hexsha
+            short_sha = repo.git.rev_parse(sha, short=6)
+            dirty = '.mod' if repo.is_dirty() else ''
+            __version__ = '{}+{}{}'.format(__version__, short_sha, dirty)
+
+            # Write a helper file, that will be installed with the package
+            # and will provide a true version of the installed dvc
+            with open(VERSIONPATH, 'w+') as fd:
+                fd.write('# AUTOGENERATED by dvc/__init__.py')
+                fd.write('version = "{}"'.format(__version__))
+        except:
+            pass
+    else:
+        # Remove version.py so that it doesn't get into the release
+        if os.path.exists(VERSIONPATH):
+            os.unlink(VERSIONPATH)
+else:
+    # dvc was installed with pip or something. Hopefully we have our
+    # auto-generated version.py to help us provide a true version
     try:
-        import git
-        repo = git.Repo(os.curdir, search_parent_directories=True)
-        sha = repo.head.object.hexsha
-        short_sha = repo.git.rev_parse(sha, short=6)
-        dirty = '.mod' if repo.is_dirty() else ''
-        VERSION = '{}+{}{}'.format(VERSION, short_sha, dirty)
+        from dvc.version import version
+        __version__ = version
     except:
         pass
 
-__version__ = VERSION
+VERSION = __version__


Test patch:
Repository: iterative/dvc

Issue:
Not to write empty and null values in DVC files
```
$ cat images.zip.dvc
cmd: null
deps: []
outs:
- cache: true
  md5: fd47d523fa82bb9c282b897b55be0442
  path: images.zip
```

`cmd` and `deps` should not be in the file. It is needed for tutorials mostly and it looks better in general.


Discussion:

Resolving patch:
diff --git a/dvc/stage.py b/dvc/stage.py
--- a/dvc/stage.py
+++ b/dvc/stage.py
@@ -161,11 +161,17 @@ def dumpd(self):
         deps = [x.dumpd(self.cwd) for x in self.deps]
         outs = [x.dumpd(self.cwd) for x in self.outs]
 
-        return {
-            Stage.PARAM_CMD: self.cmd,
-            Stage.PARAM_DEPS: deps,
-            Stage.PARAM_OUTS: outs,
-        }
+        ret = {}
+        if self.cmd:
+            ret[Stage.PARAM_CMD] = self.cmd
+
+        if len(deps):
+            ret[Stage.PARAM_DEPS] = deps
+
+        if len(outs):
+            ret[Stage.PARAM_OUTS] = outs
+
+        return ret
 
     def dump(self, fname=None):
         if not fname:


Test patch:
Repository: iterative/dvc

Issue:
Output format: do not print empty lines
Some of the commands print empty lines.

```
$ dvc pull # Nothing to pull
# This is an empty line
```

```
$ dvc status # Nothing to show
# This is an empty line
```

Probably we have more commands with the same issue.


Discussion:
```
$ dvc repro
Reproducing 'model.p.dvc':
[SKIP]
Reproducing 'Dvcfile':
# <-- This is an empty line
```
Status with changes:
```
$ dvc status
# <-- EMPTY LINE
	new file:	/Users/dmitry/src/myrepo_1/.dvc/cache/62f8c2ba93cfe5a6501136078f0336f9
```

Resolving patch:
diff --git a/dvc/progress.py b/dvc/progress.py
--- a/dvc/progress.py
+++ b/dvc/progress.py
@@ -47,10 +47,6 @@ def finish_target(self, name):
 
             self._n_finished += 1
 
-    def finish(self):
-        if sys.stdout.isatty():
-            print()
-
     def _bar(self, target_name, current, total):
         """
         Make a progress bar out of info, which looks like:
diff --git a/dvc/utils.py b/dvc/utils.py
--- a/dvc/utils.py
+++ b/dvc/utils.py
@@ -129,7 +129,5 @@ def map_progress(func, targets, n_threads):
         ret = pool.map(wrapper, targets)
     except Exception as exc:
         raise
-    finally:
-        progress.finish()
 
     return list(zip(targets, ret))


Test patch:
Repository: iterative/dvc

Issue:
Display more user-friendly error on dvc repro file.txt
A more user-friendly error could be shown when running `dvc repro file.txt` instead of `dvc repro file.txt.dvc`.

Example:
```
echo "Hello" > file.txt
dvc add file.txt
dvc repro file.txt
# Failed to reproduce 'file.txt' - unexpected error: Stage file file.txt does not exist
```




Discussion:
Hi @prihoda !

Thank you for the feedback! That is a great point! Something like `'file.txt' is not a dvc file. Maybe you meant 'file.txt.dvc'?` or something like that should be more user-friendly. I'll take a look into it soon. 


Resolving patch:
diff --git a/dvc/command/repro.py b/dvc/command/repro.py
--- a/dvc/command/repro.py
+++ b/dvc/command/repro.py
@@ -1,5 +1,8 @@
+import os
+
 from dvc.command.common.base import CmdBase
-from dvc.project import ReproductionError
+from dvc.project import ReproductionError, StageNotFoundError
+from dvc.stage import Stage
 
 
 class CmdRepro(CmdBase):
@@ -14,6 +17,16 @@ def run(self):
                 msg = 'Failed to reproduce \'{}\''.format(target)
                 self.project.logger.error(msg, ex)
                 return 1
+            except StageNotFoundError as ex:
+                if os.path.exists(target):
+                    msg = '\'{}\' is not a dvc file.'.format(target)
+                    dvcfile = target + Stage.STAGE_FILE_SUFFIX
+                    if os.path.exists(target) and os.path.isfile(dvcfile):
+                        msg += ' Maybe you meant \'{}\'?'.format(dvcfile)
+                else:
+                    msg = '\'{}\' does not exist.'.format(target)
+                self.project.logger.error(msg)
+                return 1
             except Exception as ex:
                 msg = 'Failed to reproduce \'{}\' - unexpected error'.format(target)
                 self.project.logger.error(msg, ex)


Test patch:
Repository: iterative/dvc

Issue:
Checkout single file
It would be great if the `dvc checkout` command could support checking out only a single file. This is useful when a file is accidentally changed or removed.


Discussion:
Hi @prihoda !

That is a great idea! I'll look into implementing it for the next version.

Resolving patch:
diff --git a/dvc/cli.py b/dvc/cli.py
--- a/dvc/cli.py
+++ b/dvc/cli.py
@@ -91,6 +91,10 @@ def parse_args(argv=None):
                         'checkout',
                         parents=[parent_parser],
                         help='Checkout data files from cache')
+    checkout_parser.add_argument(
+                        'targets',
+                        nargs='*',
+                        help='DVC files')
     checkout_parser.set_defaults(func=CmdCheckout)
 
     # Run
diff --git a/dvc/command/checkout.py b/dvc/command/checkout.py
--- a/dvc/command/checkout.py
+++ b/dvc/command/checkout.py
@@ -1,12 +1,12 @@
 import os
-import stat
-
-from dvc.system import System
 from dvc.command.common.base import CmdBase
-from dvc.logger import Logger
 
 
 class CmdCheckout(CmdBase):
     def run(self):
-        self.project.checkout()
+        if not self.args.targets:
+            self.project.checkout()
+        else:
+            for target in self.args.targets:
+                self.project.checkout(target=target)
         return 0
diff --git a/dvc/project.py b/dvc/project.py
--- a/dvc/project.py
+++ b/dvc/project.py
@@ -185,9 +185,17 @@ def _remove_untracked_hardlinks(self):
                 Logger.info(u'Remove empty directory \'{}\''.format(dir))
                 os.removedirs(dir)
 
-    def checkout(self):
+    def checkout(self, target=None):
         self._remove_untracked_hardlinks()
-        for stage in self.stages():
+
+        if target:
+            if not Stage.is_stage_file(target):
+                raise StageNotFoundError(target)
+            stages = [Stage.load(self, target)]
+        else:
+            stages = self.stages()
+
+        for stage in stages:
             stage.checkout()
 
     def _used_cache(self, target=None):


Test patch:
diff --git a/tests/test_checkout.py b/tests/test_checkout.py
--- a/tests/test_checkout.py
+++ b/tests/test_checkout.py
@@ -16,7 +16,7 @@ class TestCheckout(TestRepro):
     def setUp(self):
         super(TestCheckout, self).setUp()
 
-        self.dvc.add(self.DATA_DIR)
+        self.data_dir_stage = self.dvc.add(self.DATA_DIR)
 
         self.orig = 'orig'
         shutil.copy(self.FOO, self.orig)
@@ -35,6 +35,13 @@ def _test_checkout(self):
         self.assertTrue(filecmp.cmp(self.FOO, self.orig))
 
 
+class TestCheckoutSingleStage(TestCheckout):
+    def test(self):
+        self.dvc.checkout(self.foo_stage.path)
+        self.dvc.checkout(self.data_dir_stage.path)
+        self._test_checkout()
+
+
 class TestCheckoutCorruptedCacheFile(TestRepro):
     def test(self):
         cache = self.foo_stage.outs[0].cache
Repository: iterative/dvc

Issue:
command as a dependency
`dvc run --cmd-dep condition.py -d input -o output cmd input output`


Discussion:

Resolving patch:
diff --git a/dvc/stage.py b/dvc/stage.py
--- a/dvc/stage.py
+++ b/dvc/stage.py
@@ -92,6 +92,10 @@ def changed_md5(self):
 
     def changed(self):
         ret = False
+
+        if not self.is_data_source and len(self.deps) == 0:
+            ret = True
+
         for entry in itertools.chain(self.outs, self.deps):
             if entry.changed():
                 ret = True


Test patch:
diff --git a/tests/test_repro.py b/tests/test_repro.py
--- a/tests/test_repro.py
+++ b/tests/test_repro.py
@@ -28,6 +28,22 @@ def setUp(self):
                      cmd='python {} {} {}'.format(self.CODE, self.FOO, self.file1))
 
 
+class TestReproNoDeps(TestRepro):
+    def test(self):
+        out = 'out'
+        code_file = 'out.py'
+        stage_file = 'out.dvc'
+        code = 'import uuid\nwith open("{}", "w+") as fd:\n\tfd.write(str(uuid.uuid4()))\n'.format(out)
+        with open(code_file, 'w+') as fd:
+            fd.write(code)
+        self.dvc.run(fname=stage_file,
+                     outs=[out],
+                     cmd='python {}'.format(code_file))
+
+        stages = self.dvc.reproduce(stage_file)
+        self.assertEqual(len(stages), 1)
+
+
 class TestReproForce(TestRepro):
     def test(self):
         stages = self.dvc.reproduce(self.file1_stage, force=True)
Repository: iterative/dvc

Issue:
DVC pull fails on unstaged files when running in subdirectory
Hi, I noticed `dvc pull` fails when I run it inside a subdirectory and some unstaged files are present.

How to reproduce:
```
git init
dvc init
dvc config Core.Cloud AWS
dvc config AWS.StoragePath dvc-example-bucket/cache
git add .dvc

mkdir example
cd example

echo "Hello World" > hello.txt
dvc add hello.txt

git add .

touch other.txt

dvc push
# (1/1): [##############################] 100% e5/9ff97941044f85df5297e1c302d260

# Fails:
dvc pull hello.txt.dvc
# Failed to pull data from the cloud: [Errno 2] No such file or directory: 'example/other.txt'

# Running from parent directory works.
cd ..
dvc pull example/hello.txt.dvc
```


Discussion:

Resolving patch:
diff --git a/dvc/scm.py b/dvc/scm.py
--- a/dvc/scm.py
+++ b/dvc/scm.py
@@ -137,7 +137,8 @@ def branch(self, branch):
         self.repo.git.branch(branch)
 
     def untracked_files(self):
-        return self.repo.untracked_files
+        files = self.repo.untracked_files
+        return [os.path.join(self.repo.working_dir, fname) for fname in files]
 
     def is_tracked(self, path):
         return len(self.repo.git.ls_files(path)) != 0


Test patch:
Repository: iterative/dvc

Issue:
Save data file size
#669 

From @villasv:

> File sizes would be helpful. Once I had an empty output file and had to trace back the step in which it became empty through the pipeline and the commit that introduced the bug. Would have been pretty easy if I had that info was also tracked.


Discussion:
Btw, we should consider printing a warning when output is empty.

Resolving patch:
diff --git a/dvc/output.py b/dvc/output.py
--- a/dvc/output.py
+++ b/dvc/output.py
@@ -98,6 +98,10 @@ def save(self):
         if not os.path.isfile(self.path) and not os.path.isdir(self.path):
             raise CmdOutputIsNotFileOrDirError(self.rel_path)
 
+        if (os.path.isfile(self.path) and os.path.getsize(self.path) == 0) or \
+           (os.path.isdir(self.path) and len(os.listdir(self.path)) == 0):
+            self.project.logger.warn("File/directory '{}' is empty.".format(self.rel_path))
+
         self.md5 = self.project.state.update(self.path)
 
     @staticmethod
diff --git a/dvc/scm.py b/dvc/scm.py
--- a/dvc/scm.py
+++ b/dvc/scm.py
@@ -87,7 +87,7 @@ def _get_gitignore(self, path):
         gitignore = os.path.abspath(os.path.realpath(gitignore))
 
         if not gitignore.startswith(self.root_dir):
-            raise FileNotInRepoError()
+            raise FileNotInRepoError(path)
 
         return entry, gitignore
 


Test patch:
Repository: iterative/dvc

Issue:
Terminal encoding and `dvc run`
@dmpetrov discovered an issue with `dvc run` when it throws an error(same w/o -v):
```
Unexpected error: 'ascii' codec can't decode byte 0xef in position 137272: ordinal not in range(128)
```

Adding 
```
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
```
helps.

Probably just python2 problem, as python3 uses utf8 by default instead of ascii.


Discussion:
+1

I just encountered this doing the tutorial at the first `dvc run` step:
```
$ dvc run -d data/Posts.xml.tgz -o data/Posts.xml tar -zxf data/Posts.xml.tgz -C data/
Using 'Posts.xml.dvc' as a stage file
Reproducing 'Posts.xml.dvc':
	tar -zxf data/Posts.xml.tgz -C data/
Unexpected error: 'ascii' codec can't decode byte 0xef in position 137272: ordinal not in range(128)
```

Is there a hot fix? Like incorporating that setdefaultencoding('utf-8') somewhere?

My installation:
```
$ brew info dvc
dataversioncontrol/dvc/dvc: stable 0.9.6
Git for data science projects
https://dataversioncontrol.com/
/usr/local/Cellar/dvc/0.9.6 (9,515 files, 118.9MB) *
  Built from source on 2018-05-15 at 16:56:22
From: https://github.com/dataversioncontrol/homebrew-dvc/blob/master/Formula/dvc.rb
==> Dependencies
Build: pkg-config ✔
Required: python ✔
```
Hi @itcarroll !

Thank you for your feeback! I am working on it right now. Should be ready soon and released in 0.9.7(released until tomorrow).

Resolving patch:
diff --git a/dvc/utils.py b/dvc/utils.py
--- a/dvc/utils.py
+++ b/dvc/utils.py
@@ -3,6 +3,7 @@
 """
 import os
 import re
+import sys
 import json
 import stat
 import shutil
@@ -41,6 +42,8 @@ def file_md5(fname):
                 if binary:
                     chunk = data
                 else:
+                    if sys.version_info[0] == 2:
+                        data = data.decode('utf-8')
                     chunk = dos2unix(data).encode('utf-8')
 
                 hash_md5.update(chunk)


Test patch:
Repository: iterative/dvc

Issue:
Project graph construction takes a long time
Hi guys, creating the project graph takes minutes on my project with 54 stages. 

Vast majority of the time is spent in the output.py `Dependency.stage` function that is repeatedly called for each dependency:
```
        for stage in self.project.stages():
            for out in stage.outs:
                if self.path == out.path:
                    return stage
```

Any ideas how to improve that? I haven't had the chance to understand all the code yet.


Discussion:
Hi @prihoda ! Great point! Current implementation is not very efficient and we are aware of that and plan to look into performance improvements in the future. In the current issue can be easily solved by not re-collecting stages each time. I'll look into it today and will try to release the fix with 0.9.7. Thank you!

Resolving patch:
diff --git a/dvc/output.py b/dvc/output.py
--- a/dvc/output.py
+++ b/dvc/output.py
@@ -135,13 +135,6 @@ def loads(cls, project, s, cwd=os.curdir):
     def loads_from(cls, project, s_list, cwd=os.curdir):
         return [cls.loads(project, x, cwd=cwd) for x in s_list]
 
-    def stage(self):
-        for stage in self.project.stages():
-            for out in stage.outs:
-                if self.path == out.path:
-                    return stage
-        return None
-
 
 class Output(Dependency):
     PARAM_CACHE = 'cache'
diff --git a/dvc/project.py b/dvc/project.py
--- a/dvc/project.py
+++ b/dvc/project.py
@@ -239,7 +239,7 @@ def _local_status(self, target=None):
         else:
             stages = self.stages()
 
-        for stage in self.stages():
+        for stage in stages:
             status.update(stage.status())
 
         return status
@@ -326,12 +326,18 @@ def metrics(self, path, json_path=None, tsv_path=None, htsv_path=None):
 
     def graph(self):
         G = nx.DiGraph()
+        stages = self.stages()
 
-        for stage in self.stages():
+        outs_map = {}
+        for stage in stages:
+            for o in stage.outs:
+                outs_map[o.path] = stage
+
+        for stage in stages:
             node = os.path.relpath(stage.path, self.root_dir)
             G.add_node(node, stage=stage)
             for dep in stage.deps:
-                dep_stage = dep.stage()
+                dep_stage = outs_map.get(dep.path, None)
                 if not dep_stage:
                     continue
                 dep_node = os.path.relpath(dep_stage.path, self.root_dir)


Test patch:
Repository: iterative/dvc

Issue:
run: automatic dvcfile name generation for directories
When -f not specified and -o is a directory and --cwd is specified, `dvc run` generates wrong fname for the resulting dvcfile.

Kudos @Nicolabo for discovering!


Discussion:

Resolving patch:
diff --git a/dvc/command/run.py b/dvc/command/run.py
--- a/dvc/command/run.py
+++ b/dvc/command/run.py
@@ -43,6 +43,6 @@ def stage_file_name(args_file, args_outs, args_outs_no_cache):
     def _stage_file_basename(fname):
         result = os.path.basename(fname)
         if len(result) == 0:
-            result = os.path.dirname(fname)
+            result = os.path.basename(os.path.dirname(fname))
         result += Stage.STAGE_FILE_SUFFIX
         return result


Test patch:
Repository: iterative/dvc

Issue:
Dvc checkout for an empty dir should create the dir itself



Discussion:

Resolving patch:
diff --git a/dvc/remote/local.py b/dvc/remote/local.py
--- a/dvc/remote/local.py
+++ b/dvc/remote/local.py
@@ -160,6 +160,11 @@ def checkout(self, path_info, checksum_info):
         msg = u'Checking out directory \'{}\' with cache \'{}\''
         Logger.debug(msg.format(os.path.relpath(path), os.path.relpath(cache)))
 
+        # Create dir separately so that dir is created
+        # even if there are no files in it
+        if not os.path.exists(path):
+            os.makedirs(path)
+
         dir_cache = self.dir_cache(cache)
         for relpath, c in dir_cache.items():
             p = os.path.join(path, relpath)


Test patch:
diff --git a/tests/test_checkout.py b/tests/test_checkout.py
--- a/tests/test_checkout.py
+++ b/tests/test_checkout.py
@@ -170,3 +170,20 @@ def test(self):
             yaml.dump(d, fd)
 
         self.dvc.checkout()
+
+
+class TestCheckoutEmptyDir(TestDvc):
+    def test(self):
+        dname = 'empty_dir'
+        os.mkdir(dname)
+
+        stage = self.dvc.add(dname)
+        self.assertEqual(len(stage.outs), 1)
+
+        stage.outs[0].remove()
+        self.assertFalse(os.path.exists(dname))
+
+        self.dvc.checkout()
+
+        self.assertTrue(os.path.isdir(dname))
+        self.assertEqual(len(os.listdir(dname)), 0)
Repository: iterative/dvc

Issue:
dvc remote without a command throws an error
```
$ dvc remote
Initialization error: 'Namespace' object has no attribute 'func'
```
It should probably print out help instead.


Discussion:
Nice catch! This bug only occurs on python3, but on python2 the help is printed as expected:
```
(dvc) ➜  dvc git:(master) ✗ dvc remote                          
usage: dvc remote [-h] [-q] [-v] {add,remove,modify,list} ...   
dvc remote: error: too few arguments                            
(dvc) ➜  dvc git:(master) ✗ workon python3-dvc                  
(python3-dvc) ➜  dvc git:(master) ✗ dvc remote                  
Initialization error: 'Namespace' object has no attribute 'func'
(python3-dvc) ➜  dvc git:(master) ✗                             
```

Resolving patch:
diff --git a/dvc/cli.py b/dvc/cli.py
--- a/dvc/cli.py
+++ b/dvc/cli.py
@@ -23,6 +23,16 @@
 from dvc import VERSION
 
 
+def _fix_subparsers(subparsers):
+    # NOTE: Workaround for bug in Python 3
+    # More info at:
+    #  https://bugs.python.org/issue16308
+    #  https://github.com/iterative/dvc/issues/769
+    if sys.version_info[0] == 3:
+        subparsers.required = True
+        subparsers.dest = 'cmd'
+
+
 def parse_args(argv=None):
     # Common args
     parent_parser = argparse.ArgumentParser(add_help=False)
@@ -58,10 +68,7 @@ def parse_args(argv=None):
                         dest='cmd',
                         help='Use dvc CMD --help for command-specific help')
 
-    # NOTE: Workaround for bug in Python 3
-    if sys.version_info[0] == 3:
-        subparsers.required = True
-        subparsers.dest = 'cmd'
+    _fix_subparsers(subparsers)
 
     # Init
     init_parser = subparsers.add_parser(
@@ -283,6 +290,8 @@ def parse_args(argv=None):
                         dest='cmd',
                         help='Use dvc remote CMD --help for command-specific help')
 
+    _fix_subparsers(remote_subparsers)
+
     remote_add_parser = remote_subparsers.add_parser(
                         'add',
                         parents=[parent_parser],


Test patch: