internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: iterative/dvc

Issue:
stage: don't write default wdir
https://github.com/iterative/dvc/blob/0.57.0/dvc/stage.py#L632

`.` doesn't evaluate to False and so not skipped.


Discussion:
@efiop , related? https://github.com/iterative/dvc/issues/2210
What is the reasoning for not writing `wdir`? We actually have [test](https://github.com/iterative/dvc/blob/41351a2df2f36ed04846d5e3d976d6d99b71a151/tests/func/test_run.py#L563) checking that wdir `.` is written.

Resolving patch:
diff --git a/dvc/stage.py b/dvc/stage.py
--- a/dvc/stage.py
+++ b/dvc/stage.py
@@ -629,12 +629,16 @@ def load(repo, fname):
 
     def dumpd(self):
         rel_wdir = relpath(self.wdir, os.path.dirname(self.path))
+
+        wdir = pathlib.PurePath(rel_wdir).as_posix()
+        wdir = wdir if wdir != "." else None
+
         return {
             key: value
             for key, value in {
                 Stage.PARAM_MD5: self.md5,
                 Stage.PARAM_CMD: self.cmd,
-                Stage.PARAM_WDIR: pathlib.PurePath(rel_wdir).as_posix(),
+                Stage.PARAM_WDIR: wdir,
                 Stage.PARAM_LOCKED: self.locked,
                 Stage.PARAM_DEPS: [d.dumpd() for d in self.deps],
                 Stage.PARAM_OUTS: [o.dumpd() for o in self.outs],


Test patch:
diff --git a/tests/func/test_run.py b/tests/func/test_run.py
--- a/tests/func/test_run.py
+++ b/tests/func/test_run.py
@@ -560,18 +560,18 @@ def test_not_cached(self):
 
 
 class TestCmdRunWorkingDirectory(TestDvc):
-    def test_default_wdir_is_written(self):
+    def test_default_wdir_is_not_written(self):
         stage = self.dvc.run(
             cmd="echo test > {}".format(self.FOO), outs=[self.FOO], wdir="."
         )
         d = load_stage_file(stage.relpath)
-        self.assertEqual(d[Stage.PARAM_WDIR], ".")
+        self.assertNotIn(Stage.PARAM_WDIR, d.keys())
 
         stage = self.dvc.run(
             cmd="echo test > {}".format(self.BAR), outs=[self.BAR]
         )
         d = load_stage_file(stage.relpath)
-        self.assertEqual(d[Stage.PARAM_WDIR], ".")
+        self.assertNotIn(Stage.PARAM_WDIR, d.keys())
 
     def test_fname_changes_path_and_wdir(self):
         dname = "dir"
diff --git a/tests/func/test_stage.py b/tests/func/test_stage.py
--- a/tests/func/test_stage.py
+++ b/tests/func/test_stage.py
@@ -102,16 +102,10 @@ def test_ignored_in_checksum(self):
         )
 
         d = stage.dumpd()
-        self.assertEqual(d[stage.PARAM_WDIR], ".")
+        self.assertNotIn(Stage.PARAM_WDIR, d.keys())
 
         d = load_stage_file(stage.relpath)
-        self.assertEqual(d[stage.PARAM_WDIR], ".")
-
-        del d[stage.PARAM_WDIR]
-        dump_stage_file(stage.relpath, d)
-
-        d = load_stage_file(stage.relpath)
-        self.assertIsNone(d.get(stage.PARAM_WDIR))
+        self.assertNotIn(Stage.PARAM_WDIR, d.keys())
 
         with self.dvc.state:
             stage = Stage.load(self.dvc, stage.relpath)
Repository: iterative/dvc

Issue:
`dvc move` fails to move datafile to a directory, when the tracking file has a different name than data file
I was playing with `dvc move` and I got this problem.

Script to reproduce it:
```bash
#!/bin/bash -x

rm -rf test1
mkdir test1
cd test1/

echo 'test 1' > file1.txt
dvc init --no-scm
dvc add file1.txt
mv file1.txt.dvc file1.dvc
dvc status
tree -a

mkdir dir1
dvc move file1.txt dir1
dvc status
tree -a
# file1.txt has not been moved to dir1/
```

I am not sure if this is a glitch or everything is as expected, since the `.dvc` file has a different name from the datafile. When tracking file and data file have the same name everything works fine.


Discussion:
AFAIK `file1.txt` should be moved. Ill invesitgate.
A workaround for this (in the context of the example above) would be:
```bash
mv file1.txt dir1/
rm file1.dvc
dvc add -f file1.dvc dir1/file1.txt
```
or this:
```bash
mv file1.txt dir1/
# edit file1.dvc and change: 'path: dir1/file1.txt'
dvc commit file1.dvc
```

Resolving patch:
diff --git a/dvc/repo/move.py b/dvc/repo/move.py
--- a/dvc/repo/move.py
+++ b/dvc/repo/move.py
@@ -59,9 +59,9 @@ def move(self, from_path, to_path):
             os.path.join(os.curdir, os.path.dirname(to_path))
         )
 
-    to_out = Output.loads_from(
-        stage, [os.path.basename(to_path)], out.use_cache, out.metric
-    )[0]
+    to_path = os.path.relpath(to_path, stage.wdir)
+
+    to_out = Output.loads_from(stage, [to_path], out.use_cache, out.metric)[0]
 
     with self.state:
         out.move(to_out)


Test patch:
diff --git a/tests/func/test_move.py b/tests/func/test_move.py
--- a/tests/func/test_move.py
+++ b/tests/func/test_move.py
@@ -200,3 +200,13 @@ def test_move_should_save_stage_info(dvc_repo, repo_dir):
     dvc_repo.move(repo_dir.DATA_DIR, "new_name")
 
     assert dvc_repo.status() == {}
+
+
+def test_should_move_to_dir_on_non_default_stage_file(dvc_repo, repo_dir):
+    stage_file_name = "stage.dvc"
+
+    dvc_repo.add(repo_dir.FOO, fname=stage_file_name)
+
+    dvc_repo.move(repo_dir.FOO, repo_dir.DATA_DIR)
+
+    assert os.path.exists(os.path.join(repo_dir.DATA_DIR, repo_dir.FOO))
Repository: iterative/dvc

Issue:
remote: base: load_dir_cache - open dir cache instead of download 
Currently, `load_dir_cache` is trying to [download](https://github.com/iterative/dvc/blob/af8a7aa25d9243e24f1ee0e4f61362d48146588c/dvc/remote/base.py#L244) dir json to temporary file. It seems like this behaviour could be replaced using 
`self.open`.

Related #2464 #2503 


Discussion:

Resolving patch:
diff --git a/dvc/logger.py b/dvc/logger.py
--- a/dvc/logger.py
+++ b/dvc/logger.py
@@ -44,7 +44,7 @@ class ColorFormatter(logging.Formatter):
     }
 
     footer = (
-        "{yellow}Having any troubles?{nc}."
+        "{yellow}Having any troubles?{nc}"
         " Hit us up at {blue}https://dvc.org/support{nc},"
         " we are always happy to help!"
     ).format(
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -1,9 +1,8 @@
 from __future__ import unicode_literals
 
 from dvc.ignore import DvcIgnore
-from dvc.utils.compat import str, basestring, urlparse
+from dvc.utils.compat import str, basestring, urlparse, FileNotFoundError
 
-import os
 import json
 import logging
 import tempfile
@@ -250,23 +249,16 @@ def get_dir_cache(self, checksum):
     def load_dir_cache(self, checksum):
         path_info = self.checksum_to_path_info(checksum)
 
-        fobj = tempfile.NamedTemporaryFile(delete=False)
-        path = fobj.name
-        to_info = PathInfo(path)
-        self.cache.download(path_info, to_info, no_progress_bar=True)
-
         try:
-            with open(path, "r") as fobj:
+            with self.cache.open(path_info, "r") as fobj:
                 d = json.load(fobj)
-        except ValueError:
+        except (ValueError, FileNotFoundError):
             logger.exception("Failed to load dir cache '{}'".format(path_info))
             return []
-        finally:
-            os.unlink(path)
 
         if not isinstance(d, list):
             msg = "dir cache file format error '{}' [skipping the file]"
-            logger.error(msg.format(relpath(path)))
+            logger.error(msg.format(relpath(path_info)))
             return []
 
         for info in d:


Test patch:
diff --git a/tests/func/test_cache.py b/tests/func/test_cache.py
--- a/tests/func/test_cache.py
+++ b/tests/func/test_cache.py
@@ -46,13 +46,15 @@ def _do_test(self, ret):
         self.assertEqual(len(ret), 0)
 
     def test(self):
-        fname = self.dvc.cache.local.get("123.dir")
+        checksum = "123.dir"
+        fname = self.dvc.cache.local.get(checksum)
         self.create(fname, "<clearly>not,json")
-        self._do_test(self.dvc.cache.local.load_dir_cache(fname))
+        self._do_test(self.dvc.cache.local.load_dir_cache(checksum))
 
-        fname = self.dvc.cache.local.get("234.dir")
+        checksum = "234.dir"
+        fname = self.dvc.cache.local.get(checksum)
         self.create(fname, '{"a": "b"}')
-        self._do_test(self.dvc.cache.local.load_dir_cache(fname))
+        self._do_test(self.dvc.cache.local.load_dir_cache(checksum))
 
 
 class TestExternalCacheDir(TestDvc):
Repository: iterative/dvc

Issue:
ui: pbar during md5 calc doesn't need to be multithreaded 
The "Mutithreaded" prefix is confusing, also jumping through screen doesn't work with redirection well.


Discussion:
💯 on this!

Resolving patch:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -21,7 +21,7 @@
     ConfirmRemoveError,
     DvcIgnoreInCollectedDirError,
 )
-from dvc.progress import Tqdm, TqdmThreadPoolExecutor
+from dvc.progress import Tqdm
 from dvc.utils import LARGE_DIR_SIZE, tmp_fname, move, relpath, makedirs
 from dvc.state import StateNoop
 from dvc.path_info import PathInfo, URLInfo
@@ -178,9 +178,7 @@ def get_file_checksum(self, path_info):
 
     def _calculate_checksums(self, file_infos):
         file_infos = list(file_infos)
-        with TqdmThreadPoolExecutor(
-            max_workers=self.checksum_jobs
-        ) as executor:
+        with ThreadPoolExecutor(max_workers=self.checksum_jobs) as executor:
             tasks = executor.map(self.get_file_checksum, file_infos)
 
             if len(file_infos) > LARGE_DIR_SIZE:


Test patch:
Repository: iterative/dvc

Issue:
py=2, py<3.5 compatibility
https://github.com/iterative/dvc/blob/5c76b46b460e00d3bb5c3f2ea87b19d961491f12/dvc/logger.py#L161

is invalid for `python<3.5`.


Discussion:

Resolving patch:
diff --git a/dvc/logger.py b/dvc/logger.py
--- a/dvc/logger.py
+++ b/dvc/logger.py
@@ -2,7 +2,7 @@
 
 from __future__ import unicode_literals
 
-from dvc.utils.compat import str, StringIO
+from dvc.utils.compat import str, StringIO, RecursionError
 from dvc.progress import Tqdm
 
 import logging
diff --git a/dvc/utils/compat.py b/dvc/utils/compat.py
--- a/dvc/utils/compat.py
+++ b/dvc/utils/compat.py
@@ -18,6 +18,11 @@
 # simplified version of ipython_genutils/encoding.py
 DEFAULT_ENCODING = sys.getdefaultencoding()
 
+if _ver[:2] < (3, 5):
+    RecursionError = RuntimeError
+else:
+    RecursionError = RecursionError
+
 
 def no_code(x, encoding=None):
     return x


Test patch:
Repository: iterative/dvc

Issue:
.gitignore not ignoring files with square brackets
Conditions to reproduce: I am trying to `dvc add` a file that contains brackets in the file name, like `[4] hello.png`. The .gitignore file contains the same text as the file name, but it doesn't work because git will interpret the square brackets differently: https://superuser.com/questions/1125249/filenames-having-square-brackets-not-ignored-by-gitignore

A solution is to add a backslash to [ ] , as \[ and \]. I believe the same issue may be happening with '?'

(at least happens on Linux, but I think it extends to other platforms too)


Discussion:
@cbecker good catch! Looks like we need to escape other special symbols as well - https://git-scm.com/docs/gitignore . At least leading !, * . I'm not sure if can overescape them "safely", i.e. - `\*\*\*` should be parsed as `***` w/o any special meaning.

I believe, the code to fix it located at:

`dvc/scm/git/__init__.py:134 - _add_entry_to_gitignore`



Resolving patch:
diff --git a/dvc/scm/git/__init__.py b/dvc/scm/git/__init__.py
--- a/dvc/scm/git/__init__.py
+++ b/dvc/scm/git/__init__.py
@@ -6,6 +6,7 @@
 import logging
 
 from funcy import cached_property
+from pathspec.patterns import GitWildMatchPattern
 
 from dvc.exceptions import GitHookAlreadyExistsError
 from dvc.utils.compat import str, open
@@ -149,8 +150,9 @@ def ignore(self, path):
 
         self.ignored_paths.append(path)
 
-    @staticmethod
-    def _add_entry_to_gitignore(entry, gitignore):
+    def _add_entry_to_gitignore(self, entry, gitignore):
+        entry = GitWildMatchPattern.escape(entry)
+
         with open(gitignore, "a+", encoding="utf-8") as fobj:
             fobj.seek(0, os.SEEK_END)
             if fobj.tell() == 0:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -63,7 +63,7 @@ def run(self):
     "humanize>=0.5.1",
     "ruamel.yaml>=0.16.1",
     "funcy>=1.12",
-    "pathspec>=0.5.9",
+    "pathspec>=0.6.0",
     "shortuuid>=0.5.0",
     "tqdm>=4.35.0",
     "packaging>=19.0",


Test patch:
diff --git a/tests/func/test_add.py b/tests/func/test_add.py
--- a/tests/func/test_add.py
+++ b/tests/func/test_add.py
@@ -634,3 +634,20 @@ def test_should_protect_on_repeated_add(link, dvc_repo, repo_dir):
     dvc_repo.add(repo_dir.FOO)
 
     assert not os.access(repo_dir.FOO, os.W_OK)
+
+
+def test_escape_gitignore_entries(git, dvc_repo, repo_dir):
+    fname = "file!with*weird#naming_[1].t?t"
+    ignored_fname = r"/file\!with\*weird\#naming_\[1\].t\?t"
+
+    if os.name == "nt":
+        # Some characters are not supported by Windows in the filename
+        # https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file
+        fname = "file!with_weird#naming_[1].txt"
+        ignored_fname = r"/file\!with_weird\#naming_\[1\].txt"
+
+    os.rename(repo_dir.FOO, fname)
+
+    dvc_repo.add(fname)
+
+    assert ignored_fname in get_gitignore_content()
Repository: iterative/dvc

Issue:
[HACKTOBERFEST] Import: output is too verbose
**Script**:
```bash
dvc init --no-scm
dvc import https://github.com/iterative/dvc scripts/innosetup/dvc.ico
```
**Current output**:
```
Importing 'scripts/innosetup/dvc.ico (https://github.com/iterative/dvc)' -> 'dvc.ico'
Adding '../../../tmp/tmpxec7c6ycdvc-repo/.dvc/tmp' to '../../../tmp/tmpxec7c6ycdvc-repo/.dvc/.gitignore'.
Output 'dvc.ico' didn't change. Skipping saving.
Saving information to 'dvc.ico.dvc'.
```

**Desired output**:
```
Importing 'scripts/innosetup/dvc.ico (https://github.com/iterative/dvc)' -> 'dvc.ico'
Saving information to 'dvc.ico.dvc'.
```


Discussion:
I guess it is about dvc in general. Those two middle messages could be easilly moved to debug in general.
I like the idea of moving those to debug; can we agree on that one?
@mroutis Makes total sense to me too :slightly_smiling_face: 

Resolving patch:
diff --git a/dvc/scm/git/__init__.py b/dvc/scm/git/__init__.py
--- a/dvc/scm/git/__init__.py
+++ b/dvc/scm/git/__init__.py
@@ -155,7 +155,7 @@ def ignore(self, path):
             return
 
         msg = "Adding '{}' to '{}'.".format(relpath(path), relpath(gitignore))
-        logger.info(msg)
+        logger.debug(msg)
 
         self._add_entry_to_gitignore(entry, gitignore)
 
diff --git a/dvc/stage.py b/dvc/stage.py
--- a/dvc/stage.py
+++ b/dvc/stage.py
@@ -293,7 +293,7 @@ def changed(self):
         if ret:
             logger.warning("Stage '{}' changed.".format(self.relpath))
         else:
-            logger.info("Stage '{}' didn't change.".format(self.relpath))
+            logger.debug("Stage '{}' didn't change.".format(self.relpath))
 
         return ret
 


Test patch:
Repository: iterative/dvc

Issue:
Warning messages in stdout
```
$ dvc -V
0.60.1+39104a
$ dvc pipeline show > OUT
$ cat OUT
WARNING: assuming default target 'Dvcfile'.
data/data.xml.dvc
prepare.dvc
featurize.dvc
train.dvc
evaluate.dvc
Dvcfile
```

The warning message `WARNING: assuming default target 'Dvcfile'.` is supposed to be in stderr, not stdout. This seems to work just fine for error messages, but not for warnings.

Many other commands might have the same warning issue. It would be great if this can be fixed holistically.


Discussion:
Might be related: https://github.com/iterative/dvc/issues/1291
@pared You are right, it is the same thing. Looks like we've lost it during logger redesign.
@efiop , @pared  #1291 is not the same thing, this is about warning messages (the other one was about `ERROR` messages :slightly_smiling_face: )

Resolving patch:
diff --git a/dvc/logger.py b/dvc/logger.py
--- a/dvc/logger.py
+++ b/dvc/logger.py
@@ -19,7 +19,7 @@ def __init__(self, record):
 
 class ExcludeErrorsFilter(logging.Filter):
     def filter(self, record):
-        return record.levelno < logging.ERROR
+        return record.levelno < logging.WARNING
 
 
 class ColorFormatter(logging.Formatter):
@@ -182,7 +182,7 @@ def setup(level=logging.INFO):
                 },
                 "console_errors": {
                     "class": "dvc.logger.LoggerHandler",
-                    "level": "ERROR",
+                    "level": "WARNING",
                     "formatter": "color",
                     "stream": "ext://sys.stderr",
                 },


Test patch:
diff --git a/tests/unit/test_logger.py b/tests/unit/test_logger.py
--- a/tests/unit/test_logger.py
+++ b/tests/unit/test_logger.py
@@ -179,3 +179,10 @@ def test_progress_awareness(self, mocker, capsys, caplog):
                     logger.info("some info")
                     captured = capsys.readouterr()
                     assert captured.out == ""
+
+
+def test_handlers():
+    stdout, stderr = logger.handlers
+
+    assert stdout.level == logging.DEBUG
+    assert stderr.level == logging.WARNING
Repository: iterative/dvc

Issue:
"dvc pull" shows progress bar too early 
**Information about setup**

DVC version: 0.63.4
Method of installation: pip
Platform: Ubuntu Linux 16.04

**Description:**
I access my dvc remote via ssh. Whenever I execute `dvc pull` command dvc asks for passphrase for my RSA key and immediately shows progress bar before I've entered anything from my keyboard:

```
Enter a private key passphrase or a password for host 'ipaddress' port 'port' user 'dvcrepo':
  0%|          |Querying cache in ssh://dvcrepo@ipaddress:port/path/to/dvc                                                                                                                                                  0/1 [00:00<?,     ?file/s]
```

Such format is confusing: first time I've executed `dvc pull` I was wondering for several minutes why cache quering stuck on 0% before I realized that dvc expects me to enter passphrase.

Progress bar should be shown **after** I've entered passphrase. 



Discussion:
@casperdcl could you take a look please?
ah, the bar might actually be showing before the prompt, but it's just drawn below all text (including the prompt).


Resolving patch:
diff --git a/dvc/remote/ssh/__init__.py b/dvc/remote/ssh/__init__.py
--- a/dvc/remote/ssh/__init__.py
+++ b/dvc/remote/ssh/__init__.py
@@ -115,9 +115,10 @@ def _try_get_ssh_config_keyfile(user_ssh_config):
             return identity_file[0]
         return None
 
-    def ssh(self, path_info):
+    def ensure_credentials(self, path_info=None):
+        if path_info is None:
+            path_info = self.path_info
         host, user, port = path_info.host, path_info.user, path_info.port
-
         # NOTE: we use the same password regardless of the server :(
         if self.ask_password and self.password is None:
             with saved_passwords_lock:
@@ -133,13 +134,16 @@ def ssh(self, path_info):
                     )
                 self.password = password
 
+    def ssh(self, path_info):
+        self.ensure_credentials(path_info)
+
         from .connection import SSHConnection
 
         return get_connection(
             SSHConnection,
-            host,
-            username=user,
-            port=port,
+            path_info.host,
+            username=path_info.user,
+            port=path_info.port,
             key_filename=self.keyfile,
             timeout=self.timeout,
             password=self.password,
@@ -306,6 +310,9 @@ def cache_exists(self, checksums, jobs=None, name=None):
         if not self.no_traverse:
             return list(set(checksums) & set(self.all()))
 
+        # possibly prompt for credentials before "Querying" progress output
+        self.ensure_credentials()
+
         with Tqdm(
             desc="Querying "
             + ("cache in " + name if name else "remote cache"),


Test patch:
Repository: iterative/dvc

Issue:
skip dag checks for large repos
"dvc add <file>" can take hours in a repo with 1000s of files, due to the `repo.check_modified_graph(stages)` step. Can we make this configurable so I can avoid checking the graph if i know I'm not introducing cycles? 

Similarly, when we `checkout` a file, the `repo.collect` step will always create a graph `G = graph or self.graph`, even if i explicitly pass in the path for a single file. This graph construction is extremely costly with large repos, and isn't used for checking out individual files.


Discussion:
Hi @danfischetti !

This is a lack of optimization for large DAGs on our side. Ideally, we should investigate it and optimize it, 1000 nodes in a DAG shouldn't take hours, for sure. But if you need quick and effective solution, just monkey patch `check_acyclic` with `noop` and you should be set. So something along the lines of:
```
import dvc.repo.graph as dvc_graph

def noop(G):
    pass

dvc_graph.check_acyclic = noop
```
I suppose you are only using dvc through API, right? If so, then monkey patch is the best quick and dirty solution here. I don't think we should introduce config option for such a hack, but we should fix it properly by optimizing :slightly_smiling_face: 
@danfischetti Btw, could you post cProfile output, please? That way we can have a better overlook at what is slowing you down. You've used cProfile, right? Or something else?
@danfischetti Ok, found the issue. We are using `nx.simple_cycles` instead of simply using `nx.find_cycle` which is a lot faster. Preparing a patch right now.

Resolving patch:
diff --git a/dvc/repo/graph.py b/dvc/repo/graph.py
--- a/dvc/repo/graph.py
+++ b/dvc/repo/graph.py
@@ -5,10 +5,17 @@ def check_acyclic(graph):
     import networkx as nx
     from dvc.exceptions import CyclicGraphError
 
-    cycle = next(nx.simple_cycles(graph), None)
+    try:
+        edges = nx.find_cycle(graph, orientation="original")
+    except nx.NetworkXNoCycle:
+        return
 
-    if cycle:
-        raise CyclicGraphError(cycle)
+    stages = set()
+    for from_node, to_node, _ in edges:
+        stages.add(from_node)
+        stages.add(to_node)
+
+    raise CyclicGraphError(list(stages))
 
 
 def get_pipeline(pipelines, node):


Test patch: