Repository: iterative/dvc

Issue:
optimize dvcignore
From https://github.com/iterative/dvc/pull/3867 it is apparent that we need to bench dvcignore and research if anything else there requires optimization. A user (@courentin ) reported that something simple like
```
data/something
bash_projects
mlruns
notebooks
externals
examples
.circleci
.vscode
.pytest_cache
wandb
```
takes a significant time.

- [ ] add benchmark for this to dvc-bench project
remote modify: add `name` param?
So you can change the name of a remote e.g.

```console
$ dvc remote add myremote some/path
$ dvc remote list
myremote some/path
$ dvc remote modify myremote name supercoolremote
$ dvc remote list
supercoolremote some/path
```

The workaround rn is to just add the remote again and remove the previous one. But if it has some special configuration then it's more difficult. A better workaroud is to edit the .dvc/config file but relatively advanced.

`git remote` lets you change a remote name BTW.


Discussion:

BTW if you actually run my example command to modify the inexistent `name` param, it gets written to .dvc/config, which causes an error output whenever you try to change the config later:

```
ERROR: configuration error - config file error:
extra keys not allowed @ data['remote']['myremote']['name']
```

You have to manually fix the config file to fix this.
> which causes an error output whenever you try to change the config later:

(Yep, I hit the same issue here https://github.com/iterative/dvc/issues/3552 - quite annoying one :) )

> A better workaroud is to edit the .dvc/config file but relatively advanced.

not sure this is very advanced to be honest

Not advanced in terms of skill, but you need to know that remote config is in that file. It won't be obvious knowledge for DVC beginners. Also for gdrive remotes the file in .dvc/tmp/...json would also need to be changed manually, I think (if it already exists).
>  Also for gdrive remotes the file in .dvc/tmp/...json would also need to be changed manually, I think (if it already exists).

it doesn't contain the remote name yet. With GDrive multiple remotes are not supported effectively yet.

>  It won't be obvious knowledge for DVC beginners.

it's a good question. I think for people familiar with command line, git, etc it should be more or less intuitive that there is a config somewhere. 
OK. No strong opinion. I would err on the side of adding this which I'm assuming should be pretty easy, since `git remote` allows you to do it, so you may expect this if you're used to Git. + the other reasons outlines. Feel fee to close this though, up to you! Thanks
I'd like to look into it. @jorgeorpinel How about make it `dvc remote rename <old> <new>` following the similar command in Git `git remote rename <old> <new>`.
Good idea, indeed that's how it works on Git. But since we already have a `modify` subcommand to change all possible remote config I'd prefer to add a `name` config param for that.
It does not hurt to add both? Is there some modify command analog in GIt?
Not really. They have a subcommand for each remote config param you can change: https://git-scm.com/docs/git-remote
`name` is ugly. `dvc remote rename` is much better.
Plus `name` complicates the argument resolution, I really don't like that and don't think it is worth bothering with. Let's just introduce `dvc remote rename` as suggested by @karajan1001 .
I have no strong opinion. Either introduce `name` to https://dvc.org/doc/command-reference/remote/modify#available-parameters-for-all-remotes (implies less maintenance both in core code and docs) or introduce `rename` subcommand.

Resolving patch:
diff --git a/dvc/ignore.py b/dvc/ignore.py
--- a/dvc/ignore.py
+++ b/dvc/ignore.py
@@ -1,12 +1,15 @@
 import logging
 import os
+import re
+from itertools import groupby
 
 from funcy import cached_property
-from pathspec import PathSpec
 from pathspec.patterns import GitWildMatchPattern
+from pathspec.util import normalize_file
 
 from dvc.path_info import PathInfo
 from dvc.scm.tree import BaseTree
+from dvc.system import System
 from dvc.utils import relpath
 
 logger = logging.getLogger(__name__)
@@ -27,7 +30,16 @@ def __init__(self, ignore_file_path, tree):
         self.dirname = os.path.normpath(os.path.dirname(ignore_file_path))
 
         with tree.open(ignore_file_path, encoding="utf-8") as fobj:
-            self.ignore_spec = PathSpec.from_lines(GitWildMatchPattern, fobj)
+            path_spec_lines = fobj.readlines()
+            regex_pattern_list = map(
+                GitWildMatchPattern.pattern_to_regex, path_spec_lines
+            )
+            self.ignore_spec = [
+                (ignore, re.compile("|".join(item[0] for item in group)))
+                for ignore, group in groupby(
+                    regex_pattern_list, lambda x: x[1]
+                )
+            ]
 
     def __call__(self, root, dirs, files):
         files = [f for f in files if not self.matches(root, f)]
@@ -48,7 +60,16 @@ def matches(self, dirname, basename):
         else:
             return False
 
-        return self.ignore_spec.match_file(path)
+        if not System.is_unix():
+            path = normalize_file(path)
+        return self.ignore(path)
+
+    def ignore(self, path):
+        result = False
+        for ignore, pattern in self.ignore_spec:
+            if pattern.match(path):
+                result = ignore
+        return result
 
     def __hash__(self):
         return hash(self.ignore_file_path)


Test patch:
diff --git a/tests/unit/test_ignore.py b/tests/unit/test_ignore.py
--- a/tests/unit/test_ignore.py
+++ b/tests/unit/test_ignore.py
@@ -99,3 +99,18 @@ def test_should_ignore_dir(omit_dir):
     new_dirs, _ = ignore(root, dirs, files)
 
     assert set(new_dirs) == {"dir1", "dir2"}
+
+
+def test_ignore_order():
+    dvcignore_path = os.path.join(os.path.sep, "ignore_order", ".dvcignore")
+
+    patterns = ["!ac*", "a*", "!ab*"]
+
+    root = os.path.dirname(dvcignore_path)
+    dirs = ["ignore_order"]
+    files = ["ac", "ab", "aa"]
+
+    ignore = mock_dvcignore(dvcignore_path, patterns)
+    _, new_files = ignore(root, dirs, files)
+
+    assert {"ab"} == set(new_files)