Repository: iterative/dvc

Issue:
optimize dvcignore
From https://github.com/iterative/dvc/pull/3867 it is apparent that we need to bench dvcignore and research if anything else there requires optimization. A user (@courentin ) reported that something simple like
```
data/something
bash_projects
mlruns
notebooks
externals
examples
.circleci
.vscode
.pytest_cache
wandb
```
takes a significant time.

- [ ] add benchmark for this to dvc-bench project


Discussion:
Looked at this and I see several issues with code, most of which also affect performance:

1. `DvcIgnore*` classes designed around using them in `walk()`, which leads to excessively complex code handling check for full path.
2. All the complexity above is added to `CleanTree`, which makes it go into `DvcIgnore*` domain. It should really look like:
    ```python
    def isfile(self, path):
        return self.tree.isfile(path) and not self.dvcignore.match_file(path)
    ```
3. `DvcIgnore*` hierarchy forces sequential check against all rules and regexes. This might be optimized by constructing single structure, i.e. a big regex or a prefix tree. This is complicated by the fact that dvcignores may contain negations though.
4. `pathspec` library also checks against all regexes sequentially, which adds to 3.
5. **High level issue**. I suspect that we recheck files found via walk, so we run ignores twice. It needs to be estimated whether this is an issue.
Thank you @Suor 

It seems consistent with what I experienced. The more I add lines in `.dvcignore`, the slowest `dvc status` is.
> Looked at this and I see several issues with code, most of which also affect performance:
> 
> 1. `DvcIgnore*` classes designed around using them in `walk()`, which leads to excessively complex code handling check for full path.
> 2. All the complexity above is added to `CleanTree`, which makes it go into `DvcIgnore*` domain. It should really look like:
>    ```python
>    def isfile(self, path):
>        return self.tree.isfile(path) and not self.dvcignore.match_file(path)
>    ```
> 3. `DvcIgnore*` hierarchy forces sequential check against all rules and regexes. This might be optimized by constructing single structure, i.e. a big regex or a prefix tree. This is complicated by the fact that dvcignores may contain negations though.
> 4. `pathspec` library also checks against all regexes sequentially, which adds to 3.
> 5. **High level issue**. I suspect that we recheck files found via walk, so we run ignores twice. It needs to be estimated whether this is an issue.

@Suor 
I also have an interest in this issue and had looked into the code. And here is my solution to some of the issues above:
1. For point (1), `walk()` should use information from `DvcIgnore`. For example,  exiting the `.git` directory at the beginning of the iteration.
2. For point (3), according to [this article](https://www.freecodecamp.org/news/regex-was-taking-5-days-flashtext-does-it-in-15-minutes-55f04411025f/), tried tree or automaton would only perform better if the number of ignored-expressions was greater than several hundred. 
3. For point (4), In `pathspec`  
```python
	matched = False
	for pattern in patterns:
		if pattern.include is not None:
			if file in pattern.match((file,)):
				matched = pattern.include
	return matched
```
It should stop if any of the patterns matched the file. 
@courentin
And I think this is the main reason that It gets slower as the ignore list grows. 

I'd like to try to solve these points this weekend.
@karajan1001 
1. There is no issue with walk, the ignored dir won't we traversed. The issue is when we need to check whether `some/path/abc/file.txt` is ignored we need to build all of its parents and test them in an unnatural way.

> It should stop if any of the patterns matched the file.

So for very common case that file is not ignored it will match it against all of those.
> 1. There is no issue with walk, the ignored dir won't we traversed. The issue is when we need to check whether `some/path/abc/file.txt` is ignored we need to build all of its parents and test them in an unnatural way.

Thank you
Dos `Dvcignore` support expressions like `../*.csv` which influences files outside the current path?


Haha, underrated the difficulty of it. Only written the benchmark (https://github.com/iterative/dvc-bench/pull/30).



> 5\. **High level issue**. I suspect that we recheck files found via walk, so we run ignores twice. It needs to be estimated whether this is an issue.

@Suor, According to @courentin 's call graph in #3867 it only runs once.
@efiop @pared 
I have a question how could I test unmerged changes using `dvc-bench`.
@karajan1001 
Prepared PR explaining this in README, please take a look and review:
https://github.com/iterative/dvc-bench/pull/41
> Dos Dvcignore support expressions like ../*.csv which influences files outside the current path?

@karajan1001 No, same as gitignore, it cannot look back in the tree.
> README

Thank you
@pared Should we keep this open or are we fully done here?
@efiop sorry, autoclose. Seems to me we should leave it open. The issue potentially is still present in the case of multiple `dvcignore` files. Also, points noted by @Suor (https://github.com/iterative/dvc/issues/3869#issuecomment-635854916) still need to be addressed.

EDIT:
#3967 addresses 2 points out of 5 (addressed points are 3 and 4)
@efiop I think we should keep this issue open. There have been a lot of optimizations done thanks to @karajan1001 hard work. 
Still, looking at @Suor original comment, the first 2 issues are still present and we will need to investigate them.
@pared Oops, closed by accident. 
Benchmark result before and after #4242, it had been closed and not linked to this issue.

This update saves about 5ms.
![Screenshot from 2020-07-23 21-24-41](https://user-images.githubusercontent.com/6745454/88291692-3aae2300-cd2b-11ea-91c7-b82f85588649.png)

Thank you @karajan1001 ! Amazing stuff! :pray: 

Just to summarize: the only major thing that is left here is to tell dvcignore not to walk into output directories searching for `.dvcignore`s. So kinda like dvcignore for dvcingores :slightly_smiling_face: Related to https://github.com/iterative/dvc/issues/3369 .
@efiop, another simpler optimization could be to not reset `dvcignore` at all (except on brancher, but on those `repo._tree` changes which change `dvcignore` anyway).
@skshetry Not sure what resetting has to do with that. Currently, it doesn't know about outputs at all.
Just saying that there's no need to reset dvcignore at all, so we could only collect dvcignores (dynamically once) and use it for the whole DVC session. Right now, we reset it each time we add a new output:

https://github.com/iterative/dvc/blob/d8b03737ee4ddaee315939dd40babe881dd6b259/dvc/repo/add.py#L150

Or, every time we run a stage:
https://github.com/iterative/dvc/blob/d8b03737ee4ddaee315939dd40babe881dd6b259/dvc/stage/decorators.py#L43

But, we don't really need to start clean for dvcignores.
@skshetry That's a bit dangerous, and we do that to ensure that there is no hidden unwanted state when we use API (i mean Repo). We also need to reset it when walking the branches, as dvcignore might be different.
@efiop, when walking the branches, the `tree` will have their own `dvcignore`. Also, I have to point out that since CleanTree, we did not use to clear dvcignore (but of course, this was an error on our part), so should be safe?
@skshetry Ah, totally missed that! Great point! Indeed, seems like we no longer need to reset it and it won't cause us problems.
> Thank you @karajan1001 ! Amazing stuff! 🙏
> 
> Just to summarize: the only major thing that is left here is to tell dvcignore not to walk into output directories searching for `.dvcignore`s. So kinda like dvcignore for dvcingores 🙂 Related to #3369 .
@efiop 
Sorry, I had look through #3369, but I can't get this word.
` tell dvcignore not to walk into output directories searching for ".dvcignore" s`
Can you give me some more details?
@karajan1001, right now, we have to collect stages most of the time. So, even though we have dynamic dvcignore now, we still have to look in every nook and corner inside of the repo to collect those stages. We might even be traversing to the depths of directories that are dvc-tracked and might be searching for ".dvcignore" dynamically.

So, the idea is to either read ".gitignore"s or, @efiop mentioned, as we are already collecting for the stages, we could find out the outputs, and never enter into those directories at all.

Also to extend this idea, we could create a concept of "exhaustive" search inside dvcignore, so that after we are done looking for the stages, we can tell dvcignore to not even try updating dvcignore now (should make later dynamic calls faster), and what we have is what it is.
@skshetry , @efiop 
Seems that after we collecting stages, all the node of the dvcignore trie tree would be filled.
https://github.com/iterative/dvc/blob/172032d3b303b5c4d90110320f609e93f0cf1191/dvc/ignore.py#L251-L254
`DvcIgnoreFilter` would never try to update a node that it had looked in before. Actually it will never try to update a node twice and will update once only when a path calls it. 

`_reset()` is, absolutely a big problem.  I'm going to open a PR for it with a benchmark result.
@karajan1001, yes, I got confused by that too. If there was a complete search, it won't try to look for dvcignore again, but it will still look into every places, even those directories that are dvc-tracked.

But I am not sure if this would make it slower or faster, as there will never be a dvcignore in those directories (so, an `.exists()` call is only wasted), and os.walk seems cheap enough. Maybe, if the dvc-tracked directories is of too-many levels, it might help.

---

To get an idea about the impact of this change, I tried to `walk()` Linux's source tree by converting it to DVC repo (added .dvc folder and renamed all .gitignore to .dvcignore, not quite a dvc repo but good enough for understanding dvcignore's impact).

```
0.94   -> 16m20s
latest -> 2.76s at first, 2.02s later on
latest (without dvcignore) -> 202ms
```

Half of those times are spent on `.match()` for regex patterns. So, that's quite an optimization. Cheers.
Okay, I missed another important point: dvcignore has to still apply to the inside of the dvc-tracked directories, it's just the `dvcignore` that's not needed to be collected. 

Resolving patch:
diff --git a/dvc/repo/__init__.py b/dvc/repo/__init__.py
--- a/dvc/repo/__init__.py
+++ b/dvc/repo/__init__.py
@@ -615,4 +615,3 @@ def _reset(self):
         self.__dict__.pop("graph", None)
         self.__dict__.pop("stages", None)
         self.__dict__.pop("pipelines", None)
-        self.tree.__dict__.pop("dvcignore", None)


Test patch:
diff --git a/tests/func/test_tree.py b/tests/func/test_tree.py
--- a/tests/func/test_tree.py
+++ b/tests/func/test_tree.py
@@ -258,6 +258,7 @@ def test_walk_dont_ignore_subrepos(tmp_dir, scm, dvc):
     scm.commit("Add subrepo")
 
     dvc_tree = dvc.tree
+    dvc_tree.__dict__.pop("dvcignore")
     scm_tree = scm.get_tree("HEAD", use_dvcignore=True)
     path = os.fspath(tmp_dir)
     get_dirs = itemgetter(1)