Repository: iterative/dvc

Issue:
On Windows, dvc repro spends an awful lot of time in rwlock
Windows 7 / 10, dvc 0.93.0+f5d89f (after merging #3645)

Running cProfile on `dvc repro -P` in a repository with around 200 steps all of which are up to date. On my system this takes 65s, of which 42s is spent in `_edit_rwlock`. 

I wonder, is it necessary to lock each individual file here? Couldn't the whole repository be locked instead? 

Maybe a broader question: what are the respective purposes of `.dvc/tmp/rwlock`, `.dvc/lock`and `.dvc/updater.lock`?




Discussion:
> Running cProfile on dvc repro -P in a repository with around 200 steps all of which are up to date. On my system this takes 65s, of which 42s is spent in _edit_rwlock.

Thanks for the investigation! That is indeed pretty strange, we need to take a closer look. Could you show your `dvc version` output btw?

> I wonder, is it necessary to lock each individual file here? Couldn't the whole repository be locked instead?

Actually, that's what we used to do, but rwlock was introduced to support granular locking, so that you could run dvc in parallel.

> Maybe a broader question: what are the respective purposes of .dvc/tmp/rwlock, .dvc/lockand .dvc/updater.lock?

Useful link: https://dvc.org/doc/user-guide/dvc-files-and-directories  :slightly_smiling_face: `rwlock` is for granular locking of specific deps and outs, `lock` is for locking the whole repo (when you need to push/pull/etc or need to create/modify/delete dvc-file). `updater.lock` is for our updater worker, that checks for updates once a week.

@charlesbaynham Btw, have you tried running that on Linux? Wonder if this is windows-specific or if we have troubles everywhere.
Thanks for the run down @efiop, that's helpful. The version that ran the above was `0.93.0+f5d89f`, which is v0.93.0 + the changes in PR #3645. That's necessary because otherwise the effect is swamped by all the time spent reproducing duplicate steps. 

I'll give it a try on linux: annoyingly the repository I tested on has some case sensitivity problems which I've glossed over so far on Windows but which prevent it from working on linux, but I should fix that anyway. 

The cProfile results for my repo on Windows:
[repro_nowork.zip](https://github.com/iterative/dvc/files/4509341/repro_nowork.zip)

Here's the cProfile results for `dvc repro -P` in the same repo, run on Linux and Windows. 

[repro_profile.zip](https://github.com/iterative/dvc/files/4516967/repro_profile.zip)


It looks like the same fraction of time is spend writing to `rwlock` in both (just via `posix.fsync` instead of `nt.fsync`)
Here's a screenshot of the profile for Windows:

![image](https://user-images.githubusercontent.com/4397637/79999655-10429480-84b4-11ea-8f52-f1db911dc918.png)

Zooming in, you can see that the actual calls to `Stage.changed()` don't take much time at all, though that's where I'd have expected most of the load to be. 

![image](https://user-images.githubusercontent.com/4397637/79999776-3c5e1580-84b4-11ea-9b7a-5d8636f60f9d.png)


@charlesbaynham Sorry for the delay. Admitedly didn't have much time to look deeply into it, but could you give my patch a try, please:
```
pip uninstall -y dvc
# make sure there is no old dvc version installed
pip install git+https://github.com/efiop/dvc@3653
```
?
Yes that makes a lot of difference. Here are two profiles in the same repo on Window 7, one at v0.93.0+f5d89f and the other at your patch:

[data_processing.zip](https://github.com/iterative/dvc/files/4552125/data_processing.zip)

Your patch cuts the time spent in `_reproduce_stages` from 62s:
![image](https://user-images.githubusercontent.com/4397637/80605571-cb25e180-8a22-11ea-9d5a-21c01c73337d.png)

to 23s

![image](https://user-images.githubusercontent.com/4397637/80605657-e2fd6580-8a22-11ea-99be-fdf3973007d9.png)


But doesn't that just effectively disable the lock?
I also notice that we're making 1.4 million calls to `normpath` which seems like a lot for a repo with ~200 files in it!
Took a quick look at the profiler results to see if the excessive `normpath` calls were pathinfo related, and they are not, but we do make 220k calls to `relpath` in `dvc.utils` (with string paths)
@charlesbaynham Sorry for the delay. It doesn't really disable it, just doesn't try to be paranoid about sync-ing to disk. We'll need to take a closer look. We don't have the capacity to do it right now, but we'll try to do that in the nearest future.
No need for any apology, the work you guys do is amazing.

Maybe a compromise could be to only flush the lock if dvc determines that actual work needs to be done to reproduce a stage. In that scenario it'd be a small fraction of the time taken. 
> Maybe a compromise could be to only flush the lock if dvc determines that actual work needs to be done to reproduce a stage. In that scenario it'd be a small fraction of the time taken.

The thing is that it modifies rwlock _before_ checking that something has changed, so that no other stage is in the middle of modifying something that we will be checking. :slightly_frowning_face: It might be as simple as my patch that removes paranoid fsync-ing, but need to double check that. If not, I'm sure we'll find another way to optimize this.

You and @pmrowla have also noticed something that we know is problematic: relpath calls. The solution there is not stop trying to `abspath` every path we use and just use whatever paths we get. That would also fix the issues with our abspaths getting longer than PATH_MAX limit. dvc doesn't really chdir at any point in its operation, so that should be very safe and efficient. The reason why we didn't do that from day 0 is that we weren't really sure which direction dvc will go, so just wanted to be safe with using deterministic abspaths, but we can surely do the switch now.

Resolving patch:
diff --git a/dvc/rwlock.py b/dvc/rwlock.py
--- a/dvc/rwlock.py
+++ b/dvc/rwlock.py
@@ -51,9 +51,6 @@ def _edit_rwlock(lock_dir):
     yield lock
     with open(path, "w+") as fobj:
         json.dump(lock, fobj)
-        # NOTE: flush and fsync to ensure that rwlock contents are saved
-        fobj.flush()
-        os.fsync(fobj.fileno())
 
 
 def _infos_to_str(infos):


Test patch:
diff --git a/tests/func/test_repro_multistage.py b/tests/func/test_repro_multistage.py
--- a/tests/func/test_repro_multistage.py
+++ b/tests/func/test_repro_multistage.py
@@ -449,7 +449,7 @@ def test_repro_when_new_outs_added_does_not_exist(tmp_dir, dvc):
         {
             "stages": {
                 "run-copy": {
-                    "cmd": "python copy {} {}".format("foo", "foobar"),
+                    "cmd": "python copy.py {} {}".format("foo", "foobar"),
                     "deps": ["foo"],
                     "outs": ["foobar", "bar"],
                 }