Repository: iterative/dvc

Issue:
Feature request: Better support for folder dependencies
Say I have the following structure:

```
script.py
module/submodule/foo.py
```

Then:

```
dvc run script.py -d module -o bar.npy
```

If I modify foo.py, `dvc repro` does not notice. The reason is that neither the `inode` nor the `mtime` of `module` is updated when a modification is made to `foo.py`.

For now it is possible to add all the files in `module` as dependencies but it would surely be better to have a way to handle folders recursively.


Discussion:
Hi @tdeboissiere !

Great point! Preparing a patch right now.

Thanks,
Ruslan
Hi @tdeboissiere !

The patch is merged, I'm releasing 0.18.6 with it right now. Should be ready in an hour or so. I'll let you know when it is out. Thank you so much for the feedback!
@tdeboissiere 0.18.6 is out :slightly_smiling_face:  Please feel free to upgrade and give it a try.

Thanks,
Ruslan
Lightning fast as usual, the fix seems to work well !

2 observations:

- The same problem seems to exist for outputs. If I have `dvc run -d XXX -o output_folder`, then `dvc repro output_folder.dvc` will not reproduce the steps if something within `output_folder` is changed.

- Say I have a pipeline with two stages, and each stage has the same `-d folder` dependency. In that case, if I change a file in `folder` which would only be used in the second stage, `dvc repro` will reproduce the first stage as well because of the `-d folder` dependency. Solving this problem would require DVC to know in advance all the files which are used at a given stage, which is probably a bit tricky and best left to the user...
Hi @tdeboissiere !

> The same problem seems to exist for outputs. If I have dvc run -d XXX -o output_folder, then dvc repro output_folder.dvc will not reproduce the steps if something within output_folder is changed.

Are you experiencing it even with the never version?

> Say I have a pipeline with two stages, and each stage has the same -d folder dependency. In that case, if I change a file in folder which would only be used in the second stage, dvc repro will reproduce the first stage as well because of the -d folder dependency. Solving this problem would require DVC to know in advance all the files which are used at a given stage, which is probably a bit tricky and best left to the user...

Yes, this is something that should be handled by the user. Dvc can only know that you are using something if you explicitly specify it with `-d`. The scenario you've described can be worked around by separating files in a folder appropriately and then specifying separated and common parts as dependencies in your pipeline stages. I.e. something like:
```
$ dvc run -d common_dir -d dir_1 ...
$ dvc run -d common_dir -d dir_2 ...
```

Thanks,
Ruslan
- Yes, even with the new version I have verified that the problem exists for outputs.
- Thanks for the tip for my second point ! Rather than splitting sub directories, I think it is simpler if I externally enforce the same git status for all my depencies at all stages.
Hm, I'll investigate that ASAP, seems like a bug. Thank you for reporting the issue!
Reopening the issue to track the progress on the `outputs` bug.
@tdeboissiere Got it. Was able to reproduce. Preparing a patch right now and will release 0.18.8 in an hour or so. Thank you!

Resolving patch:
diff --git a/dvc/state.py b/dvc/state.py
--- a/dvc/state.py
+++ b/dvc/state.py
@@ -96,18 +96,14 @@ def dump(self):
 
     @staticmethod
     def mtime(path):
+        mtime = os.path.getmtime(path)
         if os.path.isdir(path):
-            mtime = -1
             for root, dirs, files in os.walk(path):
                 for fname in files:
                     f = os.path.join(root, fname)
                     m = os.path.getmtime(f)
                     if m > mtime:
                         mtime = m
-            if mtime == -1:
-                mtime = os.path.getmtime(path)
-        else:
-            mtime = os.path.getmtime(path)
 
         return str(int(nanotime.timestamp(mtime)))
 


Test patch:
diff --git a/tests/test_repro.py b/tests/test_repro.py
--- a/tests/test_repro.py
+++ b/tests/test_repro.py
@@ -281,6 +281,12 @@ def test(self):
         self.assertEqual(len(stages), 1)
         self.assertTrue(stages[0] is not None)
 
+        # Check that dvc indeed registers changed output dir
+        shutil.move(self.BAR, dir_name)
+        stages = self.dvc.reproduce(stage.path)
+        self.assertEqual(len(stages), 1)
+        self.assertTrue(stages[0] is not None)
+
 
 class TestReproMissingMd5InStageFile(TestRepro):
     def test(self):