Repository: iterative/dvc

Issue:
dvc: .dvcignore trouble with nfs mounted directory
I have a large NFS mounted in a directory that I would like dvc to ignore. 

Directory Structure:
```
directory
|___nfs
|___...
|___.dvc
|___.dvcignore
```
My *.dvcignore* has the following line:
`/nfs/` (I've tried `nfs/` and `nfs/*`)

The problem is that when I run `dvc status` or `dvc pull` the processes will just hang:
```
DEBUG: PRAGMA user_version;
DEBUG: fetched: [(3,)]
DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
DEBUG: PRAGMA user_version = 3; 
```

Here is the traceback from `KeyboardInterrupt`:
```
   File "/home/ec2-user/app/proc/.env/lib/python3.7/site-packages/dvc/repo/__init__.py", line 499, in dvcignore
    return DvcIgnoreFilter(self.root_dir)
  File "/home/ec2-user/app/proc/.env/lib/python3.7/site-packages/dvc/ignore.py", line 67, in __init__
    for root, dirs, _ in os.walk(root_dir):
  File "/home/ec2-user/app/proc/.env/lib64/python3.7/os.py", line 410, in walk
    yield from walk(new_path, topdown, onerror, followlinks)
  File "/home/ec2-user/app/proc/.env/lib64/python3.7/os.py", line 368, in walk
    is_dir = entry.is_dir() 
```
Which makes me feel like the directory is not being ignored.

***Additonal***
I've unmounted the NFS directory and ran `dvc status` with no problem so I believe the issue stems from dvc trying to traverse it.

System Information:
``` 
DVC version: 0.66.6
Python version: 3.7.4
Platform: Linux 4.14.109-99.92.amzn2.x86_64
Installation: pip
```




Discussion:
thanks a lot for the report, @chadlohrli !
Hi @chadlohrli !

So nfs is mounted outside of your dvc repo, right? And where is your dvcignore?
@mroutis could you please include a link to the Discord discussion as we usually do?
> So nfs is mounted outside of your dvc repo, right? And where is your dvcignore?

The nfs is mounted **inside** my dvc repo. The .dvcignore is at the same level as the nfs mount directory and .dvc directory. (I've tried moving the .dvcignore file around but no luck)

> I have a large NFS mounted in a directory that I would like dvc to ignore.

@chadlohrli  As far as I know, `.dvcignore` ignores only DVC-files (those that have extension `.dvc`). So, I would say it works as designed, although this might not be what is expected.

To understand better how `.dvcignore` works, please check this example: https://katacoda.com/dvc/courses/examples/dvcignore
@dashohoxha I don't believe that's correct. 
Here is the official documentation: https://dvc.org/doc/user-guide/dvcignore
@chadlohrli Got it! Sorry, only now noticed that you've provided that info in the PR comments. Could you please try adding `/nfs` to `.gitignore` and committing those changes? Have a suspicion that it might be git that is hanging there.
@efiop tried adding `/nfs` to the root .gitignore file and committing. No luck.
@chadlohrli Thanks for the info! So probably a bug on our side, looking into it...
@chadlohrli Indeed, looks like our dvcignore filter has a midlife crisis and has no respect for himself :smile:  Jokes aside, it simply doesn't account for itself when collecting dvcignores. Preparing a patch right now.

Resolving patch:
diff --git a/dvc/ignore.py b/dvc/ignore.py
--- a/dvc/ignore.py
+++ b/dvc/ignore.py
@@ -6,6 +6,7 @@
 from pathspec import PathSpec
 from pathspec.patterns import GitWildMatchPattern
 
+from dvc.utils import dvc_walk
 from dvc.utils import relpath
 from dvc.utils.compat import open
 
@@ -47,6 +48,9 @@ def __hash__(self):
         return hash(self.ignore_file_path)
 
     def __eq__(self, other):
+        if not isinstance(other, DvcIgnorePatterns):
+            return NotImplemented
+
         return self.ignore_file_path == other.ignore_file_path
 
 
@@ -59,12 +63,21 @@ def __call__(self, root, dirs, files):
 
         return dirs, files
 
+    def __hash__(self):
+        return hash(tuple(self.basenames))
+
+    def __eq__(self, other):
+        if not isinstance(other, DvcIgnoreDirs):
+            return NotImplemented
+
+        return self.basenames == other.basenames
+
 
 class DvcIgnoreFilter(object):
     def __init__(self, root_dir):
         self.ignores = {DvcIgnoreDirs([".git", ".hg", ".dvc"])}
         self._update(root_dir)
-        for root, dirs, _ in os.walk(root_dir):
+        for root, dirs, _ in dvc_walk(root_dir, self):
             for d in dirs:
                 self._update(os.path.join(root, d))
 


Test patch:
diff --git a/tests/func/test_ignore.py b/tests/func/test_ignore.py
--- a/tests/func/test_ignore.py
+++ b/tests/func/test_ignore.py
@@ -5,6 +5,9 @@
 
 from dvc.exceptions import DvcIgnoreInCollectedDirError
 from dvc.ignore import DvcIgnore
+from dvc.ignore import DvcIgnoreDirs
+from dvc.ignore import DvcIgnoreFilter
+from dvc.ignore import DvcIgnorePatterns
 from dvc.utils.compat import cast_bytes
 from dvc.utils.fs import get_mtime_and_size
 from tests.basic_env import TestDvc
@@ -131,3 +134,21 @@ def test_should_raise_on_dvcignore_in_out_dir(dvc_repo, repo_dir):
 
     with pytest.raises(DvcIgnoreInCollectedDirError):
         dvc_repo.add(repo_dir.DATA_DIR)
+
+
+@pytest.mark.parametrize("dname", [TestDvc.DATA_DIR, TestDvc.DATA_SUB_DIR])
+def test_ignore_collecting_dvcignores(repo_dir, dname):
+    top_ignore_file = os.path.join(
+        repo_dir.root_dir, os.path.dirname(dname), DvcIgnore.DVCIGNORE_FILE
+    )
+    repo_dir.create(top_ignore_file, os.path.basename(dname))
+
+    ignore_file = os.path.join(
+        repo_dir.root_dir, dname, DvcIgnore.DVCIGNORE_FILE
+    )
+    repo_dir.create(ignore_file, repo_dir.FOO)
+
+    assert DvcIgnoreFilter(repo_dir.root_dir).ignores == {
+        DvcIgnoreDirs([".git", ".hg", ".dvc"]),
+        DvcIgnorePatterns(top_ignore_file),
+    }