Repository: iterative/dvc

Issue:
Issue with csv files in windows system while they are under editing
I receive below error while csv is open and its modified. and dvc repro is executed.

$ dvc repro dvc/Dvcfile
Failed to reproduce 'dvc/Dvcfile': Failed to reproduce 'dvc\train.dvc': [WinError 6] The handle is invalid.



Discussion:
Could you run it again with `-v` flag and post logs here please?
```
$ dvc repro dvc/matrix.dvc -v
Data file or dir 'data\TestSet.csv' with cache '.dvc\cache\3e\3de0f9b0a1067bae03c182c845bf59' didn't change
Dvc file 'dvc\test.dvc' didn't change
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\site-packages\dvc\project.py", line 149, in _reproduce_stages
    result += self._reproduce_stage(stages, n, force)
  File "c:\programdata\anaconda3\lib\site-packages\dvc\project.py", line 128, in _reproduce_stage
    stage = stages[node].reproduce(force=force)
  File "c:\programdata\anaconda3\lib\site-packages\dvc\stage.py", line 99, in reproduce
    if not self.changed() and not force:
  File "c:\programdata\anaconda3\lib\site-packages\dvc\stage.py", line 78, in changed
    if entry.changed():
  File "c:\programdata\anaconda3\lib\site-packages\dvc\output.py", line 232, in changed
    ret = self._changed_file(self.path, self.cache)
  File "c:\programdata\anaconda3\lib\site-packages\dvc\output.py", line 198, in _changed_file
    System.samefile(path, cache) and \
  File "c:\programdata\anaconda3\lib\site-packages\dvc\system.py", line 44, in samefile
    return winlink.samefile(path1, path2)
  File "c:\programdata\anaconda3\lib\site-packages\ntfsutils\hardlink.py", line 30, in samefile
    info1 = fs.getfileinfo(path1)
  File "c:\programdata\anaconda3\lib\site-packages\ntfsutils\fs.py", line 111, in getfileinfo
    return _getfileinfo(path, 0)
  File "c:\programdata\anaconda3\lib\site-packages\ntfsutils\fs.py", line 96, in _getfileinfo
    raise WinError()
OSError: [WinError 6] The handle is invalid.

Failed to reproduce 'dvc/matrix.dvc': Failed to reproduce 'dvc\train.dvc': [WinError 6] The handle is invalid.
```
Hm, so you are running `dvc repro` while you have a file dependency opened and modified in another tab? If so, this is normal for Windows, since when you are modifying the file, its handle is changing, not allowing dvc to obtain file info using an older handle.
Though, this scenario is indeed not rare at all and can occur not only on Windows. We can probably try again if we didn't succeed to obtain file info the first time. I'll take a look shortly. Great catch, thank you!
Thanks. So far i have seen it with csv files under edit in Microsoft excel. Edition or under edit csv with notepad++ files just work fine with Notepad++
Hm, interesting. Well, at least notepad is not affected. I'll try to reproduce it myself later and see if I can do something about it.
Ok, I was able to reproduce even without Excel. Looking into it right now.
Sorry that it took so long, only managed to get to it right now. After a bit more digging I was able to reproduce the bug and figured that it was caused by the exclusive access on the file and getfileinfo() from ntfsutils using GENERIC_READ flag for CreateFile(). I've sent a PR to that project https://github.com/sunshowers/ntfs/pull/11 with a fix. I'll introduce a temporary workaround(along with a test) for us soon while that PR is being considered by the maintainer. 

Resolving patch:
diff --git a/dvc/system.py b/dvc/system.py
--- a/dvc/system.py
+++ b/dvc/system.py
@@ -39,11 +39,30 @@ def symlink(source, link_name):
     def reflink(source, link_name):
         return reflink.reflink(source, link_name)
 
+    # FIXME
+    # Temporary fix while waiting for the PR to be merged and released:
+    # https://github.com/sunshowers/ntfs/pull/11
+    @staticmethod
+    def getdirinfo(path):
+        from ntfsutils.fs import FILE_FLAG_BACKUP_SEMANTICS, FILE_SHARE_READ, OPEN_EXISTING
+        from ntfsutils.fs import CreateFile, WinError, BY_HANDLE_FILE_INFORMATION
+        from ntfsutils.fs import GetFileInformationByHandle, CloseHandle
+        flags = FILE_FLAG_BACKUP_SEMANTICS
+        hfile = CreateFile(path, 0, FILE_SHARE_READ, None, OPEN_EXISTING, flags, None)
+        if hfile is None:
+            raise WinError()
+        info = BY_HANDLE_FILE_INFORMATION()
+        rv = GetFileInformationByHandle(hfile, info)
+        CloseHandle(hfile)
+        if rv == 0:
+            raise WinError()
+        return info
+
     @staticmethod
     def inode(path):
         if System.is_unix():
             return os.stat(path).st_ino
 
         # getdirinfo from ntfsutils works on both files and dirs
-        info = getdirinfo(path)
+        info = System.getdirinfo(path)
         return hash((info.dwVolumeSerialNumber, info.nFileIndexHigh, info.nFileIndexLow))


Test patch:
diff --git a/tests/test_system.py b/tests/test_system.py
new file mode 100644
--- /dev/null
+++ b/tests/test_system.py
@@ -0,0 +1,19 @@
+import os
+
+from dvc.system import System
+from tests.basic_env import TestDir
+
+
+class TestSystem(TestDir):
+    def test_getdirinfo(self):
+        if os.name != 'nt':
+            return
+
+        # simulate someone holding an exclussive access
+        locked = 'locked'
+        fd = os.open(locked, os.O_WRONLY | os.O_CREAT | os.O_EXCL)
+
+        # should not raise WinError
+        info = System.getdirinfo(locked)
+
+        os.close(fd)