Repository: iterative/dvc

Issue:
assert inode >=0
I have ~23,000 files that I want to add to DVC. remote backing is S3 bucket. These are underneath several layers of directories. Eg S3 folder is called S1. I am trying to dvc add S1/X1/Y1/23,000 files.

First attempt failed with 
"assert inode >= 0".

I wanted to rule out the possibility of the nested subdirectories causing the issue. So on S3, I created subfolders X1 and Y1.

Now it fails again.

Here is the output:
dvc add -v *
Debug: PRAGMA user_version;
Debug: fetched: [(2,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 2;
Saving '10000.wav' to cache '../../../../.dvc/cache'.
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(0,)]
Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
Error: Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/main.py", line 22, in main
    ret = cmd.run_cmd()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/command/base.py", line 40, in run_cmd
    return self.run()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/command/add.py", line 10, in run
    self.project.add(target, recursive=self.args.recursive)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/project.py", line 173, in add
    stage.save()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/stage.py", line 393, in save
    out.save()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/output/local.py", line 113, in save
    self.info = self.project.cache.local.save(self.path_info)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/remote/local.py", line 346, in save
    return self._save_file(path_info)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/remote/local.py", line 297, in _save_file
    md5 = self.state.update(path)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/state.py", line 300, in update
    return self._do_update(path)[0]
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/state.py", line 257, in _do_update
    inode = self.inode(path)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/state.py", line 250, in inode
    return System.inode(path)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/system.py", line 188, in inode
    assert inode >= 0
AssertionError

Error: Unexpected error:

Having any troubles? Hit us up at dvc.org/support, we are always happy to help!

With pdb added to /home/ubuntu/.local/lib/python2.7/site-packages/dvc/system.py:

dvc add -v *
Debug: PRAGMA user_version;
Debug: fetched: [(2,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 2;
Saving '10000.wav' to cache '../../../../.dvc/cache'.
> /home/ubuntu/.local/lib/python2.7/site-packages/dvc/system.py(190)inode()
-> assert inode >= 0
(Pdb) inode
-7187427192193941402
(Pdb) c
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(0,)]
Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
Error: Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/main.py", line 22, in main
    ret = cmd.run_cmd()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/command/base.py", line 40, in run_cmd
    return self.run()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/command/add.py", line 10, in run
    self.project.add(target, recursive=self.args.recursive)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/project.py", line 173, in add
    stage.save()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/stage.py", line 393, in save
    out.save()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/output/local.py", line 113, in save
    self.info = self.project.cache.local.save(self.path_info)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/remote/local.py", line 346, in save
    return self._save_file(path_info)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/remote/local.py", line 297, in _save_file
    md5 = self.state.update(path)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/state.py", line 300, in update
    return self._do_update(path)[0]
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/state.py", line 257, in _do_update
    inode = self.inode(path)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/state.py", line 250, in inode
    return System.inode(path)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/dvc/system.py", line 190, in inode
    assert inode >= 0
AssertionError

Error: Unexpected error:

Having any troubles? Hit us up at dvc.org/support, we are always happy to help!

================
dvc --version
0.19.12

pip install

Linux  4.4.0-1060-aws #69-Ubuntu SMP S x86_64 GNU/Linux


Discussion:
I'm going to try tarring all files and checking in just the tarred file.
Hi @satya-venneti !

This is definitely a bug. I'm looking into it right now.
Found the issue https://bugs.python.org/issue29619 . Turns out python's os.stat() casts st_ino to signed type, which makes it negative for large enough inodes. This bug was fixed in 3.7, but we should add a workaround to dvc for it, since we need to work on older python versions as well. Preparing a patch right now.

Thank you for the feedback!

Resolving patch:
diff --git a/dvc/system.py b/dvc/system.py
--- a/dvc/system.py
+++ b/dvc/system.py
@@ -178,7 +178,12 @@ class BY_HANDLE_FILE_INFORMATION(Structure):
     @staticmethod
     def inode(path):
         if System.is_unix():
+            import ctypes
             inode = os.lstat(path).st_ino
+            # NOTE: See https://bugs.python.org/issue29619 and
+            # https://stackoverflow.com/questions/34643289/
+            # pythons-os-stat-is-returning-wrong-inode-value
+            inode = ctypes.c_ulong(inode).value
         else:
             # getdirinfo from ntfsutils works on both files and dirs
             info = System.getdirinfo(path)


Test patch: