Repository: iterative/dvc

Issue:
dvc init fails if repository has files marked with --skip-worktree
Using dvc 0.9.3, running `dvc init` will yield `Unexpected Error: ` (but creates the `.dvc` directory anyway).

Observed in Python 3.6 on Windows and Python 3.5 on WSL (Ubuntu) - both installed from PyPI.


Discussion:
Hi @villasv !

Is that the full error that it throws?
Yep, no extra messages. And it's going directly to `stdout`, nothing to `stderr`.
Running `dvc init` again will output `Unexpected error: [Errno 17] File exists: '/mnt/c/Users/villasv/Projects/Gupy/red-datadriven/.dvc'` as expected, I guess. So error reporting should be working fine.
@villasv Trying to reproduce right now... Is `red-datadriven` a git repository?
Yes, with around 500 commits already.

In fact, I'm trying to reproduce this in a brand new repo and failing. So there must be something about this repo that makes `dvc init`fail.
@villasv Hm... Very interesting. Tried with a bunch of exsiting git repos of different forms and sizes but `dvc init` seems to work as expected. Any chance you could share that particular repo with me?
Btw, cloud you try running:
```
echo -e "import git\nrepo = git.Repo('.')" | python
```
In your repository?
Unfortunately I can't share it, so I'll have to probe it on my own.
`echo -e "import git\nrepo = git.Repo('.')" | python` outputs nothing, but doesn't error either.

I noticed that cloning my own repo anew, `dvc init` will work. So it must be something related to a specific `git` repo state.
Oh, I think I got something. The error starts happening when I use `git update-index --skip-worktree .env` to ignore the local environment file.
Ah, I see. Ok, lets try to bypass all the dvc exception traps by running the methods directly. Please run this script in your repo:
```
echo -e "from dvc.project import Project\nProject.init()" | python
```
```
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python3.5/dist-packages/dvc/project.py", line 75, in init
    scm.add([config.config_file, ignore_file])
  File "/usr/local/lib/python3.5/dist-packages/dvc/scm.py", line 116, in add
    self.repo.index.add(paths)
  File "/usr/local/lib/python3.5/dist-packages/git/index/base.py", line 793, in add    self.entries[(entry.path, 0)] = IndexEntry.from_base(entry)
  File "/usr/local/lib/python3.5/dist-packages/gitdb/util.py", line 256, in __getattr__
    self._set_cache_(attr)
  File "/usr/local/lib/python3.5/dist-packages/git/index/base.py", line 132, in _set_cache_
    self._deserialize(stream)
  File "/usr/local/lib/python3.5/dist-packages/git/index/base.py", line 161, in _deserialize
    self.version, self.entries, self._extension_data, conten_sha = read_cache(stream)  # @UnusedVariable
  File "/usr/local/lib/python3.5/dist-packages/git/index/fun.py", line 188, in read_cache
    version, num_entries = read_header(stream)
  File "/usr/local/lib/python3.5/dist-packages/git/index/fun.py", line 167, in read_header
    assert version in (1, 2)
AssertionError
```
Aha! So if we look into GitPython's code where that assertion points us:
```
def read_header(stream):                                               
    """Return tuple(version_long, num_entries) from the given stream"""
    type_id = stream.read(4)                                           
    if type_id != b"DIRC":                                             
        raise AssertionError("Invalid index file header: %r" % type_id)
    version, num_entries = unpack(">LL", stream.read(4 * 2))           
                                                                       
    # TODO: handle version 3: extended data, see read-cache.c          
    assert version in (1, 2)                                           
    return version, num_entries                                        
```
Very interesting, looking into solving it.
It seems that this won't affect `dvc add` or `dvc run` commands, so as long as I use `--skip-worktree` after running `dvc init` I'm good to go. I'll rename the issue accordingly.
You are correct, it should not affect other dvc commands, as it is actually the one and only place where dvc tries to add anything using git. But this is definitely an important issue to consider. Lets leave this issue open, I might get to digging into GitPython and implementing v3 and v4 indexes in the future.

Thank you!

Resolving patch:
diff --git a/dvc/scm.py b/dvc/scm.py
--- a/dvc/scm.py
+++ b/dvc/scm.py
@@ -1,6 +1,7 @@
 import os
 
 from dvc.exceptions import DvcException
+from dvc.logger import Logger
 
 
 class SCMError(DvcException):
@@ -131,7 +132,14 @@ def ignore_remove(self, path):
             fd.writelines(filtered)
 
     def add(self, paths):
-        self.repo.index.add(paths)
+        # NOTE: GitPython is not currently able to handle index version >= 3.
+        # See https://github.com/iterative/dvc/issues/610 for more details.
+        try:
+            self.repo.index.add(paths)
+        except AssertionError as exc:
+            msg = 'Failed to add \'{}\' to git. You can add those files manually using \'git add\'. '
+            msg += 'See \'https://github.com/iterative/dvc/issues/610\' for more details.'
+            Logger.error(msg.format(str(paths)), exc)
 
     def commit(self, msg):
         self.repo.index.commit(msg)


Test patch: