Repository: iterative/dvc

Issue:
cannot rebase if a new file is created which is not present in .gitignore in the base branch
**Please provide information about your setup**
DVC version(i.e. `dvc --version`), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))

Note: checking out '171d6c61b9a643be0d2d998f2d3b644c33bc4e4c'.                          
I created some new files namely cavchon.parquet and cavachon_result.csv.
I am getting the following error when I am trying to rebase it to the base branch.
                                                                                       
You are in 'detached HEAD' state. You can look around, make experimental                
changes and commit them, and you can discard any commits you make in this               
state without impacting any branches by performing another checkout.                    
                                                                                        
If you want to create a new branch to retain commits you create, you may                
do so (now or later) by using -b with the checkout command again. Example:              
                                                                                        
  git checkout -b <new-branch-name>                                                     
                                                                                        
HEAD is now at 171d6c6 ENH: add the device modifier based on adgroup type and device typ
WARNING: Cache 'd273ac33145468f344dd2a08b5dae827' not found. File 'data/cavachon_result.
WARNING: data 'data/beagle_predictions.parquet' exists. Removing before checkout.
WARNING: Cache '78e5b1b8677dec94d2f2f64b5c7b649f' not found. File 'data/cavachon.parquet
WARNING: data 'data/beagle.parquet' exists. Removing before checkout.
WARNING: data 'models/beagle.joblib' exists. Removing before checkout.
WARNING: data 'data/beagle_train.parquet' exists. Removing before checkout.
WARNING: data 'data/beagle_test.parquet' exists. Removing before checkout.
ERROR: unexpected error - Checkout failed for following targets:
 data/cavachon_result.csv
data/cavachon.parquet
Did you forget to fetch?


Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
could not detach HEAD



Discussion:
Hi @ankurrajdev !

Could you provide `dvc version` output, please?

Do you have our git hooks installed?
> Hi @ankurrajdev !
> 
> Could you provide `dvc version` output, please?
> 
> Do you have our git hooks installed?

 dvc version is 0.80.0.

Yes I have installed the git hooks.
I'm not able to understand the problem :see_no_evil: 

@efiop , @ankurrajdev , could you provide a reproduction script for it?
@mroutis When rebasing git runs `git checkout`, which triggers `dvc checkout`, which is not able to checkout and looks like that is interrupting the rebase. Need to take a closer look.
@ankurrajdev For now a workaround would be to disable (remove) .git/hooks/post-checkout.py , it would prevent `dvc checkout` from being run automatically after `git checkout`, but you could still run `dvc checkout` by-hand.
> @ankurrajdev For now a workaround would be to disable (remove) .git/hooks/post-checkout.py , it would prevent `dvc checkout` from being run automatically after `git checkout`, but you could still run `dvc checkout` by-hand.

Got it thanks. I copied everything again and created a new branch and it's working right now. If it happens again, I'll let you know.

Thanks for the support.


@efiop , I think `git rebase` with `--no-verify` option should also work.
@efiop  @mroutis 
I am getting the same error, but this time I have a clear picture.
For instance the base branch is A. My teammate and I branched out to branches B and C respectively. My teammate merged branch B to A, so I need to rebase. In branch C, on which I was working, I deleted a few files which were present in both branch B and C. So, when I am trying to rebase, I am getting an error 'File not found'. 

![image](https://user-images.githubusercontent.com/33640392/72903022-7de6f500-3cfa-11ea-88d2-cca8c709a14d.png)


In the above screenshot, all the parquet files are present in branch A and B but not in C.

I tried rebasing with --no-verify as well.

Any suggestions how to deal with it?
@ankurrajdev Thanks for clarifying! As I've noted before, the workaround is to remove .git/hooks/post-checkout, so that this hook won't run at all. As to proper solution, we need to make our checkout hook smarter and not run checkouts on rebases. Need to take a closer look.
Okay thanks. Just wanted to clarify the issue :)

Resolving patch:
diff --git a/dvc/scm/git/__init__.py b/dvc/scm/git/__init__.py
--- a/dvc/scm/git/__init__.py
+++ b/dvc/scm/git/__init__.py
@@ -247,11 +247,12 @@ def list_tags(self):
     def list_all_commits(self):
         return [c.hexsha for c in self.repo.iter_commits("--all")]
 
-    def _install_hook(self, name, cmd):
-        command = (
-            '[ "$3" = "0" ]'
-            ' || [ -z "$(git ls-files --full-name .dvc)" ]'
-            " || exec dvc {}".format(cmd)
+    def _install_hook(self, name, preconditions, cmd):
+        # only run in dvc repo
+        in_dvc_repo = '[ -n "$(git ls-files --full-name .dvc)" ]'
+
+        command = "if {}; then exec dvc {}; fi".format(
+            " && ".join([in_dvc_repo] + preconditions), cmd
         )
 
         hook = self._hook_path(name)
@@ -269,9 +270,19 @@ def _install_hook(self, name, cmd):
     def install(self):
         self._verify_dvc_hooks()
 
-        self._install_hook("post-checkout", "checkout")
-        self._install_hook("pre-commit", "status")
-        self._install_hook("pre-push", "push")
+        self._install_hook(
+            "post-checkout",
+            [
+                # checking out some reference and not specific file.
+                '[ "$3" = "1" ]',
+                # check that we are on some branch/tag and not in detached HEAD
+                # state, so we don't accidentally break a rebase.
+                '[ "$(git rev-parse --abbrev-ref HEAD)" != "HEAD" ]',
+            ],
+            "checkout",
+        )
+        self._install_hook("pre-commit", [], "status")
+        self._install_hook("pre-push", [], "push")
 
     def cleanup_ignores(self):
         for path in self.ignored_paths:


Test patch: