Repository: pre-commit/pre-commit

Issue:
Destructive operation on submodules with `submodule.recurse = true` git setting
### describe your issue

I added pre-commit to an ansible playbook repository that does extensive usage of git submodules for roles.

I had two submodules with extensive changes:

```shell
$ gst
On branch master
Your branch is ahead of 'origin/master' by 3 commits.
  (use "git push" to publish your local commits)
...
        modified:   roles/dnscrypt-proxy (modified content) <-- submodule with extensive (non-commited) changes
        modified:   roles/syncthing (modified content) <-- another submodule with extensive (non-commited) changes
...
```

Upon doing a commit I received an error (check ~/.cache/pre-commit/pre-commit.log section) due to having some problems in my repo and its submodules.
Notice that I cold not even run `git status` afterwards:
```shell
$ gst
fatal: not a git repository: roles/matomo/../../.git/modules/roles/matomo
```
After fixing the problems:
```shell
$ rm -rf roles/matomo/ .git/modules/roles/matomo/
$ git submodule update
Cloning into '/home/apoc/w/posg-ops/roles/matomo'...
Submodule path 'roles/matomo': checked out '######'
fatal: could not get a repository handle for submodule 'roles/php'
$ rm -rf roles/php/ .git/modules/roles/php
$ git submodule update
Cloning into '/home/apoc/w/posg-ops/roles/php'...
Submodule path 'roles/php': checked out '######'
```
and running `git apply` to have my changes back:
```shell
$ git apply /home/a666/.cache/pre-commit/patch1632709290-1076629
```

I noticed that my two submodules now had no changes in them... 
And unfortunately I see nothing in pre-commit cache area to help me recover them.

### pre-commit --version

pre-commit 2.15.0

### .pre-commit-config.yaml

```yaml
---
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.0.1
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-toml
      - id: check-added-large-files

  - repo: https://github.com/editorconfig-checker/editorconfig-checker.python
    rev: 2.3.54
    hooks:
      - id: editorconfig-checker

  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.51.0
    hooks:
      - id: terraform_validate
      - id: terraform_fmt
      - id: checkov
      - id: terrascan
      - id: terraform_tfsec
```


### ~/.cache/pre-commit/pre-commit.log (if present)

### version information

```
pre-commit version: 2.15.0
sys.version:
    3.9.7 (default, Aug 31 2021, 13:28:12)
    [GCC 11.1.0]
sys.executable: /home/a666/.local/pipx/venvs/pre-commit/bin/python
os.name: posix
sys.platform: linux
```

### error information

```
An unexpected error has occurred: CalledProcessError: command: ('/usr/lib/git-core/git', 'checkout', '--', '.')
return code: 255
expected return code: 0
stdout: (none)
stderr:
    Migrating git directory of 'roles/fail2ban' from
    '/home/a666/ops/roles/fail2ban/.git' to
    '/home/a666/ops/.git/modules/roles/fail2ban'
    fatal: not a git repository: ../../.git/modules/roles/matomo
    error: Submodule 'roles/matomo' could not be updated.
    Migrating git directory of 'roles/openproject' from
    '/home/a666/ops/roles/openproject/.git' to
    '/home/a666/ops/.git/modules/roles/openproject'
    Migrating git directory of 'roles/oryhydra' from
    '/home/a666/ops/roles/oryhydra/.git' to
    '/home/a666/ops/.git/modules/roles/oryhydra'
    fatal: not a git repository: ../../.git/modules/roles/php
    error: Submodule 'roles/php' could not be updated.

```

```
Traceback (most recent call last):
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/error_handler.py", line 65, in error_handler
    yield
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/main.py", line 368, in main
    return hook_impl(
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/commands/hook_impl.py", line 237, in hook_impl
    return retv | run(config, store, ns)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/commands/run.py", line 399, in run
    exit_stack.enter_context(staged_files_only(store.directory))
  File "/usr/lib/python3.9/contextlib.py", line 448, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/staged_files_only.py", line 92, in staged_files_only
    with _intent_to_add_cleared(), _unstaged_changes_cleared(patch_dir):
  File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/staged_files_only.py", line 61, in _unstaged_changes_cleared
    cmd_output_b('git', 'checkout', '--', '.', env=no_checkout_env)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/util.py", line 154, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/usr/lib/git-core/git', 'checkout', '--', '.')
return code: 255
expected return code: 0
stdout: (none)
stderr:
    Migrating git directory of 'roles/fail2ban' from
    '/home/a666/ops/roles/fail2ban/.git' to
    '/home/a666/ops/.git/modules/roles/fail2ban'
    fatal: not a git repository: ../../.git/modules/roles/matomo
    error: Submodule 'roles/matomo' could not be updated.
    Migrating git directory of 'roles/openproject' from
    '/home/a666/ops/roles/openproject/.git' to
    '/home/a666/ops/.git/modules/roles/openproject'
    Migrating git directory of 'roles/oryhydra' from
    '/home/a666/ops/roles/oryhydra/.git' to
    '/home/a666/ops/.git/modules/roles/oryhydra'
    fatal: not a git repository: ../../.git/modules/roles/php
    error: Submodule 'roles/php' could not be updated.
```



Discussion:
there's a test in our testsuite which shows this works correctly -- do you have more information on how to reproduce this?

looking at your output it looks like your submodules may have already been broken since `git` was unable to clean them manage them properly?

your report is also incomplete -- you have not shared the command you ran and its output so I cannot help you further without that
also, do you have any `post-checkout` hooks installed?
> looking at your output it looks like your submodules may have already been broken since `git` was unable to clean them manage them properly?

There was definitively a problem with them, but the changes to these submodules date from months back, there's around 20 commits following those changes.

> your report is also incomplete -- you have not shared the command you ran and its output so I cannot help you further without that

From the history, it was a simple `git commit -v`. 
Unfortunately I no longer have the pane history due to machine reboot, the error displayed was very similar to the error information section, sorry about that.

> also, do you have any `post-checkout` hooks installed?

None.

without a reproduction there isn't really much I can fix here :( -- the operations run are _supposed_ to be safe (especially because [`--recurse-submodules`](https://git-scm.com/docs/git-checkout#Documentation/git-checkout.txt---recurse-submodules) is not passed) so I can only guess there's some bug with `git` itself that caused your issue
ahah -- is it possible you have this setting enabled? https://git-scm.com/docs/git-config#Documentation/git-config.txt-submodulerecurse

let me see if that's what can cause this ?
Well, apparently the last sysadmin decided to put it in the config, and I ended getting it too after rsyncking the repo from his account...

```shell
$ git config --list --show-origin | rg recurse
file:.git/config        submodule.recurse=true
```
rough!  well I'll at least be able to fix that by ignoring that setting for pre-commit's programmatic uses of git
So if I understand well, having `submodule.recurse=true` made pre-commit run a `git checkout .` inside the submodules?
`submodule.recurse = 1` makes `git checkout -- .` also run `git checkout -- .` in each submodule -- seems like a really bad `git` feature
Well, submodules were always like a weird git workaround to a difficult problem (I think) and not very though out...

Resolving patch:
diff --git a/pre_commit/staged_files_only.py b/pre_commit/staged_files_only.py
--- a/pre_commit/staged_files_only.py
+++ b/pre_commit/staged_files_only.py
@@ -13,6 +13,12 @@
 
 logger = logging.getLogger('pre_commit')
 
+# without forcing submodule.recurse=0, changes in nested submodules will be
+# discarded if `submodule.recurse=1` is configured
+# we choose this instead of `--no-recurse-submodules` because it works on
+# versions of git before that option was added to `git checkout`
+_CHECKOUT_CMD = ('git', '-c', 'submodule.recurse=0', 'checkout', '--', '.')
+
 
 def _git_apply(patch: str) -> None:
     args = ('apply', '--whitespace=nowarn', patch)
@@ -58,7 +64,7 @@ def _unstaged_changes_cleared(patch_dir: str) -> Generator[None, None, None]:
 
         # prevent recursive post-checkout hooks (#1418)
         no_checkout_env = dict(os.environ, _PRE_COMMIT_SKIP_POST_CHECKOUT='1')
-        cmd_output_b('git', 'checkout', '--', '.', env=no_checkout_env)
+        cmd_output_b(*_CHECKOUT_CMD, env=no_checkout_env)
 
         try:
             yield
@@ -74,7 +80,7 @@ def _unstaged_changes_cleared(patch_dir: str) -> Generator[None, None, None]:
                 # We failed to apply the patch, presumably due to fixes made
                 # by hooks.
                 # Roll back the changes made by hooks.
-                cmd_output_b('git', 'checkout', '--', '.', env=no_checkout_env)
+                cmd_output_b(*_CHECKOUT_CMD, env=no_checkout_env)
                 _git_apply(patch_filename)
 
             logger.info(f'Restored changes from {patch_filename}.')


Test patch:
diff --git a/tests/staged_files_only_test.py b/tests/staged_files_only_test.py
--- a/tests/staged_files_only_test.py
+++ b/tests/staged_files_only_test.py
@@ -181,9 +181,11 @@ def test_img_conflict(img_staged, patch_dir):
 
 
 @pytest.fixture
-def submodule_with_commits(tempdir_factory):
+def repo_with_commits(tempdir_factory):
     path = git_dir(tempdir_factory)
     with cwd(path):
+        open('foo', 'a+').close()
+        cmd_output('git', 'add', 'foo')
         git_commit()
         rev1 = cmd_output('git', 'rev-parse', 'HEAD')[1].strip()
         git_commit()
@@ -196,18 +198,21 @@ def checkout_submodule(rev):
 
 
 @pytest.fixture
-def sub_staged(submodule_with_commits, tempdir_factory):
+def sub_staged(repo_with_commits, tempdir_factory):
     path = git_dir(tempdir_factory)
     with cwd(path):
+        open('bar', 'a+').close()
+        cmd_output('git', 'add', 'bar')
+        git_commit()
         cmd_output(
-            'git', 'submodule', 'add', submodule_with_commits.path, 'sub',
+            'git', 'submodule', 'add', repo_with_commits.path, 'sub',
         )
-        checkout_submodule(submodule_with_commits.rev1)
+        checkout_submodule(repo_with_commits.rev1)
         cmd_output('git', 'add', 'sub')
         yield auto_namedtuple(
             path=path,
             sub_path=os.path.join(path, 'sub'),
-            submodule=submodule_with_commits,
+            submodule=repo_with_commits,
         )
 
 
@@ -242,6 +247,34 @@ def test_sub_something_unstaged(sub_staged, patch_dir):
     _test_sub_state(sub_staged, 'rev2', 'AM')
 
 
+def test_submodule_does_not_discard_changes(sub_staged, patch_dir):
+    with open('bar', 'w') as f:
+        f.write('unstaged changes')
+
+    foo_path = os.path.join(sub_staged.sub_path, 'foo')
+    with open(foo_path, 'w') as f:
+        f.write('foo contents')
+
+    with staged_files_only(patch_dir):
+        with open('bar') as f:
+            assert f.read() == ''
+
+        with open(foo_path) as f:
+            assert f.read() == 'foo contents'
+
+    with open('bar') as f:
+        assert f.read() == 'unstaged changes'
+
+    with open(foo_path) as f:
+        assert f.read() == 'foo contents'
+
+
+def test_submodule_does_not_discard_changes_recurse(sub_staged, patch_dir):
+    cmd_output('git', 'config', 'submodule.recurse', '1', cwd=sub_staged.path)
+
+    test_submodule_does_not_discard_changes(sub_staged, patch_dir)
+
+
 def test_stage_utf8_changes(foo_staged, patch_dir):
     contents = '\u2603'
     with open('foo', 'w', encoding='UTF-8') as foo_file: