Repository: iterative/dvc

Issue:
`exp run`: doesn't work with submodule as dependency
# Bug Report

## Description
`dvc exp run` fails and `dvc repro` runs successfully when a cmd is executed from inside of a submodule and the submodule is included as a dependency.

### Reproduce
#### Set Up:
```bash
mkdir test
cd test

mkdir submodule
cd submodule
git init
echo "cp ../data/a.txt ../models/b.txt" > run.sh
git add run.sh
git commit -m "initial commit"

cd ..
mkdir main
cd main
git init
dvc init
dvc config cache.type=hardlink,symlink,copy
git submodule add ../submodule src
mkdir data
echo "test" > data/a.txt
mkdir models
dvc stage add -n cp -w src -d ../src -d ../data/a.txt -o ../models/b.txt bash run.sh
dvc repro
git add .
git commit -m "initial commit"
```

**This works**
`dvc repro -f`

**This fails:**
`dvc exp run -f`

**This also fails:**
`dvc exp run -f --temp`

### Expected

`dvc repro` and `dvc exp run` run and succeed similarly.

### Environment information
**Output of `dvc doctor`:**

```console
$ dvc doctor

DVC version: 2.9.2 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.11.0-1021-aws-x86_64-with-glibc2.29
Supports:
	hdfs (fsspec = 2021.10.1, pyarrow = 5.0.0),
	webhdfs (fsspec = 2021.10.1),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2021.10.1, boto3 = 1.17.106)
Cache types: hardlink, symlink
Cache directory: lustre on 172.16.38.30@tcp:/skl3jbmv
Caches: local
Remotes: s3
Workspace directory: lustre on 172.16.38.30@tcp:/skl3jbmv
Repo: dvc, git
```

**Additional Information (if any):**
There's an error in the verbose output that indicates
```console
ERROR: unexpected error - invalid data in index - invalid entry
```

`dvc exp run`: git-submodule destroys workspace
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description
If your repository contains a git submodule and a dvc stage executes a file from that submodule, producing an output in the project's root folder,  `dvc exp run` does not run the experiment successfully but leaves a cluttered/broken git repository behind.

<!--
A clear and concise description of what the bug is.
-->

### Reproduce
1. Create a local git repository, initialize with dvc.
2. Use another git repository, in my case I use a simple script `run.py` with the content:
```
import os

path = "model"
filename = "output.txt"

if not os.path.exists(path):
    os.makedirs(path)

with open(os.path.join(path, filename), 'wb') as temp_file:
    temp_file.write(b"Output")
```
to create a "fake" output.
3. Setup your `dvc.yaml` pipeline as:
```
stages:
  run-test:
    cmd: python external/run.py
    deps:
      - external/
    outs:
      - model/output.txt
```
4. Include the former subrepository with `git submodule add <my-subrepo-url> external/`
5. Commit the added submodule and all changes to your local main repository
6. Run `dvc exp run`
```
Reproducing experiment 'meaty-teff'
Building workspace index                                                                                                                                                               
Comparing indexes                                                                                                                                                                     
Applying changes                                                                                                                                                                       
Stage 'run-test' didn't change, skipping
ERROR: unexpected error - invalid data in index - invalid entry
```
After termination, the workspace is heavily cluttered by git changes like:
```
      new file:   .git/objects/ff/b6ed23131c48887b35e4e0d9e9bb8954b547bb
      new file:   .git/objects/ff/b81afb96b85488c92dd2a4ce0c7ebf68f533f6
      new file:   .git/objects/ff/b87e48727d95d793d8bb17ed20bc847107339f
      new file:   .git/objects/ff/b8e5f7ed10543ab4940212978c3eea6dd6d19f
      new file:   .git/objects/ff/b999f452746ae926fe6592eb4fc499804072c9
      new file:   .git/objects/ff/ba93b2cb261ac8c0235c4608cb6d0e79087078
      new file:   .git/objects/ff/bc49cd878675079f05cc65d0ebfa42590d52d3
      new file:   .git/objects/ff/bd5f4a623f5a0dd293e555e73f2f6c17be9cb2
      new file:   .git/objects/ff/bdc6ad64e18a10a82e7ea828fbdb2ed1c4fab6
      new file:   .git/objects/ff/c0207458f1ea7d2b70e9f4ba6d81107ac32147
      new file:   .git/objects/ff/c1d5324ceae9a256cd0a5180ed4b05012eb26c
      new file:   .git/objects/ff/c1dc8332b015572fe01371fde74173a6087aaf
      new file:   .git/objects/ff/c629a3e3484340fc326324889e4aa705164517
      new file:   .git/objects/ff/c7535b45a35b7e39f18332bdbf2e722a3104a6
      new file:   .git/objects/ff/c79bd1e5b4f91c81a40816d090880962d4a746
```
As well as changes like
```
        modified:   .git/HEAD
        modified:   .git/index
        modified:   .git/logs/HEAD
        deleted:    .git/refs/exps/exec/EXEC_BASELINE
        deleted:    .git/refs/exps/exec/EXEC_MERGE
```
7. See the full stacktrace attached at the end

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

### Expected
This setup used to work in the past. I reactivated a training-pipeline with a setup like this recently and found it broken. `dvc exp run` should execute the job and successfully add the experiment to the git refs for further consumption.
<!--
A clear and concise description of what you expect to happen.
-->

### Environment information
Tested with Ubuntu 24.04 and python 3.10 as well as Windows WSL2 Ubuntu 24.04 Python 3.12.
<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
DVC version: 3.61.0 (pip)
-------------------------
Platform: Python 3.12.3 on Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39
Subprojects:
        dvc_data = 3.16.10
        dvc_objects = 5.1.1
        dvc_render = 1.0.2
        dvc_task = 0.40.2
        scmrepo = 3.4.0
Supports:
        http (aiohttp = 3.12.15, aiohttp-retry = 2.9.1),
        https (aiohttp = 3.12.15, aiohttp-retry = 2.9.1),
        s3 (s3fs = 2025.7.0, boto3 = 1.39.8)
Config:
        Global: /home/hemker/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdd
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/sdd
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/cf372b30375dfe9c40a81de6a73671ef
```

**Additional Information (if any):**
Hint: I modified the `scmrepo` files to temporarily exclude the `pygit2`-backend. This solves the error as `scmrepo` then iterates over the remaining `dulwich` and `gitpython` backends:

File changed: `lib/python3.12/site-packages/scmrepo/git/__init__.py`
```
class GitBackends(Mapping):
    DEFAULT: ClassVar[dict[str, BackendCls]] = {
        "dulwich": DulwichBackend,
        #"pygit2": Pygit2Backend, <-------
        "gitpython": GitPythonBackend,
    }
```

Output of `dvc exp run -vvv`.
```
Traceback (most recent call last):
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
          ^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/cli/command.py", line 30, in do_run
    return self.run()
           ^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/commands/experiments/run.py", line 14, in run
    self.repo.experiments.run(
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/experiments/__init__.py", line 354, in run
    return run(self.repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/experiments/run.py", line 77, in run
    return repo.experiments.reproduce_one(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/experiments/__init__.py", line 126, in reproduce_one
    results = self._reproduce_queue(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/experiments/utils.py", line 62, in wrapper
    ret = f(exp, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/experiments/__init__.py", line 249, in _reproduce_queue
    exec_results = queue.reproduce(copy_paths=copy_paths, message=message)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/experiments/queue/workspace.py", line 93, in reproduce
    self._reproduce_entry(
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/experiments/queue/workspace.py", line 137, in _reproduce_entry
    executor.cleanup(infofile)
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/dvc/repo/experiments/executor/local.py", line 251, in cleanup
    with self._detach_stack:
  File "/usr/lib/python3.12/contextlib.py", line 610, in __exit__
    raise exc_details[1]
  File "/usr/lib/python3.12/contextlib.py", line 595, in __exit__
    if cb(*exc_details):
       ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/scmrepo/git/__init__.py", line 468, in detach_head
    self.reset()
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/scmrepo/git/__init__.py", line 308, in _backend_func
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/scmrepo/git/backend/pygit2/__init__.py", line 868, in reset
    self.repo.index.read(False)
    ^^^^^^^^^^^^^^^
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/pygit2/repository.py", line 650, in index
    check_error(err, io=True)
  File "/home/hemker/dev/submodule-test/env/lib/python3.12/site-packages/pygit2/errors.py", line 66, in check_error
    raise GitError(message)
_pygit2.GitError: invalid data in index - invalid entry
```



Discussion:
Wondering if this is related to an earlier resolved issue that I posted about. https://github.com/iterative/dvc/issues/5740
@pmrowla ping
This is definitely a bug for the `--temp/--queue` use case, we don't handle submodules at all there. We probably need to be doing submodule pull/update in the temp workspaces that we create.

I'm not sure why `exp run` does not work for workspace runs though, as we should not have to do anything special there, since the submodule is already set up properly.

@woodshop it would help if you could provide the entire error traceback from the `-v` command output, the error message alone doesn't provide us with enough information to debug the issue.
Apologies for the slow reply!

```shell
(tensorflow-stable) asarroff@neu:/fsx/test/main$ dvc exp run -f -v
2022-01-24 16:52:02,910 DEBUG: Adding '/fsx/test/main/.dvc/config.local' to gitignore file.
2022-01-24 16:52:02,921 DEBUG: Adding '/fsx/test/main/.dvc/tmp' to gitignore file.
2022-01-24 16:52:02,922 DEBUG: Adding '/fsx/test/main/.dvc/cache' to gitignore file.
2022-01-24 16:52:03,523 DEBUG: Stashed experiment '3e6d5d6' with baseline '1fab656' for future execution.
2022-01-24 16:52:03,574 DEBUG: Reproducing experiment revs '3e6d5d6'
2022-01-24 16:52:03,711 DEBUG: Init workspace executor in '/fsx/test/main'
2022-01-24 16:52:03,841 DEBUG: Adding '/fsx/test/main/.dvc/config.local' to gitignore file.
2022-01-24 16:52:03,849 DEBUG: Adding '/fsx/test/main/.dvc/tmp' to gitignore file.
2022-01-24 16:52:03,849 DEBUG: Adding '/fsx/test/main/.dvc/cache' to gitignore file.
2022-01-24 16:52:03,853 DEBUG: Running repro in '/fsx/test/main'
2022-01-24 16:52:03,853 DEBUG: Removing '/fsx/test/main/.dvc/tmp/repro.dat'
2022-01-24 16:52:05,455 DEBUG: Removing output 'models/b.txt' of stage: 'cp'.
2022-01-24 16:52:05,455 DEBUG: Removing '/fsx/test/main/models/b.txt'
Running stage 'cp':
> bash run.sh
2022-01-24 16:52:05,551 DEBUG: staged tree 'object md5: 7836100ad7371e5f9125fbeb2b24a8e5.dir'
2022-01-24 16:52:05,552 DEBUG: state save (144115339624539791, 16d176444d6ed86e1a7e908b91b81625, 33) 7836100ad7371e5f9125fbeb2b24a8e5.dir
2022-01-24 16:52:05,557 DEBUG: Adding '/fsx/test/main/models/b.txt' to gitignore file.
2022-01-24 16:52:05,567 DEBUG: state save (144115339624540011, 1643061125000000000, 5) d8e8fca2dc0f896fd7cb4cb0031ba249
2022-01-24 16:52:05,583 DEBUG: state save (144115339624540011, 1643061125000000000, 5) d8e8fca2dc0f896fd7cb4cb0031ba249
2022-01-24 16:52:05,585 DEBUG: Computed stage: 'cp' md5: '44286a707e35ea3bf08062b5fe4b7152'
2022-01-24 16:52:05,595 DEBUG: staged tree 'object md5: 7836100ad7371e5f9125fbeb2b24a8e5.dir'
2022-01-24 16:52:05,596 DEBUG: state save (144115339624539791, 16d176444d6ed86e1a7e908b91b81625, 33) 7836100ad7371e5f9125fbeb2b24a8e5.dir
2022-01-24 16:52:05,607 DEBUG: staged tree 'object md5: 7836100ad7371e5f9125fbeb2b24a8e5.dir'
2022-01-24 16:52:05,607 DEBUG: state save (144115339624539791, 16d176444d6ed86e1a7e908b91b81625, 33) 7836100ad7371e5f9125fbeb2b24a8e5.dir
2022-01-24 16:52:05,622 DEBUG: Preparing to transfer data from '/fsx/test/main/.dvc/cache' to '/fsx/test/main/.dvc/cache'
2022-01-24 16:52:05,627 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>: [Errno 95] Operation not supported
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 28, in _link
    func(from_path, to_path)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/local.py", line 148, in reflink
    System.reflink(from_info, to_info)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/system.py", line 112, in reflink
    System._reflink_linux(source, link_name)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/system.py", line 96, in _reflink_linux
    fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
OSError: [Errno 95] Operation not supported

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 69, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 32, in _link
    raise OSError(
OSError: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 124, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 77, in _try_links
    raise OSError(
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2022-01-24 16:52:05,635 DEBUG: Removing '/fsx/test/main/models/.V7EcLywV6a78oWQAv9fYxa.tmp'
2022-01-24 16:52:05,636 DEBUG: Uploading '/fsx/test/main/.dvc/cache/.FeWKPhhRkQ8TQt3zZ9NyDG.tmp' to '/fsx/test/main/models/.V7EcLywV6a78oWQAv9fYxa.tmp'
2022-01-24 16:52:05,643 DEBUG: Removing '/fsx/test/main/models/.V7EcLywV6a78oWQAv9fYxa.tmp'
2022-01-24 16:52:05,644 DEBUG: Removing '/fsx/test/main/.dvc/cache/.FeWKPhhRkQ8TQt3zZ9NyDG.tmp'
2022-01-24 16:52:05,645 DEBUG: Removing '/fsx/test/main/models/b.txt'
2022-01-24 16:52:05,647 DEBUG: Uploading '/fsx/test/main/.dvc/cache/d8/e8fca2dc0f896fd7cb4cb0031ba249' to '/fsx/test/main/models/b.txt'
2022-01-24 16:52:05,654 DEBUG: state save (144115339624540020, 1643061125000000000, 5) d8e8fca2dc0f896fd7cb4cb0031ba249
2022-01-24 16:52:05,673 DEBUG: state save (144115339624540020, 1643061125000000000, 5) d8e8fca2dc0f896fd7cb4cb0031ba249
2022-01-24 16:52:05,679 DEBUG: stage: 'cp' was reproduced
2022-01-24 16:52:05,706 DEBUG: Staging files: {'dvc.yaml', 'src', 'data/a.txt'}

To track the changes with git, run:

    git add dvc.yaml src data/a.txt

To enable auto staging, run:

	dvc config core.autostage true
2022-01-24 16:52:06,189 DEBUG: Commit to new experiment branch 'refs/exps/1f/ab656a477d19c52d1d99ce1e151191afb74cd9/exp-1c9fb'
2022-01-24 16:52:06,528 DEBUG: Collected experiment '1e1c03b'.
2022-01-24 16:52:06,563 ERROR: unexpected error - invalid data in index - invalid entry
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/command/experiments/run.py", line 32, in run
    results = self.repo.experiments.run(
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 812, in run
    return run(self.repo, *args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/run.py", line 32, in run
    return repo.experiments.reproduce_one(
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 433, in reproduce_one
    results = self._reproduce_revs(
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 51, in wrapper
    return f(exp, *args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 636, in _reproduce_revs
    exec_results.update(self._executors_repro(manager, **kwargs))
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 62, in wrapper
    ret = f(exp, *args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 667, in _executors_repro
    return manager.exec_queue(**kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager.py", line 350, in exec_queue
    self.cleanup_executor(exec_name, executor)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager.py", line 257, in cleanup_executor
    executor.cleanup()
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/executor/local.py", line 165, in cleanup
    self.scm.set_ref(EXEC_APPLY, checkpoint)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/contextlib.py", line 525, in __exit__
    raise exc_details[1]
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/contextlib.py", line 510, in __exit__
    if cb(*exc_details):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/scmrepo/git/__init__.py", line 380, in detach_head
    self.reset()
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/scmrepo/git/__init__.py", line 253, in _backend_func
    return func(*args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/scmrepo/git/backend/pygit2.py", line 484, in reset
    self.repo.index.read(False)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/pygit2/repository.py", line 646, in index
    check_error(err, io=True)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/pygit2/errors.py", line 65, in check_error
    raise GitError(message)
_pygit2.GitError: invalid data in index - invalid entry
------------------------------------------------------------
2022-01-24 16:52:08,665 DEBUG: Adding '/fsx/test/main/.dvc/config.local' to gitignore file.
2022-01-24 16:52:08,673 DEBUG: Adding '/fsx/test/main/.dvc/tmp' to gitignore file.
2022-01-24 16:52:08,673 DEBUG: Adding '/fsx/test/main/.dvc/cache' to gitignore file.
2022-01-24 16:52:08,678 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>: [Errno 95] Operation not supported
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/command/experiments/run.py", line 32, in run
    results = self.repo.experiments.run(
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 812, in run
    return run(self.repo, *args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/run.py", line 32, in run
    return repo.experiments.reproduce_one(
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 433, in reproduce_one
    results = self._reproduce_revs(
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 51, in wrapper
    return f(exp, *args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 636, in _reproduce_revs
    exec_results.update(self._executors_repro(manager, **kwargs))
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 62, in wrapper
    ret = f(exp, *args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 667, in _executors_repro
    return manager.exec_queue(**kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager.py", line 350, in exec_queue
    self.cleanup_executor(exec_name, executor)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/executor/manager.py", line 257, in cleanup_executor
    executor.cleanup()
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/repo/experiments/executor/local.py", line 165, in cleanup
    self.scm.set_ref(EXEC_APPLY, checkpoint)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/contextlib.py", line 525, in __exit__
    raise exc_details[1]
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/contextlib.py", line 510, in __exit__
    if cb(*exc_details):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/scmrepo/git/__init__.py", line 380, in detach_head
    self.reset()
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/scmrepo/git/__init__.py", line 253, in _backend_func
    return func(*args, **kwargs)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/scmrepo/git/backend/pygit2.py", line 484, in reset
    self.repo.index.read(False)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/pygit2/repository.py", line 646, in index
    check_error(err, io=True)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/pygit2/errors.py", line 65, in check_error
    raise GitError(message)
_pygit2.GitError: invalid data in index - invalid entry

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 28, in _link
    func(from_path, to_path)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/local.py", line 148, in reflink
    System.reflink(from_info, to_info)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/system.py", line 112, in reflink
    System._reflink_linux(source, link_name)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/system.py", line 96, in _reflink_linux
    fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
OSError: [Errno 95] Operation not supported

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 69, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 32, in _link
    raise OSError(
OSError: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 124, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/home/asarroff/miniconda3/envs/tensorflow-stable/lib/python3.8/site-packages/dvc/fs/utils.py", line 77, in _try_links
    raise OSError(
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2022-01-24 16:52:08,678 DEBUG: Removing '/fsx/test/.jw3rPE3S2u4AbGdwDSuHS7.tmp'
2022-01-24 16:52:08,680 DEBUG: Removing '/fsx/test/.jw3rPE3S2u4AbGdwDSuHS7.tmp'
2022-01-24 16:52:08,680 DEBUG: Removing '/fsx/test/.jw3rPE3S2u4AbGdwDSuHS7.tmp'
2022-01-24 16:52:08,681 DEBUG: Removing '/fsx/test/main/.dvc/cache/.4BUAToT43jGip9MS4pSMZ3.tmp'
2022-01-24 16:52:08,729 DEBUG: Version info for developers:
DVC version: 2.9.3 (pip)
---------------------------------
Platform: Python 3.8.8 on Linux-4.15.0-1065-aws-x86_64-with-glibc2.10
Supports:
	webhdfs (fsspec = 2021.10.1),
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	s3 (s3fs = 2021.8.1, boto3 = 1.17.106)
Cache types: hardlink, symlink
Cache directory: lustre on 172.16.38.30@tcp:/skl3jbmv
Caches: local
Remotes: None
Workspace directory: lustre on 172.16.38.30@tcp:/skl3jbmv
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-01-24 16:52:08,733 DEBUG: Analytics is disabled.
```
@pmrowla this issue is no longer "awaiting response" but I cannot change the label.
Just wondering @daavoo if this issue is on any roadmap for resolution.  
@woodshop unfortunately we have not been able to get to this issue yet, and it's not currently planned. There's a few submodule related `dvc exp` issues that are open, but given that it's not a very common setup (at least based on user reports we've received so far) we have had to prioritize other work over addressing the submodule problems.
I would actually like to see this working as well for the --temp, --queue case.
I have had the same issue, but it happens even if `dvc exp run` is run from top directory of the project, and when submodule changes are committed (`git status` returns all clear).

What is important is that `dvc exp run` not only fails with _cryptic_ error message, but it **makes a mess** out of repository:

```console
$ git status
On branch jn/dvc.

nothing to commit, working tree clean
$ dvc exp run
[...]
ERROR: unexpected error - invalid data in index - invalid entry

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
$ git status
On branch jn/dvc

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        new file:   .git/COMMIT_EDITMSG
        new file:   .git/FETCH_HEAD
        new file:   .git/HEAD
[... lots and lots of files, including all submodule files ...]

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   .git/HEAD
        modified:   .git/index
        modified:   .git/logs/HEAD
        deleted:    .git/refs/exps/exec/EXEC_BASELINE
        deleted:    .git/refs/exps/exec/EXEC_MERGE
```
To get rid of this I need to do `git reset --hard HEAD` in repository (where I get lots of "error: invalid path" warnings) and in submodule.

DVC should at least **detect** that it cannot run `dvc exp run`, instead of messing up the state of Git repository.

Output of `dvc doctor`:
```console
$ dvc doctor
DVC version: 3.25.0 (pip)
-------------------------
Platform: Python 3.11.4 on Linux-6.4.0-2-amd64-x86_64-with-glibc2.37
Subprojects:
        dvc_data = 2.18.1
        dvc_objects = 1.0.1
        dvc_render = 0.6.0
        dvc_task = 0.3.0
        scmrepo = 1.3.1
Supports:
        http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.9.2, boto3 = 1.28.17)
```
Hey team -- I'm running into this same error message:

> ERROR: unexpected error - invalid data in index - invalid entry 

Also similarly, ` dvc repro -R` seems to work but `dvc exp run -R src/pipelines/` does not.

```
$ dvc doctor
DVC version: 3.33.4 (pip)
-------------------------
Platform: Python 3.10.13 on macOS-12.6-arm64-arm-64bit
Subprojects:
        dvc_data = 2.24.0
        dvc_objects = 2.0.1
        dvc_render = 1.0.0
        dvc_task = 0.3.0
        scmrepo = 1.6.0
Supports:
        http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.12.2, boto3 = 1.33.13)
Config:
        Global: /Users/dflatow/Library/Application Support/dvc
        System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/ceb4be7a51a3752fe6157394a646b490
```

Not sure if different or related issue. 
Also unsure if this is related but some strange git things seem to be going on - DVC has added a few files (metric files) outside of my repo:

```
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        new file:   ../../../../var/folders/2h/2_3k3c317ts84_86kzzcpqjh0000gn/T/tmprlj4wq1v/noise/evaluate/metrics/metrics.json
        new file:   ../../../../var/folders/2h/2_3k3c317ts84_86kzzcpqjh0000gn/T/tmptg4lvlwb/noise/evaluate/metrics/metrics.json
        new file:   ../../../../var/folders/2h/2_3k3c317ts84_86kzzcpqjh0000gn/T/tmpvjg0amiu/noise/evaluate/metrics/metrics.json
        new file:   ../../../../var/folders/2h/2_3k3c317ts84_86kzzcpqjh0000gn/T/tmpvjyepa6l/noise/evaluate/metrics/metrics.json

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    ../../../../var/folders/2h/2_3k3c317ts84_86kzzcpqjh0000gn/T/tmprlj4wq1v/noise/evaluate/metrics/metrics.json
        deleted:    ../../../../var/folders/2h/2_3k3c317ts84_86kzzcpqjh0000gn/T/tmptg4lvlwb/noise/evaluate/metrics/metrics.json
        deleted:    ../../../../var/folders/2h/2_3k3c317ts84_86kzzcpqjh0000gn/T/tmpvjg0amiu/noise/evaluate/metrics/metrics.json
        deleted:    ../../../../var/folders/2h/2_3k3c317ts84_86kzzcpqjh0000gn/T/tmpvjyepa6l/noise/evaluate/metrics/metrics.json
```
hmmm seems the errant metrics files were created but not properly cleaned up in a tmp directory I was using for some testing.
Ok so I think the issue here was due to some sort of git issue caused by running DVC Live in a temporary folder via python library with save_exp=True.

@hemker, could you please share the versions of the libraries installed?

```console
pip show dulwich pygit2 gitpython # or,
uv pip show dulwich pygit2 gitpython
```
Sure, it's:
```console
Name: dulwich
Version: 0.23.2
Summary: Python Git Library
Home-page:
Author:
Author-email: Jelmer Vernooij <jelmer@jelmer.uk>
License:
Location: /home/hemker/dev/submodule-test/env/lib/python3.12/site-packages
Requires: urllib3
Required-by: dvc, dvc-studio-client, scmrepo
---
Name: pygit2
Version: 1.18.1
Summary: Python bindings for libgit2.
Home-page: https://github.com/libgit2/pygit2
Author:
Author-email:
License: GPLv2 with linking exception
Location: /home/hemker/dev/submodule-test/env/lib/python3.12/site-packages
Requires: cffi
Required-by: scmrepo
---
Name: GitPython
Version: 3.1.45
Summary: GitPython is a Python library used to interact with Git repositories
Home-page: https://github.com/gitpython-developers/GitPython
Author: Sebastian Thiel, Michael Trier
Author-email: byronimo@gmail.com, mtrier@gmail.com
License: BSD-3-Clause
Location: /home/hemker/dev/submodule-test/env/lib/python3.12/site-packages
Requires: gitdb
Required-by: scmrepo
```
There seems to be a couple of different issues here:

1. We are adding everything from the root due to what I think is a bug on expanding paths from submodules. This is corrupting index.
   https://github.com/iterative/scmrepo/blob/27a0f2001fd0d6b0ea5307011fb10fa65aeb72b0/src/scmrepo/git/backend/dulwich/__init__.py#L353-L366

2. dulwich and pygit2 `reset()` implementation seems to remove everything from submodules. `git` seems to preserve it. Even if 1) was fixed, you'd get an error in the above reproducer saying `external/run.py` does not exist.


2) is where our support for submodules is unclear. Until now, it's been mostly best-effort and happens to work by accident than design.





 



@hemker, I have a patch in https://github.com/iterative/scmrepo/pull/431. 

I'd really appreciate if you could try it out and let me know how well it works for you.
Another problematic issue here is that you have added the whole `git` submodule as a dependency.

You would not run into 1st issue if you added some paths inside submodule. But you'd run into 2nd issue as I mentioned earlier.
Exactly, initially I was facing an empty submodule as described in 2). This lead me to investigate the issue and remove as much potential overhead as possible to nail it down a bit.

Concerning adding the submodule as a dependency - in the more complex original setup I use a whole deep-learning repository. As marking things as dependencies causes them to be copied to the experiment's location, it is quite handy to have all transitive file imports available there by specifying the submodule as dep only.

Will check the provided PR now!
@skshetry The provided PR fixes my MWE and also the original pipeline! Thank you very much for the fast reply!



Patch:
diff --git a/dvc/repo/experiments/executor/base.py b/dvc/repo/experiments/executor/base.py
index 6ac5315327..76514e46f7 100644
--- a/dvc/repo/experiments/executor/base.py
+++ b/dvc/repo/experiments/executor/base.py
@@ -298,7 +298,9 @@ def save(
                 stages = dvc.commit([], recursive=recursive, force=True, relink=False)
             exp_hash = cls.hash_exp(stages)
             if include_untracked:
-                dvc.scm.add(include_untracked, force=True)  # type: ignore[call-arg]
+                from dvc.scm import add_no_submodules
+
+                add_no_submodules(dvc.scm, include_untracked, force=True)  # type: ignore[call-arg]
 
             with cls.auto_push(dvc):
                 cls.commit(
@@ -513,7 +515,9 @@ def reproduce(
             stages = dvc.reproduce(*args, **kwargs)
             if paths := cls._get_top_level_paths(dvc):
                 logger.debug("Staging top-level files: %s", paths)
-                dvc.scm_context.add(paths)
+                from dvc.scm import add_no_submodules
+
+                add_no_submodules(dvc.scm, paths)
 
             exp_hash = cls.hash_exp(stages)
             if not repro_dry:
diff --git a/dvc/repo/scm_context.py b/dvc/repo/scm_context.py
index bea823de20..c3979cec49 100644
--- a/dvc/repo/scm_context.py
+++ b/dvc/repo/scm_context.py
@@ -41,8 +41,10 @@ def _make_git_add_cmd(paths: Union[str, Iterable[str]]) -> str:
     def add(self, paths: Union[str, Iterable[str]]) -> None:
         from scmrepo.exceptions import UnsupportedIndexFormat
 
+        from dvc.scm import add_no_submodules
+
         try:
-            return self.scm.add(paths)
+            add_no_submodules(self.scm, paths)
         except UnsupportedIndexFormat:
             link = "https://github.com/iterative/dvc/issues/610"
             add_cmd = self._make_git_add_cmd([relpath(path) for path in paths])
diff --git a/dvc/scm.py b/dvc/scm.py
index 1e6de3e72b..f99209b86d 100644
--- a/dvc/scm.py
+++ b/dvc/scm.py
@@ -1,17 +1,18 @@
 """Manages source control systems (e.g. Git)."""
 
 import os
-from collections.abc import Iterator, Mapping
+from collections.abc import Iterable, Iterator, Mapping
 from contextlib import contextmanager
 from functools import partial
 from typing import TYPE_CHECKING, Literal, Optional, Union, overload
 
 from funcy import group_by
-from scmrepo.base import Base  # noqa: F401
+from scmrepo.base import Base  # noqa: TC002
 from scmrepo.git import Git
 from scmrepo.noscm import NoSCM
 
 from dvc.exceptions import DvcException
+from dvc.log import logger
 from dvc.progress import Tqdm
 
 if TYPE_CHECKING:
@@ -19,6 +20,8 @@
 
     from dvc.fs import FileSystem
 
+logger = logger.getChild(__name__)
+
 
 class SCMError(DvcException):
     """Base class for source control management errors."""
@@ -283,3 +286,32 @@ def lfs_prefetch(fs: "FileSystem", paths: list[str]):
             include=[(path if path.startswith("/") else f"/{path}") for path in paths],
             progress=pbar.update_git,
         )
+
+
+def add_no_submodules(
+    scm: "Base",
+    paths: Union[str, Iterable[str]],
+    **kwargs,
+) -> None:
+    """Stage paths to Git, excluding those inside submodules."""
+
+    if isinstance(paths, str):
+        paths = [paths]
+
+    submodule_roots = {os.path.join(scm.root_dir, sub) for sub in scm.list_submodules()}
+
+    repo_paths: list[str] = []
+    skipped_paths: list[str] = []
+
+    for p in paths:
+        abs_path = os.path.abspath(p)
+        if abs_path in submodule_roots or abs_path.startswith(tuple(submodule_roots)):
+            skipped_paths.append(p)
+        else:
+            repo_paths.append(p)
+
+    if skipped_paths:
+        msg = "Skipping staging for path(s) inside submodules: %s"
+        logger.debug(msg, ", ".join(skipped_paths))
+
+    scm.add(repo_paths, **kwargs)
diff --git a/pyproject.toml b/pyproject.toml
index 86703d5154..013c89ed2b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -67,7 +67,7 @@ dependencies = [
     "requests>=2.22",
     "rich>=12",
     "ruamel.yaml>=0.17.11",
-    "scmrepo>=3.3.8,<4",
+    "scmrepo>=3.5.2,<4",
     "shortuuid>=0.5",
     "shtab<2,>=1.3.4",
     "tabulate>=0.8.7",


Test patch:
diff --git a/tests/func/experiments/test_experiments.py b/tests/func/experiments/test_experiments.py
index 69ab6ba1be..2782319563 100644
--- a/tests/func/experiments/test_experiments.py
+++ b/tests/func/experiments/test_experiments.py
@@ -793,3 +793,18 @@ def test_custom_commit_message(tmp_dir, scm, dvc, tmp):
         )
     )
     assert scm.resolve_commit(exp).message == "custom commit message"
+
+
+@pytest.mark.parametrize("dep", ["submodule", "submodule/file"])
+def test_experiments_run_with_submodule_dependencies(dvc, scm, make_tmp_dir, dep):
+    external_repo = make_tmp_dir("external_repo", scm=True)
+    external_repo.scm_gen("file", "content", commit="add file")
+
+    submodules = scm.pygit2.repo.submodules
+    submodules.add(os.fspath(external_repo), "submodule")
+    submodules.update(init=True)
+    scm.add_commit([".gitmodules"], message="add submodule")
+
+    dvc.stage.add(cmd="echo foo", deps=[dep], name="foo")
+
+    assert dvc.experiments.run()
diff --git a/tests/unit/repo/test_scm_context.py b/tests/unit/repo/test_scm_context.py
index 21978c0a15..d9ff258e35 100644
--- a/tests/unit/repo/test_scm_context.py
+++ b/tests/unit/repo/test_scm_context.py
@@ -39,7 +39,7 @@ def test_scm_track_changed_files(scm_context):
 
     scm_context.track_file("foo")
     scm_context.track_changed_files()
-    scm_context.scm.add.assert_called_once_with({"foo"})
+    scm_context.scm.add.assert_called_once_with(["foo"])
 
 
 def test_ignore(scm_context):
@@ -73,7 +73,7 @@ def test_scm_context_autostage_changed_files(scm_context):
 
     assert not scm_context.files_to_track
     assert not scm_context.ignored_paths
-    scm_context.scm.add.assert_called_once_with({"foo"})
+    scm_context.scm.add.assert_called_once_with(["foo"])
 
 
 def test_scm_context_clears_ignores_on_error(scm_context):
@@ -141,4 +141,4 @@ def test_method(repo, *args, **kwargs):
     method = mocker.MagicMock(wraps=test_method)
     decorator(method, autostage=True)(repo, "arg", kw=1)
     method.assert_called_once_with(repo, "arg", kw=1)
-    scm_context.scm.add.assert_called_once_with({"foo"})
+    scm_context.scm.add.assert_called_once_with(["foo"])