Repository: iterative/dvc

Issue:
pull: pulling from private github dataregistry repo with git credential helper not working anymore
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description
With latest 2.44.0 version 'dvc pull' no longer works when pulling assets from private GitHub repo. Using git credential helper for authentication.
<!--
A clear and concise description of what the bug is.
-->

### Reproduce
```console
$ dvc pull -v
2023-02-13 15:02:05,305 DEBUG: v2.44.0 (deb), CPython 3.10.8 on Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.31
2023-02-13 15:02:05,305 DEBUG: command: pull -v
2023-02-13 15:02:06,332 DEBUG: Creating external repo https://github.com/company/repo.git@4673682286885bda923460f92cc829602
2023-02-13 15:02:06,332 DEBUG: erepo: git clone 'https://github.com/company/repo.git' to a temporary dir
2023-02-13 15:02:11,410 ERROR: unexpected error - Git failed to fetch ref from 'https://github.com/company/repo.git': remote authentication required but no callback set
Traceback (most recent call last):
  File "funcy/flow.py", line 112, in reraise
  File "scmrepo/git/backend/pygit2.py", line 508, in fetch_refspecs
  File "pygit2/remote.py", line 146, in fetch
  File "pygit2/callbacks.py", line 98, in check_error
  File "pygit2/errors.py", line 65, in check_error
_pygit2.GitError: remote authentication required but no callback set

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc/cli/__init__.py", line 205, in main
  File "dvc/cli/command.py", line 26, in do_run
  File "dvc/commands/data_sync.py", line 31, in run
  File "dvc/repo/__init__.py", line 57, in wrapper
  File "dvc/repo/pull.py", line 34, in pull
  File "dvc/repo/__init__.py", line 57, in wrapper
  File "dvc/repo/fetch.py", line 86, in fetch
  File "dvc/repo/fetch.py", line 142, in _fetch
  File "dvc/repo/__init__.py", line 449, in used_objs
  File "dvc/repo/index.py", line 385, in used_objs
  File "dvc/stage/__init__.py", line 722, in get_used_objs
  File "dvc/output.py", line 1074, in get_used_objs
  File "dvc/output.py", line 1133, in get_used_external
  File "dvc/dependency/repo.py", line 102, in get_used_objs
  File "dvc/dependency/repo.py", line 116, in _get_used_and_obj
  File "contextlib.py", line 135, in __enter__
  File "dvc/external_repo.py", line 45, in external_repo
  File "dvc/external_repo.py", line 173, in _cached_clone
  File "funcy/decorators.py", line 45, in wrapper
  File "funcy/flow.py", line 274, in wrap_with
  File "funcy/decorators.py", line 66, in __call__
  File "dvc/external_repo.py", line 241, in _clone_default_branch
  File "dvc/scm.py", line 182, in clone
  File "dvc/repo/experiments/utils.py", line 288, in fetch_all_exps
  File "scmrepo/git/__init__.py", line 279, in _backend_func
  File "scmrepo/git/backend/pygit2.py", line 504, in fetch_refspecs
  File "contextlib.py", line 153, in __exit__
  File "funcy/flow.py", line 116, in reraise
  File "<string>", line 3, in raise_from
scmrepo.exceptions.SCMError: Git failed to fetch ref from 'https://github.com/company/repo.git'
```
<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

### Expected
It worked fine with 2.43.4, suspecting https://github.com/iterative/dvc/pull/8907
<!--
A clear and concise description of what you expect to happen.
-->

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor
DVC version: 2.44.0 (deb)
-------------------------
Platform: Python 3.10.8 on Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.31
Subprojects:

Supports:
        azure (adlfs = 2023.1.0, knack = 0.10.1, azure-identity = 1.12.0),
        gdrive (pydrive2 = 1.15.0),
        gs (gcsfs = 2023.1.0),
        hdfs (fsspec = 2023.1.0, pyarrow = 11.0.0),
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = 2023.1.0, boto3 = 1.24.59),
        ssh (sshfs = 2023.1.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.1.0)
Cache types: hardlink, symlink
Cache directory: xfs on /dev/mapper/rootvg-home_lv
Caches: local
Remotes: azure
Workspace directory: xfs on /dev/mapper/rootvg-home_lv
Repo: dvc, git
```

**Additional Information (if any):**

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->



Discussion:
Thanks, looks like #8907 is to blame as you said. `libgit2` does not support git-credential helper. cc @karajan1001 
Thanks a lot for reporting this. I was having a hard time figuring out how to solve it!

Resolving patch:
diff --git a/dvc/external_repo.py b/dvc/external_repo.py
--- a/dvc/external_repo.py
+++ b/dvc/external_repo.py
@@ -253,7 +253,7 @@ def _pull(git: "Git", unshallow: bool = False):
 
     git.fetch(unshallow=unshallow)
     _merge_upstream(git)
-    fetch_all_exps(git, "origin")
+    fetch_all_exps(git, "origin", backends=["dulwich"])
 
 
 def _merge_upstream(git: "Git"):
diff --git a/dvc/repo/experiments/utils.py b/dvc/repo/experiments/utils.py
--- a/dvc/repo/experiments/utils.py
+++ b/dvc/repo/experiments/utils.py
@@ -5,6 +5,7 @@
 from functools import wraps
 from typing import (
     TYPE_CHECKING,
+    Any,
     Callable,
     Dict,
     Generator,
@@ -279,17 +280,15 @@ def check_ref_format(scm: "Git", ref: ExpRefInfo):
         )
 
 
-def fetch_all_exps(scm: "Git", url: str, progress: Optional[Callable] = None):
+def fetch_all_exps(
+    scm: "Git", url: str, progress: Optional[Callable] = None, **kwargs: Any
+):
     refspecs = [
         f"{ref}:{ref}"
-        for ref in iter_remote_refs(scm, url, base=EXPS_NAMESPACE)
+        for ref in iter_remote_refs(scm, url, base=EXPS_NAMESPACE, **kwargs)
         if not (ref.startswith(EXEC_NAMESPACE) or ref in STASHES)
     ]
-    scm.fetch_refspecs(
-        url,
-        refspecs,
-        progress=progress,
-    )
+    scm.fetch_refspecs(url, refspecs, progress=progress, **kwargs)
 
 
 def get_random_exp_name(scm, baseline_rev):
diff --git a/dvc/scm.py b/dvc/scm.py
--- a/dvc/scm.py
+++ b/dvc/scm.py
@@ -179,7 +179,7 @@ def clone(url: str, to_path: str, **kwargs):
         try:
             git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
             if "shallow_branch" not in kwargs:
-                fetch_all_exps(git, url, progress=pbar.update_git)
+                fetch_all_exps(git, url, progress=pbar.update_git, backends=["dulwich"])
             return git
         except InternalCloneError as exc:
             raise CloneError("SCM error") from exc


Test patch: