Repository: iterative/dvc

Issue:
Using dvc-tracked data in huggingface datasets
From https://github.com/iterative/dvc/pull/9903:

> PS: It unfortunately does not solve my original use case of loading a file using HuggingFace datasets, it seems that `datasets` expects the URL to be fully qualified like so: `dvc::https://gitlab.com/repository/group/my-repo/my-folder/my-file.json` but this fails because `DVCFileSystem` expects the URL to point to the root of an SCM repo. Not sure how to continue from here, according to you is it an issue with DVC or HuggingFace datasets?


Discussion:
I think this should already be resolved

https://github.com/iterative/dvc/issues/9896#issuecomment-1720724269
https://github.com/huggingface/datasets/issues/6203#issuecomment-1720728760
Hi @dberenbaum and @pmrowla, thank you for your help, I was indeed using `fsspec` incorrectly, thank you @pmrowla for your help on the syntax. We managed to narrow down the problem, `datasets` is doing something weird with the URL (see https://github.com/huggingface/datasets/issues/6203#issuecomment-1721065217), but when using pure `fsspec`, I manage to load my file correctly.

Resolving patch:
diff --git a/dvc/fs/dvc.py b/dvc/fs/dvc.py
--- a/dvc/fs/dvc.py
+++ b/dvc/fs/dvc.py
@@ -18,7 +18,7 @@
 
 if TYPE_CHECKING:
     from dvc.repo import Repo
-    from dvc.types import StrPath
+    from dvc.types import DictStrAny, StrPath
 
 logger = logging.getLogger(__name__)
 
@@ -76,7 +76,7 @@ class _DVCFileSystem(AbstractFileSystem):  # pylint:disable=abstract-method
     cachable = False
     root_marker = "/"
 
-    def __init__(
+    def __init__(  # noqa: PLR0913
         self,
         url: Optional[str] = None,
         rev: Optional[str] = None,
@@ -88,7 +88,10 @@ def __init__(
         target_options: Optional[Dict[str, Any]] = None,  # noqa: ARG002
         # pylint:disable-next=unused-argument
         target_protocol: Optional[str] = None,  # noqa: ARG002
-        **repo_kwargs: Any,
+        config: Optional["DictStrAny"] = None,
+        remote: Optional[str] = None,
+        remote_config: Optional["DictStrAny"] = None,
+        **kwargs,
     ) -> None:
         """DVC + git-tracked files fs.
 
@@ -108,6 +111,9 @@ def __init__(
                 By default, it ignores subrepos.
             repo_factory (callable): A function to initialize subrepo with.
                 The default is `Repo`.
+            config (dict): Repo config to be passed into `repo_factory`.
+            remote (str): Remote name to be passed into `repo_factory`.
+            remote_config(dict): Remote config to be passed into `repo_factory`.
 
         Examples:
             - Opening a filesystem from repo in current working directory
@@ -131,7 +137,14 @@ def __init__(
         self._repo_stack = ExitStack()
         if repo is None:
             url = url if url is not None else fo
-            repo = self._make_repo(url=url, rev=rev, subrepos=subrepos, **repo_kwargs)
+            repo = self._make_repo(
+                url=url,
+                rev=rev,
+                subrepos=subrepos,
+                config=config,
+                remote=remote,
+                remote_config=remote_config,
+            )
             assert repo is not None
             # pylint: disable=protected-access
             repo_factory = repo._fs_conf["repo_factory"]
@@ -187,6 +200,7 @@ def _get_key(self, path: "StrPath") -> Key:
         return parts
 
     def _get_key_from_relative(self, path) -> Key:
+        path = self._strip_protocol(path)
         parts = self.path.relparts(path, self.root_marker)
         if parts and parts[0] == os.curdir:
             return parts[1:]


Test patch: