Support loading from a DVC remote repository
Opened by bilelomrani1 (closed)

### Feature request

Adding support for loading a file from a DVC repository, tracked remotely on a SCM.

### Motivation

DVC is a popular version control system to version and manage datasets. The files are stored on a remote object storage platform, but they are tracked using Git. Integration with DVC is possible through the `DVCFileSystem`.

I have a Gitlab repository where multiple files are tracked using DVC and stored in a GCP bucket. I would like to be able to load these files using `datasets` directly using an URL. My goal is to write a generic code that abstracts the storage layer, such that my users will only have to pass in an `fsspec`-compliant URL and the corresponding files will be loaded.

### Your contribution

I managed to instantiate a `DVCFileSystem` pointing to a Gitlab repo from a `fsspec` chained URL in [this pull request](https://github.com/iterative/dvc/pull/9903) to DVC. 

```python
from fsspec.core import url_to_fs

fs, _ = url_to_fs("dvc::https://gitlab.com/repository/group/my-repo")
```

From now I'm not sure how to continue, it seems that `datasets` expects the URL to be fully qualified like so: `dvc::https://gitlab.com/repository/group/my-repo/my-folder/my-file.json` but this fails because `DVCFileSystem` expects the URL to point to the root of an SCM repo. Is there a way to make this work with `datasets`?

Comments:
(cross-posting from the linked DVC issue)

I think this should already work out of the box with the current `datasets` and `dvc.api` releases by passing the correct `storage_options` into the datasets calls. `storage_options` is essentially just the kwargs dict that gets passed to the fsspec fs constructor.

The main thing to note here is that the fsspec DVCFileSystem URL should be `dvc://folder/file.json` (i.e. this should be the DVCFileSystem path that is relative to the DVC repo root). You cannot use a URL like `https://gitlab.com/user/repo/folder/file.json`.

I think something like this should work for you (in a venv where both DVC and datasets are installed):
```python
import datasets

# load a dataset from Git/DVC repository where Git repo is located at https://gitlab.com/user/repo.git
# and path to dataset (relative to git/dvc repo root) is 'folder/file.json'
datasets.load_from_disk(
    "dvc://folder/file.json",
    storage_options={"url": "https://gitlab.com/user/repo.git"},
)
```

basically the `dvc://` is what tells fsspec to create a `DVCFileSystem` and it will construct it like
```python
fs = DVCFileSystem(**storage_options)
```

Then the subsequent calls use the rest of the `dvc://...` URL like 
```python
fs.exists("folder/file.json")
```

Hi @pmrowla Thank you for your help, that's very helpful, I was indeed using `fsspec` incorrectly here. There is still an issue with `datasets`:

```python
import datasets
dataset = datasets.load_dataset("json", data_files="dvc://folder/file.jsonl", storage_options={"url": "https://gitlab.com/repo/folder/"})
```

results in the following exception:

```
Traceback (most recent call last):                                                                                        
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/scmrepo/fs.py", line 217, in info
    ret = self.trie.info(key)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/scmrepo/git/objects.py", line 141, in info
    obj = self.trie[key]
          ~~~~~~~~~^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/pygtrie.py", line 937, in __getitem__
    node, _ = self._get_node(key_or_slice)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/pygtrie.py", line 630, in _get_node
    raise KeyError(key)
KeyError: ('dvc:', 'datasets', 'spider', 'train.jsonl')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/load.py", line 1815, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/load.py", line 1430, in dataset_module_factory
    ).get_module()
      ^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/load.py", line 958, in get_module
    data_files = DataFilesDict.from_patterns(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/data_files.py", line 674, in from_patterns
    DataFilesList.from_patterns(
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/data_files.py", line 589, in from_patterns
    origin_metadata = _get_origin_metadata(data_files, download_config=download_config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/data_files.py", line 504, in _get_origin_metadata
    return thread_map(
           ^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/data_files.py", line 491, in _get_single_origin_metadata
    info = fs.info(data_file)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/dvc/fs/dvc.py", line 357, in info
    return self._info(key, path, ignore_subrepos=ignore_subrepos)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/dvc/fs/dvc.py", line 377, in _info
    fs_info = fs.info(fs_path)
              ^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 501, in info
    return self.fs.info(path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/scmrepo/fs.py", line 221, in info
    raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), path)
FileNotFoundError: [Errno 2] No such file or directory: '/dvc:/folder/file.jsonl'
```

Somehow the URL gets turned into `/dvc:/folder/file.jsonl` inside `datasets`. Otherwise I can confirm that using `fsspec` properly with DVC works as expected.


For the record, there was a `dvc.api.DVCFileSystem` bug which is fixed in DVC `main` and will be available in the next DVC release.

To use DVC with `datasets` you just need to pass the Git/DVC repo `url` in `storage_options` as discussed above.

(note that this requires having both `datasets` and `dvc` installed in your python environment)
```python
>>> from datasets import load_dataset
>>> load_dataset(
...   "json",
...   data_files="dvc://eval/metrics.json",
...   storage_options={"url": "https://github.com/iterative/example-get-started.git"},
... )
DatasetDict({
    train: Dataset({
        features: ['avg_prec', 'roc_auc'],
        num_rows: 1
    })
})
```

Any additional `DVCFileSystem` args can be passed in the same way, so to get a specific branch/tag/commit from the DVC repo you just need to specify the `rev` in `storage_options` like
```
storage_options={"url": "https://github.com/iterative/example-get-started.git", "rev": "main"}
```

I think this issue can probably be closed now.

Thank you for your help, closing.