Allow dataset streaming from private a private source when loading a dataset with a dataset loading script
Opened by bruno-hays (open)

### Feature request

Add arguments to the function _get_authentication_headers_for_url_ like custom_endpoint and custom_token in order to add flexibility when downloading files from a private source.

It should also be possible to provide these arguments from the dataset loading script, maybe giving them to the dl_manager

### Motivation

It is possible to share a dataset hosted on another platform by writing a dataset loading script. It works perfectly for publicly available resources.

For resources that require authentication, you can provide a [download_custom](https://huggingface.co/docs/datasets/package_reference/builder_classes#datasets.DownloadManager) method to the download_manager.

Unfortunately, this function doesn't work with **dataset streaming**.

A solution so as to allow dataset streaming from private sources would be a more flexible _get_authentication_headers_for_url_ function.

### Your contribution

Would you be interested in this improvement ? 
If so I could provide a PR. I've got something working locally, but it's not very clean, I'd need some guidance regarding integration.

Comments:
Hi ! What kind of private source ? We're exploring adding support for cloud storage and URIs like s3://, gs:// etc. with authentication in the download manager

Hello! It's a google cloud storage, so gs://, but I'm using it with https.
Being able to provide a file system like [here](https://huggingface.co/docs/datasets/main/filesystems#load-serialized-datasets) would be even more practical indeed.
I've found a quite complicated workaround which consists of monkey patching all of the functions in streaming_download_manager.py to use my own _get_authentication_headers_for_url_ . 

A support for this use case would be greatly appreciated!

For reference my _get_authentication_headers_for_url_ looks like this:
```
import os
from typing import Optional, Union

from datasets import config
from huggingface_hub import HfFolder
from gcsfs.credentials import GoogleCredentials

DEFAULT_PROJECT = os.environ.get("GCSFS_DEFAULT_PROJECT", "")
access = "full_control"
gcs_token = os.environ.get("GCS_TOKEN")


def get_authentication_headers_for_url(url: str, use_auth_token: Optional[Union[str, bool]] = None) -> dict:
    """Handle the HF authentication"""
    headers = {}
    if url.startswith(config.HF_ENDPOINT):
        if use_auth_token is False:
            token = None
        elif isinstance(use_auth_token, str):
            token = use_auth_token
        else:
            token = HfFolder.get_token()
    elif url.startswith("https://storage.googleapis.com"):
        credentials = GoogleCredentials(DEFAULT_PROJECT, access, gcs_token)
        credentials.maybe_refresh()
        token = credentials.credentials.token
    else:
        token = None
    if token:
        headers["authorization"] = f"Bearer {token}"
    return headers
```

I would be a big fan of this feature! @Hubert-Bonisseur if this doesn't become a supported feature, would you mind sharing your code? Thanks!

> I would be a big fan of this feature! @Hubert-Bonisseur if this doesn't become a supported feature, would you mind sharing your code? Thanks!

I published it here:
https://github.com/Hubert-Bonisseur/private-dataset-hub

I modified the names of a lot of functions for privacy and I don't have time to test it again so you may get import errors, but you have the code. The custom_load_dataset is the function you are interested in I think.

It relies a lot on patching, if you find a better way to do this, I'd be interested.

Given the amount of patching it does, this is likely to break at one point. I'd encourage you to wait for a proper support in `datasets` directly if you can wait.