Unable to download dataset using Azure Data Lake Gen 2
Opened by clarissesimoes (closed)

### Describe the bug

When using the DatasetBuilder method with the credentials for the cloud storage Azure Data Lake (adl) Gen2, the following error is showed:
```
Traceback (most recent call last):
  File "download_hf_dataset.py", line 143, in <module>
    main()
  File "download_hf_dataset.py", line 102, in main
    builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/datasets/builder.py", line 671, in download_and_prepare
    fs_token_paths = fsspec.get_fs_token_paths(output_dir, storage_options=storage_options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/core.py", line 639, in get_fs_token_paths
    fs = cls(**options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'account_name'
```

If I don't pass the storage_options argument (leave it as None), it requires the credentials used in ADL Gen 1:
`TypeError: __init__() missing 3 required positional arguments: 'tenant_id', 'client_id', and 'client_secret'`

Thus, it is not possible to download a dataset from the cloud using Azure Data Lake (adl) Gen2.

### Steps to reproduce the bug

Assuming that you have an account on Azure and at Storage Account that can be used for reproduce:

1. Create a dict with the format to connect to Azure Data Lake Gen 2
```
storage_options = {"account_name": ACCOUNT_NAME, "account_key": ACCOUNT_KEY)  # gen 2 filesystem
```
2. Create a dataset builder for any HF hosted dataset
```
builder = load_dataset_builder(dataset_name)
```
3. Try to download the dataset passing the storage_options as an argument
```
save_dir = 'adl://my_save_dir'
builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
```

### Expected behavior

Not seeing the error mentioned above and being able to download the dataset to the provided path on ADL

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1

Comments:
Hi ! From the `adlfs` docs, there are two filesystems you can use:
> To use the Gen1 filesystem:
> - known_implementations[‘adl’] = {‘class’: ‘adlfs.AzureDatalakeFileSystem’}
> 
> To use the Gen2 filesystem:
>  - known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}

If I'm not mistaken you're using the second one - so you should use `abfs://` instead of `adl://`, and also run this at the beginning of your script:
```python
from fsspec.registry import known_implementations
known_implementations['abfs'] = {'class': 'adlfs.AzureDatalakeFileSystem'}
```



Thank you @lhoestq . Great call.
Using the default class from `known_implementations` dict solved my problem
```
known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}
```
I'm closing this issue.

> Thank you @lhoestq . Great call. Using the default class from `known_implementations` dict solved my problem
> 
> ```
> known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}
> ```
> 
> I'm closing this issue.

Hi so here  `Saving serialized datasets

After you have processed your dataset, you can save it to your cloud storage with [Dataset.save_to_disk()](https://huggingface.co/docs/datasets/v2.17.0/en/package_reference/main_classes#datasets.Dataset.save_to_disk):` what is the encoded dataset I have failed to save it 

Uploading failed ? Did you get an error message ?