SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Dataset Viewer issue for asapp/slue

Comment:
OK. We now have an issue because the dataset cannot be streamed, and the dataset viewer relies on it.

Maybe @huggingface/datasets can help:

```
Error code:   StreamingRowsError
Exception:    NotImplementedError
Message:      Extraction protocol for TAR archives like 'https://public-dataset-model-store.awsdev.asapp.com/users/sshon/public/slue/slue-voxpopuli_v0.2_blind.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
Traceback:    Traceback (most recent call last):
               File "/src/services/worker/src/worker/responses/first_rows.py", line 337, in get_first_rows_response
                 rows = get_rows(dataset, config, split, streaming=True, rows_max_number=rows_max_number, hf_token=hf_token)
               File "/src/services/worker/src/worker/utils.py", line 123, in decorator
                 return func(*args, **kwargs)
               File "/src/services/worker/src/worker/responses/first_rows.py", line 65, in get_rows
                 ds = load_dataset(
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1739, in load_dataset
                 return builder_instance.as_streaming_dataset(split=split)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1025, in as_streaming_dataset
                 splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
               File "/tmp/modules-cache/datasets_modules/datasets/asapp--slue/adaa0c78233e1a1df9c2f054e690ec5fc3eaf453bd76b80fe5cbe5728e55d9b1/slue.py", line 189, in _split_generators
                 dl_dir = dl_manager.download_and_extract(_DL_URLS[config_name])
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 944, in download_and_extract
                 return self.extract(self.download(url_or_urls))
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 907, in extract
                 urlpaths = map_nested(self._extract, path_or_paths, map_tuple=True)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 385, in map_nested
                 return function(data_struct)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 912, in _extract
                 protocol = _get_extraction_protocol(urlpath, use_auth_token=self.download_config.use_auth_token)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 390, in _get_extraction_protocol
                 raise NotImplementedError(
             NotImplementedError: Extraction protocol for TAR archives like 'https://public-dataset-model-store.awsdev.asapp.com/users/sshon/public/slue/slue-voxpopuli_v0.2_blind.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```
Issue: Dataset Viewer issue for asapp/slue

Comment:
Thanks @severo, 

Do I have to modify the python script to support streaming so that it can be previewed?
Is there a document somewhere that I can follow?
Issue: Dataset Viewer issue for asapp/slue

Comment:
Hi @fwu-asapp thanks for reporting, and thanks @severo for the investigation.

As explained by @severo, the preview requires that your dataset loading script supports streaming.

There are several options here:
- the easiest would be to replace the source files, archived using ZIP instead TAR: the TAR format does not allow random access while streaming, but only sequential access; the ZIP files support streaming out of the box.
- alternatively, to stream TAR archives you can use `dl_manager.iter_archive`: the only prerequisite is that your "index" files (.tsv) should have been archived before their corresponding audio files, so while iterating the content of the TAR archive, the metadata files appear first. I think this is the case for voxpopuli tar but not for voxceleb.
- if your .tsv files were not archived before their corresponding audio files (I think this is the case for voxceleb), then you should extract the .tsv files and host them separately (you can host them on the same Hugging Face Hub).
  - you can take as example, e.g.: https://huggingface.co/datasets/vivos/blob/main/vivos.py

As an advanced approach, you can handle both streaming and non-streaming cases separately.
- as for example: https://huggingface.co/datasets/librispeech_asr/blob/main/librispeech_asr.py or https://huggingface.co/datasets/google/fleurs/blob/main/fleurs.py

See related discussion:
- https://github.com/huggingface/datasets/issues/4697#issuecomment-1191502492
Issue: Dataset Viewer issue for asapp/slue

Comment:
Thanks @albertvillanova for your clarification. I'll talk to my collaborators to see if we can replace those files. Let me just close this issue for now.
Issue: Dataset Viewer issue for asapp/slue

Comment:
FYI, after replacing the source files with the ZIP ones, the dataset viewer works well. Thanks again to @severo  and @albertvillanova for your help!
Issue: Dataset Viewer issue for Jean-Baptiste/wikiner_fr

Comment:
The script uses `Dataset.load_from_disk`, which as you can expect, doesn't work in streaming mode.

It would probably be more practical to load the dataset locally using `Dataset.load_from_disk` first and then `push_to_hub` to upload it in Parquet on the Hub
Issue: "no-token" is passed to `huggingface_hub` when token is `None`

Comment:
Hi @Wauplin, thanks for raising this potential issue.

The choice of passing `"no-token"` instead of `None` was made in this PR:
- #4536 

According to the PR description, the reason why it is passed is to avoid that `HfApi.dataset_info` uses the local token when no token should be used.
Issue: "no-token" is passed to `huggingface_hub` when token is `None`

Comment:
Hi @albertvillanova , thanks for finding the original issue :+1: 

As of next release of `huggingface_hub`, the `token` argument will be deprecated in favor of the `use_auth_token` argument in `dataset_info` method. This change as been done by @SBrandeis in https://github.com/huggingface/huggingface_hub/pull/928. `use_auth_token` is a bit different and allow the case "don't sent the cached token by default".

If you want to strictly avoid sending the cached token from `datasets`, you can use:
```py
# token=token if token else "no-token", <- will fail because token is not valid

use_auth_token=token if token else False,  # using the new `use_auth_token` parameter
```

And as a note, I am currently updating the "don't send the cached token by default"-rule to "don't send the cached token on public repos by default but use it in private ones" in https://github.com/huggingface/huggingface_hub/pull/1064. This will not change the fact that `use_auth_token=False` doesn't send the token at all.
Issue: "no-token" is passed to `huggingface_hub` when token is `None`

Comment:
What is current strategy in term of updating `huggingface_hub` version in `datasets` ? I don't want to break stuff in the next release so let's find a proper solution :)
Issue: "no-token" is passed to `huggingface_hub` when token is `None`

Comment:
As soon as `token` is deprecated and hfh has a new release, we'll update `datasets` to use the new argument instead. Does it sound good to you ?
Issue: "no-token" is passed to `huggingface_hub` when token is `None`

Comment:
Hi @Wauplin, thanks for the warning about the deprecation of `token` in favor of `use_auth_token`.

Indeed, in datasets we use internally `use_auth_token`, which in this case was transformed to `token` to call `HfApi.dataset_info`:
https://github.com/huggingface/datasets/blob/1a9385d7cc8a3241b44015145ef56a230fdadc51/src/datasets/load.py#L747

Therefore, for the new hfh release, the fix will be trivial: we will pass directly `use_auth_token`.

As discussed during our meeting yesterday, due to the fact that at datasets we support multiple hfh versions, I think we should handle passing `token` or `use_auth_token` depending on the hfh version.
Issue: Add `IterableDataset.from_generator` to the API

Comment:
Thanks @hamid-vakilzadeh ! Let us know if you have some questions or if we can help
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
Hi! I think you can use the newly-added `from_generator` method for that:
```python
from datasets import Dataset

def gen():
    for idx in len(torch_dataset):
        yield torch_dataset[idx]  # this has to be a dictionary
    ## or if it's an IterableDataset
    # for ex in torch_dataset:
    #     yield ex

dset = Dataset.from_generator(gen)
```
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
Maybe `Dataset.from_list` can work as well no ?
```python
from datasets import Dataset

dset = Dataset.from_list(torch_dataset)
```
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
> ```python
> from datasets import Dataset
> 
> def gen():
>     for idx in len(torch_dataset):
>         yield torch_dataset[idx]  # this has to be a dictionary
>     ## or if it's an IterableDataset
>     # for ex in torch_dataset:
>     #     yield ex
> 
> dset = Dataset.from_generator(gen)
> ```

I try to use `Dataset.from_generator()` method, and it returns an error:
```bash
AttributeError: type object 'Dataset' has no attribute 'from_generator'
```
And I think it maybe the version of my datasets package is out-of-date, so I update it
```bash
pip install --upgrade datasets
```
But after that, the code still return the above Error.
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
> ```python
> dset = Dataset.from_list(torch_dataset)
> ```

It seems that Dataset also has no `from_list` method 😂
```bash
AttributeError: type object 'Dataset' has no attribute 'from_list'
```
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
> I look through the huggingface dataset docs, and it seems that there is no offical support function to convert `torch.utils.data.Dataset` to huggingface dataset. However, there is a way to convert huggingface dataset to `torch.utils.data.Dataset`, like below:
> 
> ```python
> from datasets import Dataset
> data = [[1, 2],[3, 4]]
> ds = Dataset.from_dict({"data": data})
> ds = ds.with_format("torch")
> ds[0]
> ds[:2]
> ```
> 
> So is there something I miss, or there IS no function to convert `torch.utils.data.Dataset` to huggingface dataset. If so, is there any way to do this convert? Thanks.

My dummy code is like:
```python
import os
import json
from torch.utils import data
import datasets

def gen(torch_dataset):
    for idx in len(torch_dataset):
        yield torch_dataset[idx]  # this has to be a dictionary

class MyDataset(data.Dataset):
    def __init__(self, path):
        self.dict = []
        for line in open(path, 'r', encoding='utf-8'):
            j_dict = json.loads(line)
            self.dict.append(j_dict['context'])
            
    def __getitem__(self, idx):
        return self.dict[idx]

    def __len__(self):
        return len(self.dict)

root_path = os.path.dirname(os.path.abspath(__file__))
path = os.path.join(root_path, 'dataset', 'train.json')
torch_dataset = MyDataset(path)

dit = []
for line in open(path, 'r', encoding='utf-8'):
    j_dict = json.loads(line)
    dit.append(j_dict['context'])
dset1 = datasets.Dataset.from_list(dit)
print(dset1)
dset2 = datasets.Dataset.from_generator(gen)
print(dset2)
```
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
We're releasing `from_generator` and `from_list` today :)
In the meantime you can play with them by installing `datasets` from source
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
> We're releasing `from_generator` and `from_list` today :) In the meantime you can play with them by installing `datasets` from source

Thanks a lot for your work!
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
> > I look through the huggingface dataset docs, and it seems that there is no offical support function to convert `torch.utils.data.Dataset` to huggingface dataset. However, there is a way to convert huggingface dataset to `torch.utils.data.Dataset`, like below:
> > ```python
> > from datasets import Dataset
> > data = [[1, 2],[3, 4]]
> > ds = Dataset.from_dict({"data": data})
> > ds = ds.with_format("torch")
> > ds[0]
> > ds[:2]
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > So is there something I miss, or there IS no function to convert `torch.utils.data.Dataset` to huggingface dataset. If so, is there any way to do this convert? Thanks.
> 
> My dummy code is like:
> 
> ```python
> import os
> import json
> from torch.utils import data
> import datasets
> 
> def gen(torch_dataset):
>     for idx in len(torch_dataset):
>         yield torch_dataset[idx]  # this has to be a dictionary
> 
> class MyDataset(data.Dataset):
>     def __init__(self, path):
>         self.dict = []
>         for line in open(path, 'r', encoding='utf-8'):
>             j_dict = json.loads(line)
>             self.dict.append(j_dict['context'])
>             
>     def __getitem__(self, idx):
>         return self.dict[idx]
> 
>     def __len__(self):
>         return len(self.dict)
> 
> root_path = os.path.dirname(os.path.abspath(__file__))
> path = os.path.join(root_path, 'dataset', 'train.json')
> torch_dataset = MyDataset(path)
> 
> dit = []
> for line in open(path, 'r', encoding='utf-8'):
>     j_dict = json.loads(line)
>     dit.append(j_dict['context'])
> dset1 = datasets.Dataset.from_list(dit)
> print(dset1)
> dset2 = datasets.Dataset.from_generator(gen)
> print(dset2)
> ```
Hi, when I am using this code to build my own dataset, ` datasets.Dataset.from_generator(gen)` report `TypeError: cannot pickle generator object` whre MyDataset returns a dict like {'image': bytes, 'text': string}. How can I resolve this? Thanks a lot!
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
Hi ! Right now generator functions are expected to be picklable, so that `datasets` can hash it and use the hash to cache the resulting Dataset on disk. Maybe this can be improved.

In the meantime, can you check that you're not using unpickable objects. In your case it looks like you're using a generator object that is unpickable. It might come from an opened file, e.g. this doesn't work:
```python
with open(...) as f:

    def gen():
        for x in f:
            yield json.loads(x)

    ds = Dataset.from_generator(gen)
```
but this does work:
```python
def gen():
    with open(...) as f:
        for x in f:
            yield json.loads(x)

ds = Dataset.from_generator(gen)
```
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
> Hi ! Right now generator functions are expected to be picklable, so that `datasets` can hash it and use the hash to cache the resulting Dataset on disk. Maybe this can be improved.
> 
> In the meantime, can you check that you're not using unpickable objects. In your case it looks like you're using a generator object that is unpickable. It might come from an opened file, e.g. this doesn't work:
> 
> ```python
> with open(...) as f:
> 
>     def gen():
>         for x in f:
>             yield json.loads(x)
> 
>     ds = Dataset.from_generator(gen)
> ```
> 
> but this does work:
> 
> ```python
> def gen():
>     with open(...) as f:
>         for x in f:
>             yield json.loads(x)
> 
> ds = Dataset.from_generator(gen)
> ```

Thanks a lot! That's the reason why I have encountered this issue. Sorry for bothering you again with another problem, since my dataset is large and I use IterableDataset.from_generator which has no attribute with_transform, how can I equip it with some customed preprocessings like Dataset.from_generator? Should I move the preprocessing to the my torch Dataset?
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
Iterable datasets are lazy: exactly like `with_transform` they apply processing on the fly when accessing the examples.

Therefore you can use `my_iterable_dataset.map()` instead :)
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
@lhoestq  I am having a similar issue. Can you help me understand which kinds of generators are picklable? I previously thought that no generators are picklable so I'm intrigued to hear this.
Issue: How to convert  torch.utils.data.Dataset to huggingface dataset?

Comment:
Generator functions are generally picklable. E.g.
```python
import dill as pickle

def generator_fn():
    for i in range(10):
        yield i

pickle.dumps(generator_fn)
```

however generators are not picklable
```python
generator = generator_fn()
pickle.dumps(generator)
# TypeError: cannot pickle 'generator' object
```

Though it can happen that some generator functions are not recursively picklable if they use global objects that are not picklable:
```python
def generator_fn_not_picklable():
    for i in generator:
        yield i

pickle.dumps(generator_fn_not_picklable, recurse=True)
# TypeError: cannot pickle 'generator' object
````
Issue: Create dataset_infos.json with VALIDATION and TEST splits

Comment:
Hi again! Can you please pass the directory name containing the dataset script instead of the script name to `datasets-cli test`?
Issue: Can't create a dataset with `float16` features

Comment:
Hi @dconathan, thanks for reporting.

We rely on Arrow as a backend, and as far as I know currently support for `float16` in Arrow is not fully implemented in Python (C++), hence the `ArrowNotImplementedError` you get.

See, e.g.: https://arrow.apache.org/docs/status.html?highlight=float16#data-types
Issue: Can't create a dataset with `float16` features

Comment:
Thanks for the link…. didn’t realize arrow didn’t support it yet.  Should it be removed from https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/main_classes#datasets.Value until Arrow supports it?
Issue: Can't create a dataset with `float16` features

Comment:
Yes, you are right: maybe we should either remove it from our docs or add a comment explaining the issue.

The thing is that in Arrow it is partially supported: you can create `float16` values, but you can't cast them from/to other types. And current implementation of `Value` always tries to perform a cast from `float64` to `float16`.
Issue: Can't create a dataset with `float16` features

Comment:
Would you accept a PR to fix this? @lhoestq Do you have an idea of how hard it would be to fix?
Issue: Can't create a dataset with `float16` features

Comment:
I think the issue comes mostly from pyarrow not supporting `float16` completely.

For example you stil can't cast from/to `float16`
```python
import numpy as np
import pyarrow as pa

pa.array(range(5)).cast(pa.float16())
# ArrowNotImplementedError: Unsupported cast from int64 to halffloat using function cast_half_float
pa.array(range(5), pa.float32()).cast(pa.float16())
# ArrowNotImplementedError: Unsupported cast from float to halffloat using function cast_half_float
pa.array(range(5), pa.float16())
# ArrowTypeError: Expected np.float16 instance
pa.array(np.arange(5, dtype=np.float16())).cast(pa.float32())
# ArrowNotImplementedError: Unsupported cast from halffloat to float using function cast_float
```
Issue: Can't create a dataset with `float16` features

Comment:
Hmm it seems like we can either:
1. try to fix pyarrow upstream
2. half-support float16 with some workaround to make sure we don't ever do casting internally
Issue: Make `pyarrow` optional

Comment:
The whole datasets library is pretty much a wrapper to pyarrow (just take a look at some of the source for a Dataset) https://github.com/huggingface/datasets/blob/51aef08ad7053c0bfe8f9a961207b26df15850d3/src/datasets/arrow_dataset.py#L639 

I think removing the pyarrow dependency would involve a complete rewrite / a different library with minimal functionality (datasets-lite ?)
Issue: Make `pyarrow` optional

Comment:
Thanks for the proposal, @KOLANICH. And also thanks for your answer, @dconathan.

Indeed, we are using `pyarrow` as the backend for our datasets, in order to cache them and also allow memory-mapping (using datasets larger than your RAM memory).

One way to avoid using `pyarrow` could be loading the datasets in streaming mode, by passing `streaming=True` to `load_dataset`. This way you basically get a generator for the dataset; nothing is downloaded, nor cached.
Issue: Make `pyarrow` optional

Comment:
Thanks for the info. Could `datasets` then be made optional for `transformers` instead? I used `transformers` only to deal with pretrained models to deploy them (convert to ONNX, and then I use TVM), so I don't really need `pyarrow` and `datasets` by now.
Issue: Providing dataset size

Comment:
Hi @sashavor, thanks for your suggestion.

Until now we have the CLI command 
```
datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
```
that generates the `dataset_infos.json` with the size of the downloaded dataset, among other information.

We are currently in the middle of removing those JSON files and putting their information directly in the header of the `README.md` (as YAML tags). Normally, the CLI command should continue working but saving its output to the dataset card instead. See:
- #4926
Issue: Providing dataset size

Comment:
Additionally, the download size can be inferred by doing HEAD requests to the files to be downloaded. And for files hosted on the hub you can even get the file sizes using the Hub API
Issue: Providing dataset size

Comment:
Amazing @albertvillanova ! I think just having that information visible in the dataset info (without having to do any requests/additional coding) would be really useful :hugs:
Issue: Hope to adapt Python3.9 as soon as possible

Comment:
There is this related issue already: https://github.com/huggingface/datasets/issues/4113
And I guess we need a CI job for 3.9 ^^
Issue: [Apple M1] MemoryError: Cannot allocate write+execute memory for ffi.callback()

Comment:
Hi! This seems like a bug in `soundfile`. Could you please open an issue in their repo? `soundfile` works without any issues on my M1, so I'm not sure we can help.
Issue: [Apple M1] MemoryError: Cannot allocate write+execute memory for ffi.callback()

Comment:
Hi @hoangtnm  - I upgraded to python 3.10 and it fixed the problem for me. I was also running 3.8 on an M1 mac.
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
note i have tried the same code with `datasets` version 2.4.0, the outcome is the very same as described above.
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
Seems related to issues #4623 and #4802 so it would appear this issue has been around for a few months.
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
Hi ! `Dataset.from_dict` keeps the data in memory. You can write on disk and reload them with
```python
dataset.save_to_disk("path/to/local")
dataset = load_from_disk("path/to/local")
```
this way you'll end up with a dataset loaded from your disk using memory mapping, and it won't fill up your RAM :)

related to https://github.com/huggingface/datasets/issues/4861