SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: load_dataset from s3 file system through streaming can't not iterate data 

Comment:
I tried `storage_options` before, but it doesn't work, I checked our source code and I found that we even didn't pass this parameter to the following process. if I use `storage_options` instead of `use_auth_token`, then I also need to change another place of the code. the last line of `streaming_download_manager.py`. our code only passes the `use_auth_token` to the following handler, but does nothing to the `storage_options`
<img width="1050" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/5be90933-3331-4ecf-9e11-34f9852d8f92">
Issue: load_dataset from s3 file system through streaming can't not iterate data 

Comment:
Cloud storage support is still experimental indeed and you can expect some bugs.

I think we need to pass the storage options anywhere use_auth_token is passed in indeed. Let me know if you'd be interested in contributing a fix !
Issue: load_dataset from s3 file system through streaming can't not iterate data 

Comment:
Oh, that's great, I really like to fix it. because datasets is really useful and most of our projects need to use it, but we can store our data on the internet due to security reasons. fix it not only make our own work more efficient but also can benefit others who use it.
Issue: Prefetching for IterableDataset

Comment:
Very cool! Do you have a link to the code that you're using to eagerly fetch the data? Would also be interested in hacking around something here for pre-fetching iterable datasets
Issue: Prefetching for IterableDataset

Comment:
I ended up just switching back to the pytorch dataloader and using it's multiprocessing functionality to handle this :(. I'm just not that familiar with python multiprocessing to get something to work in jupyter (kept having weird behaviors happening with zombies living after the cell finished).
Issue: Prefetching for IterableDataset

Comment:
Ultimately settled on using webdataset to circumvent huggingface datasets entirely. Would definitely switch back if: https://github.com/huggingface/datasets/issues/5337 was resolved.
Issue: Prefetching for IterableDataset

Comment:
Hi! You can combine `datasets` with `torchdata` to prefetch `IterableDataset`'s samples:
```python
from datasets import load_dataset
from torchdata.datapipes.iter import IterableWrapper, HuggingFaceHubReader
from torch.utils.data import DataLoader

ds = load_dataset("sst", split="train", streaming=True)
# processing...
dp = IterableWrapper(ds)
dp = dp.prefetch(100)
dl = DataLoader(dp, batch_size=8)

i = iter(dl)
next(i)
```
Issue: Prefetching for IterableDataset

Comment:
Hey @mariosasko! Thanks for the tip here - introducing prefetch with `torchdata` didn't really give me any performance difference vs not prefetching, but the concept is definitely one that could be really beneficial. Are there any benchmarks that show the speed-up you can get with `torchdata`'s prefetch just for comparison?
Issue: Request for text deduplication feature

Comment:
The "exact match" deduplication will be possible when we resolve https://github.com/huggingface/datasets/issues/2514 (first, https://github.com/apache/arrow/issues/30950 needs to be addressed on the Arrow side). In the meantime, you can use Polars or DuckDB (e.g., via [datasets-sql](https://github.com/mariosasko/datasets_sql)).

Fuzzy deduplication is out-of-scope for now ([splink](https://github.com/moj-analytical-services/splink) is probably the best tool for it).
Issue: Request for text deduplication feature

Comment:
I have been using polars to remove duplicates but it would be nice to do it directly in pyarrow.

For example,

1. Read dataset with pyarrow
2. Use scan_pyarrow_dataset() with Polars to create a LazyFrame
3. Use sort and unique to remove duplicates based on a subset of columns
4. Convert to table and save data with ds.write_dataset()

There are times where that workflow makes perfect sense because I do additional transformations with Polars. Most of the time I am simply just reading dataset A and writing dataset B without duplicates though, and I wish I could use a pyarrow scanner or table directly.
Issue: Incompatibility with DataLab

Comment:
Indeed, `clobber=True` (with a warning if the existing protocol will be overwritten) should fix the issue, but maybe a better solution is to register our compression filesystem before the script is executed and unregister them afterward. WDYT @lhoestq @albertvillanova?
Issue: Incompatibility with DataLab

Comment:
I think we should use clobber and show a warning if it overwrote a registered filesystem indeed ! This way the user can re-register the filesystems if needed. Though they should probably be compatible (and maybe do the exact same thing) so I wouldn't de-register the `datasets` filesystems
Issue: Using as_dataset on a "parquet" builder  

Comment:
Hi! You can refer to [this doc](https://huggingface.co/docs/datasets/filesystems#load-and-save-your-datasets-using-your-cloud-storage-filesystem) to see the intended usage (basically, it skips the Arrow -> Parquet conversion step in `ds = load_dataset(...); ds.to_parquet("path/to/parquet")`) and allows writing Parquet to remote storage unlike `to_parquet`).

> I guess I'd expect as_dataset to generate the dataset in arrow format if it has to, or to suggest an alternative way to load the dataset (I've also tried other methods with load_dataset to no avail, probably due to misunderstandings on my part).

`as_dataset` does not work with `file_format="parquet"` files as Parquet files cannot be memory-mapped, so I think we should just raise an error in that case.
Issue: data configuration hash suffix depends on uncanonicalized data_dir

Comment:
Indeed, it makes sense to normalize `data_dir`. Feel free to submit a PR (this can be "fixed" [here](https://github.com/huggingface/datasets/blob/89f775226321ba94e5bf4670a323c0fb44f5f65c/src/datasets/builder.py#L173))
Issue: Behaviour difference between datasets.map and IterableDatasets.map

Comment:
PS - some work is definitely needed for 'special cases' docs, not explanations, just usages of 'functions' under mixture of special cases, like a combination of custom databuilder + iterable dataset for large size + dynamic .map() application.
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
Hi @PhilippeMoussalli thanks for opening a detailed issue. It seems the issue is more related to the `datasets` library so I'll ping @lhoestq @mariosasko on this one :) 

(edit: also can one of you move the issue to the datasets repo? Thanks in advance 🙏)
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
Hi ! The `Image()` info is stored in the **schema metadata**. More precisely there should be a "huggingface" field in the schema metadata that contains the `datasets` feature type of each column.

To fix your issue, you can use the same schema as the original Parquet files to write the new ones. You can also get the schema with metadata from a `Features` object, e.g.

```python
from datasets import Features, Image, Value

features = Features({"image": Image(), "text": Value("string")})
schema = features.arrow_schema
print(schema.metadata)
# {b'huggingface': b'{"info": {"features": {"image": {"_type": "Image"}, "text": {"dtype": "string", "_type": "Value"}}}}'}
```
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
It appears that the parquet files at `hf://datasets/lambdalabs/pokemon-blip-captions` don't have this metadata, and it is defined in the dataset_infos.json instead (legacy).

You can get the right schema with the HF metadata this way:

```python
from datasets import load_dataset_builder

features = load_dataset_builder("lambdalabs/pokemon-blip-captions").info.features
schema = features.arrow_schema
```
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
Btw in the future we might add support for an dedicated Image extension type in Arrow so that you won't need to add the schema metadata anymore ;)
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
Thanks @Wauplin @lhoestq for the quick reply :)! 

I tried your approach by passing the huggingface schema to the dask writer 

```
from datasets import Features, Image, Value
df = dd.read_parquet(f"hf://datasets/lambdalabs/pokemon-blip-captions",index=False)
features = Features({"image": Image(), "text": Value("string")})
schema = features.arrow_schema
dd.to_parquet(df, path = "hf://datasets/philippemo/dummy_dataset/data", schema=schema)
```
At first it didn't work as I was not able to visualize the images, so then I manually added the `dataset_infos.json` from the example dataset and it worked :)

However, It's not very ideal since there are some metadata in that file that need to be computed in order to load the data properly such as `num_of_bytes` and `num_examples` which might be unknown in my use case. 

![Screenshot from 2023-05-16 16-54-55](https://github.com/huggingface/datasets/assets/47530815/b2b448d2-d3d8-43a7-9682-9c0187a5192b)

Do you have any pointers there? you mentioned that `datasets_info.json` will be deprecated/legacy. Could you point me to some example image datasets on the hub that are stored as parquet and don't have the `datasets_info.json`?
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
You don't need the dataset_infos.json file as long as you have the schema with HF metadata ;)
I could also check that it works fine myself on the git revision without the dataset_infos.json file.

What made you think it didn't work ?
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
> You don't need the dataset_infos.json file as long as you have the schema with HF metadata ;) I could also check that it works fine myself on the git revision without the dataset_infos.json file.
> 
> What made you think it didn't work ?

Those are two identical dataset repos where both were pushed with dask with the specified schema you mentioned above. I then uploaded the `dataset_infos.json` manually taken from the original example dataset into one of them. 

* **With schema**: https://huggingface.co/datasets/philippemo/dummy_dataset_with_schema
* **Without schema**: https://huggingface.co/datasets/philippemo/dummy_dataset_without_schema

You can see that in the examples without schema the images fail to render properly. When loaded with `datasets` they return an dict and not a Pillow Image
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
I see ! I think it's a bug on our side - it should work without the metadata - let me investigate
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
Alright, it's fixed: https://huggingface.co/datasets/philippemo/dummy_dataset_without_schema

It shows the image correctly now - even without the extra metadata :)
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
Thanks @lhoestq! 
I tested pushing a dataset again without the metadata and it works perfectly! 
I appreciate the help
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
Hi @lhoestq, 

I'v tried pushing another dataset again and I think the issue reappeared again: 

```
df = dd.read_parquet(f"hf://datasets/lambdalabs/pokemon-blip-captions")
features = datasets.Features({"image": datasets.Image(), "text": datasets.Value("string")})
schema = features.arrow_schema
dd.to_parquet(df, path = "hf://datasets/philippemo/dummy_dataset_without_schema_12_06/data", schema=schema)
```

Here is the dataset:  
 https://huggingface.co/datasets/philippemo/dummy_dataset_without_schema_12_06
The one that was working 2 weeks ago still seems to be intact though, it might be that It rendered properly when it was initially submitted and after this something was reverted from your side:
https://huggingface.co/datasets/philippemo/dummy_dataset_without_schema

It's weird because nothing really changed from the implementation, might be another issue in the hub backend. Do you have any pointers on how to resolve this?
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
We're doing some changes in the way we're handling image parquet datasets right now. We'll include the fix from https://github.com/huggingface/datasets/pull/5921 in the new datasets-server version in the coming days
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
alright thanks for the update :), would that be part of the new release of datasets or is it something separate? if so, where can I track it?
Issue: Image Encoding Issue when submitting a Parquet Dataset

Comment:
Once the new version of `datasets` is released (tomorrow probably) we'll open an issue on https://github.com/huggingface/datasets-server to update to this version :)
Issue: Is it possible to change a cached file and 're-cache' it instead of re-generating?

Comment:
Arrow files/primitives (tables and arrays) are immutable, so re-generating them is the only option, I'm afraid.
Issue: Slow iteration over Torch tensors

Comment:
I am highly interested performance of dataset so I ran your example as a curious user.
```python
train_dataset.cast_column("x", Array3D(shape=img_shape, dtype="float32"))
```
have return values and "x" is a new column, it shoulde be
```python
ds=train_dataset.cast_column("img", Array3D(shape=(3,32,32), dtype="float32"))
```
I rewrite your example as
```python
train_dataset = load_dataset(
    'cifar100',
    split='train',
    use_auth_token=True,
)
transform_func = torchvision.transforms.Compose([
    ToTensor(), 
    Normalize(mean=[0.485, 0.456, 0.406], std= [0.229, 0.224, 0.225]),]          
)
    
train_dataset = train_dataset.map(
    desc=f"Preprocessing samples",
    function=lambda x: {"img": transform_func(x["img"])},
)
ds=train_dataset.cast_column("img", Array3D(shape=(3,32,32), dtype="float32"))
for i in tqdm(ds):
    pass
```
that require ~11s in my environment. While
```python
ds = load_dataset(
    'cifar100',
    split='train',
    use_auth_token=True,
)

for i in tqdm(ds):
    pass
```
only need ~6s. (So I guess it's still undesirable)
Issue: IndexError: list index out of range with data hosted on Zenodo

Comment:
This error is also raised when data is hosted on Google Drive:
- https://huggingface.co/datasets/docred/discussions/5
- https://huggingface.co/datasets/linnaeus/discussions/3
- https://huggingface.co/datasets/poleval2019_mt/discussions/3
- https://huggingface.co/datasets/reddit_tifu/discussions/2
- https://huggingface.co/datasets/species_800/discussions/3
- https://huggingface.co/datasets/wiki_lingua/discussions/1
- https://huggingface.co/datasets/yoruba_text_c3/discussions/1
Issue: Throw an error when dataset improperly indexed

Comment:
Thanks for reporting, @sarahwie.

Please note that in `datasets` we do not have vectorized operation like `pandas`. Therefore, your equality comparisons above are `False`:
- For example: `squad['question']` returns a `list`, and this list is not equal to `"Who was the Norse leader?"`

The `False` value is equivalent to `0` when indexing a dataset, thus the reason why you get the first element (with index 0): 
- For example: `squad[False]` is equivalent to `squad[0]`

Maybe we should an exception instead of assuming that `False` is equivalent to `0` (and `True` is equivalent to `1`) in the context of indexing.
Issue: Adding chemistry dataset/models in huggingface

Comment:
Hi! 

This would be a nice addition to the Hub! You can find the existing chemistry datasets/models on the Hub (using the `chemistry` tag) [here](https://huggingface.co/search/full-text?q=chemistry&type=model&type=dataset).

Feel free to ping us here on the Hub if you need help adding the datasets.
Issue: Error loading natural_questions

Comment:
Hi! You can avoid this error by using the preprocessed version:
```python
import datasets
ds = datasets.load_dataset('natural_questions')
```

PS: Once we finish https://github.com/huggingface/datasets/pull/5364, this error will no longer be a problem.
Issue: Error loading natural_questions

Comment:
> Hi! You can avoid this error by using the preprocessed version:
> 
> ```python
> import datasets
> ds = datasets.load_dataset('natural_questions')
> ```
> 
> PS: Once we finish #5364, this error will no longer be a problem.

Thanks, wish #5364  finish early
Issue: `to_tf_dataset` consumes too much memory

Comment:
Cc @amyeroberts @Rocketknight1 

Indded I think it's because it does something like this under the hood when there's no multiprocessing:

```python
tf_dataset = tf_dataset.shuffle(len(dataset))
```

PS: with multiprocessing it appears to be different:

```python
indices = np.arange(len(dataset))
if shuffle:
    np.random.shuffle(indices)
```
Issue: `to_tf_dataset` consumes too much memory

Comment:
Hi @massquantity, the dataset being shuffled there is not the full dataset. If you look at [the line above](https://github.com/huggingface/datasets/blob/main/src/datasets/utils/tf_utils.py#L182), the dataset is actually just a single indices array at that point, and that array is the only thing that gets fully loaded into memory and shuffled. We then load samples from the dataset by applying a transform function to the shuffled dataset, which fetches samples based on the indices it receives.

If your dataset is **really** gigantic, then this index tensor might be a memory issue, but since it's just an int64 tensor it will only use 1GB of memory per 125 million samples.

Still, if you're encountering memory issues, there might be another cause here - can you share some code to reproduce the error, or does it depend on some internal/proprietary dataset?
Issue: `to_tf_dataset` consumes too much memory

Comment:
Hi @Rocketknight1, you're right and I also noticed that only indices are used in shuffling. My data has shape (50000000, 10), but really the problem doesn't relate to a specific dataset. Simply running the following code costs me 10GB of memory.

```python
from datasets import Dataset

def gen():
    for i in range(50000000):
        yield {"data": i}

ds = Dataset.from_generator(gen, cache_dir="./huggingface")

tf_ds = ds.to_tf_dataset(
    batch_size=1,
    shuffle=True,
    drop_remainder=False,
    prefetch=True,
)
tf_ds = iter(tf_ds)
next(tf_ds)
# {'data': <tf.Tensor: shape=(1,), dtype=int64, numpy=array([0])>}
```

I just realized maybe it was an issue from tensorflow (I'm using tf 2.12). So I tried the following code, and it used 10GB of memory too.
```python
import numpy as np
import tensorflow as tf

data_size = 50000000
tf_dataset = tf.data.Dataset.from_tensor_slices(np.arange(data_size))
tf_dataset = iter(tf_dataset.shuffle(data_size))
next(tf_dataset)
# <tf.Tensor: shape=(), dtype=int64, numpy=24774043>
```

By the way, as @lhoestq mentioned, multiprocessing uses numpy shuffling, and it uses less than 1 GB of memory:
```python
tf_ds_mp = ds.to_tf_dataset(
    batch_size=1,
    shuffle=True,
    drop_remainder=False,
    prefetch=True,
    num_workers=2,
)
```
Issue: `to_tf_dataset` consumes too much memory

Comment:
Thanks for that reproduction script - I've confirmed the same issue is occurring for me. Investigating it now!
Issue: `to_tf_dataset` consumes too much memory

Comment:
Update: The memory usage is occurring in creation of the index and shuffle buffer. You can reproduce it very simply with:

```python
import tensorflow as tf
indices = tf.range(50_000_000, dtype=tf.int64)
dataset = tf.data.Dataset.from_tensor_slices(indices)
dataset = dataset.shuffle(len(dataset))
print(next(iter(dataset))
```
When I wrote this code I thought `tf.data` had an optimization for shuffling an entire tensor that wouldn't create the entire shuffle buffer, but evidently it's just creating the enormous buffer in memory. I'll see if I can find a more efficient way to do this - we might end up moving everything to the `numpy` multiprocessing path to avoid it.
Issue: Can not load audiofolder dataset on kaggle

Comment:
Hi! `audiofolder` requires `datasets>=2.5.0`, so please update the `datasets`' installation (`pip install -U datasets`) in the environment (and restart the env for the update to take effect) to resolve the issue.
Issue: Can not load audiofolder dataset on kaggle

Comment:
> Hi! `audiofolder` requires `datasets>=2.5.0`, so please update the `datasets`' installation (`pip install -U datasets`) in the environment to resolve the issue.

I don't think it is a problem of the version. It runs ok on colab or local machine. Only on kaggle will has this bug.
Issue: Can not load audiofolder dataset on kaggle

Comment:
Based on your dataset info, the installed version is `2.1.0`, which does not include `audiofolder`.

By default, Kaggle preinstalls `datasets` into a new env, but the version it installs is outdated and does not contain newer features such as `audiofolder`
Issue: Streaming IterableDataset not working with translation pipeline

Comment:
[`KeyDataset`](https://github.com/huggingface/transformers/blob/7f8b909189547944617741d8d3c6c84504701693/src/transformers/pipelines/pt_utils.py#L296) doesn't support iterable datasets, so you either need to implement a version that does (and also indexing nested (translation) fields):

```python
from torch.utils.data import Dataset, IterableDataset

def build_key_fetcher(key: str):
    def _key_fetcher(item):
        for sub_key in key.split("."):
            item = item[sub_key]
        return item
    return _key_fetcher

class KeyDataset(Dataset):
    def __new__(cls, dataset: Dataset, key: str):
        cls = _KeyIterableDataset if isinstance(dataset, IterableDataset) else _KeyMapDataset
        self = object.__new__(cls)
        self.dataset = dataset
        self.key = key
        self._key_fetcher = build_key_fetcher(key)
        return self

class _KeyMapDataset(KeyDataset):
    def __getitem__(self, i):
        return self._key_fetcher(self.dataset[i])
    
    def __len__(self):
        return len(self.dataset)


class _KeyIterableDataset(KeyDataset):
    def __iter__(self):
        for ex in self.dataset:
            yield self._key_fetcher(ex)

ks = KeyDataset(ds, "translation.en")
```

or use `IterableDataset`'s `map`:
```python
def fetch_en_translation(ex):
    return {"en": ex["translation"]["en"]}
ks = ds.map(fetch_en_translation, remove_columns=ds.column_names) 
```

cc @sgugger: Perhaps the `KeyDataset` + PyTorch `IterableDataset` case should be supported by Transformers