SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Using too many threads results in: Got disconnected from remote data host. Retrying in 5sec

Comment:
> This message just shows that you use the cached version of the c4 module

Ah my bad you're right about the module, but it's also using the downloaded & cached c4 dataset. There's no internet during the runs so it wouldn't work otherwise
Issue: Run 'GPT-J' failure due to download dataset fail (' ConnectionError: Couldn't reach http://eaidata.bmk.sh/data/enron_emails.jsonl.zst ' ) 

Comment:
Thanks for reporting, @shaoyuta.

We have checked and yes, apparently there is an issue with the server hosting the data of the "enron_emails" subset of "the_pile" dataset: http://eaidata.bmk.sh/data/enron_emails.jsonl.zst
It seems to be down: The connection has timed out.

Please note that at the Hugging Face Hub, we are not hosting their data for this dataset, but only a script that downloads the data from their servers. We are updating the data URL to one in another server.

In the meantime, please note that you can train your model in the entire "the_pile" dataset, by passing the "all" config (instead of the "enron_emails" one).
Issue: Run 'GPT-J' failure due to download dataset fail (' ConnectionError: Couldn't reach http://eaidata.bmk.sh/data/enron_emails.jsonl.zst ' ) 

Comment:
We have transferred this issue to the corresponding dataset Community tab: https://huggingface.co/datasets/the_pile/discussions/2

Please, follow the updates there.
Issue: How concatenate `Audio` elements using batch mapping

Comment:
You can try something like this ?
```python
def mapper_function(batch):
   return {"concatenated_audio": [np.concatenate([audio["array"] for audio in batch["audio"]])]}

dataset = dataset.map(
    mapper_function,
    batched=True,
    batch_size=3,
    remove_columns=list(dataset.features),
)
```
Issue: How concatenate `Audio` elements using batch mapping

Comment:
Thanks for the snippet!

One more question. I wonder why those two mappers are working so different that one taking 4 sec while other taking over 1 min :

```python
%%time
def mapper_function1(batch):
    # list_audio
    return {
        "audio": [
            {
                "array": np.concatenate([audio["array"] for audio in batch["audio"]]),
                "sampling_rate": 16_000,
            }
        ]
    }

dataset.map(
    mapper_function1,
    batched=True,
    batch_size=3,
    remove_columns=list(dataset.features),
)

# 100%
# 135/135 [01:13<00:00, 1.93ba/s]
# CPU times: user 1min 10s, sys: 3.21 s, total: 1min 13s
# Wall time: 1min 13s
# Dataset({
#     features: ['audio'],
#     num_rows: 135
# })

# --------------------------------
%%time
def mapper_function2(batch):
    # list_audio
    return {"audio": [np.concatenate([audio["array"] for audio in batch["audio"]])]}

dataset.map(
    mapper_function2,
    batched=True,
    batch_size=3,
    remove_columns=list(dataset.features),
)

# 100%
# 135/135 [00:03<00:00, 40.69ba/s]
# CPU times: user 1.88 s, sys: 1.48 s, total: 3.36 s
# Wall time: 4.8 s
# Dataset({
#     features: ['audio'],
#     num_rows: 135
# })
```
Issue: How concatenate `Audio` elements using batch mapping

Comment:
In the first one you get a dataset with an Audio type, and in the second one you get a dataset with a sequence of floats type.

The Audio type encodes the data as WAV to save disk space, so it takes more time to create.
The Audio type is automatically inferred because you modify the column "audio" which was already an Audio type. If you name it to something else, type inference will use a type struct with array and sampling rate fields.
Issue: IterableDataset returns duplicated data using PyTorch DDP

Comment:
If you use huggingface trainer, you will find the trainer has wrapped a `IterableDatasetShard` to avoid duplication.
See:
https://github.com/huggingface/transformers/blob/dfd818420dcbad68e05a502495cf666d338b2bfb/src/transformers/trainer.py#L835
Issue: IterableDataset returns duplicated data using PyTorch DDP

Comment:
If you want to support it by datasets natively, maybe we also need to change the code in `transformers` ?
Issue: IterableDataset returns duplicated data using PyTorch DDP

Comment:
Maybe something like this then ?
```python
from datasets.distributed import split_dataset_by_node
ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
```

For map-style datasets the implementation is trivial (it can simply use `.shard()`).

For iterable datasets we would need to implement a new ExamplesIterable that would only iterate on a subset of the (possibly shuffled and re-shuffled after each epoch) list of shards, based on the rank and world size.
Issue: IterableDataset returns duplicated data using PyTorch DDP

Comment:
My plan is to skip examples by default to not end up with duplicates.

And if a dataset has a number of shards that is a factor of the world size, then I'd make it more optimized by distributing the shards evenly across nodes instead.
Issue: IterableDataset returns duplicated data using PyTorch DDP

Comment:
Opened a PR here: https://github.com/huggingface/datasets/pull/5369

feel free to play with it and share your feedbacks :)
Issue: IterableDataset returns duplicated data using PyTorch DDP

Comment:
Hi ! Thanks for reporting, you need to pass `seed=` to `shuffle()` or the processes won't use the same seed to shuffle the shards order before assigning each shard to a node.

The issue is that the workers are not using the same seed to shuffle the shards before splitting the shards list by node.
Issue: IterableDataset returns duplicated data using PyTorch DDP

Comment:
I have the same issue
```
ds['train'] = load_dataset(streaming=True)
ds['train'] = split_dataset_by_node(ds['train'], rank=int(os.environ["RANK"]), world_size=int(os.environ["WORLD_SIZE"]))
vectorized_datasets = ds.map(
    prepare_dataset,
    remove_columns=raw_datasets_features,
).with_format("torch")

vectorized_datasets["train"] = vectorized_datasets["train"].shuffle(
    buffer_size=500,
    seed=42,
)

def prepare_dataset(batch):
  ....
  print(f"sentence: {batch['sentence']}, target_text: {batch['target_text']}")
  return batch
```
When using split_dataset_by_node(), the data being read is indeed different for each GPU ID.

```
trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=vectorized_datasets["train"] if training_args.do_train else None,
        eval_dataset=vectorized_datasets["eval"] if training_args.do_eval else None,
        tokenizer=processor,
        callbacks=[ShuffleCallback()],
    )
...
train_result = trainer.train(resume_from_checkpoint=checkpoint)
```
However, when I execute trainer.train(), the data being read is different from what I expected.
Because I print the batch value in prepare_dataset() , I observe that the data is the same for each GPU ID.

How should I handle this issue?
Issue: IterableDataset returns duplicated data using PyTorch DDP

Comment:
There are two ways an iterable dataset can be split by node:
1. if the number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU
2. otherwise, each GPU iterate on the data and at the end keeps 1 sample out of n(GPUs) - skipping the others.

In case 2. it's therefore possible to have the same examples passed to `prepare_dataset` for each GPU.

This doesn't sound optimized though, because it runs the preprocessing on samples that won't be used in the end.

Could you open a new issue so that we can discuss about this and find a solution ?
Issue: Consider using "Sequence" instead of "List"

Comment:
Hi! Linking a comment to provide more info on the issue: https://stackoverflow.com/a/39458225. This means we should replace all (most of) the occurrences of `List` with `Sequence` in function signatures.

@tranhd95 Would you be interested in submitting a PR?
Issue: Consider using "Sequence" instead of "List"

Comment:
Hi all! I tried to reproduce this issue and didn't work for me. Also in your example i noticed that the variables have different names: `list_of_filenames` and `list_of_files`, could this be related to that?
```python
#I found random data in parquet format:
!wget "https://github.com/Teradata/kylo/raw/master/samples/sample-data/parquet/userdata1.parquet"
!wget "https://github.com/Teradata/kylo/raw/master/samples/sample-data/parquet/userdata2.parquet"

#Then i try reproduce
list_of_files = ["userdata1.parquet", "userdata2.parquet"]
ds = Dataset.from_parquet(list_of_files)
```
**My output:**
```python
WARNING:datasets.builder:Using custom data configuration default-e287d097dc54e046
Downloading and preparing dataset parquet/default to /root/.cache/huggingface/datasets/parquet/default-e287d097dc54e046/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 100%
1/1 [00:00<00:00, 40.38it/s]
Extracting data files: 100%
1/1 [00:00<00:00, 23.43it/s]
Dataset parquet downloaded and prepared to /root/.cache/huggingface/datasets/parquet/default-e287d097dc54e046/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec. Subsequent calls will reuse this data.
```
P.S. This is my first experience with open source. So do not judge strictly if I do not understand something)
Issue: Consider using "Sequence" instead of "List"

Comment:
@dantema There is indeed a typo in variable names. Nevertheless, I'm sorry if I was not clear but the output is from `mypy` type checker. You can run the code snippet without issues. The problem is with the type checking.
Issue: Consider using "Sequence" instead of "List"

Comment:
@avinashsai Hi, I've assigned you the issue.

@tranhd95 Yes, feel free to report this in a new issue.
Issue: Consider using "Sequence" instead of "List"

Comment:
@avinashsai Are you still working on this? If not I would like to give it a try.
Issue: __init__() got an unexpected keyword argument 'input_size'

Comment:
Hi @J-shel, thanks for reporting.

I think the issue comes from your call to `load_dataset`. As first argument, you should pass:
- either the name of your dataset ("mrf") if this is already published on the Hub
- or the path to the loading script of your dataset ("path/to/your/local/mrf.py").
Issue: __init__() got an unexpected keyword argument 'input_size'

Comment:
Hi, following your suggestion, I changed my call to load_dataset. Below is the latest:
reader = load_dataset('data/mrf.py',"default", input_size=1024, split=split, streaming=True, keep_in_memory=None)
However, I still got the same error.
I have one question that is if I only define input_size=2048 in BUILDER_CONFIGS, may I specify input_size=1024 when loading the dataset?  Cause I found that I could only specify name="default" since I only define  name="default" in BUILDER_CONFIGS.
Issue: Do we need to implement `_prepare_split`?

Comment:
Hi! `DatasetBuilder` is a parent class for concrete builders: `GeneratorBasedBuilder`, `ArrowBasedBuilder` and `BeamBasedBuilder`. When writing a builder script, these classes are the ones you should inherit from. And since all of them implement `_prepare_split`, you only have to implement the three methods mentioned above.
Issue: Do we need to implement `_prepare_split`?

Comment:
Thanks so much @mariosasko for the fast response! I've been referencing [this page in the docs](https://huggingface.co/docs/datasets/v2.4.0/en/about_dataset_load) because it it pretty comprehensive in terms of what we have to do and I figured since we subclass the `BuilderConfig` the same pattern would hold, but I've also seen the page with those sub-classed builders as well, so that fills in a knowledge gap for me.
Issue: Do we need to implement `_prepare_split`?

Comment:
cc @stevhliu who may have some ideas on how to improve this part of the docs.
Issue: Do we need to implement `_prepare_split`?

Comment:
one more question for my understanding @mariosasko. the requirement of a loading script has always seemed counterintuitive to me. if i have to provide a script with every dataset, what is the point of using `datasets` if we're doing all the work of loading it, I can just do that in my code and skip the datasets integration (this of course discounts other potential benefits around metadata management, etc., my example is just simplest use case though for the sake of discussion).

so i figured I would implement my own `BuilderConfig` and `DatasetBuilder` to handle that portion of it and not have to make a script. i _thought_ this would result in `datasets` (via `download_and_prepare`) then making me something that I could load using `load_dataset` moving forward.

Concretely, i envisioned this pattern being possible:

 ```
class MyBuilderConfig(BuilderConfig):
    def __init__(self, name="my_named_dataset", ...):
        super().__init__(name, ...)

class MyDatasetBuilder(GeneratorBasedBuilder):
   BUILDER_CONFIG_CLASS = MyBuilderConfig
    ....

my_builder = MyDatasetBuilder(...)

# this doesn't exactly work like I thought; I don't get a dataset back, but NoneType instead
# though I can see it loading the files and it generates the cache, etc.
my_dataset = my_builder.download_and_prepare()

# load the dataset in the future by referencing it by name and loading from the cached arrow version
new_instance_of_my_dataset = load_dataset("my_named_dataset")
```

I've seen references to the `save_to_disk` method which might be the next step I need in order to load it by name, in which case, that makes sense, then i just need to debug why `download_and_prepare` isn't returning me a dataset, but I feel like I still have a larger conceptual knowledge gap on how to use the library correctly.

Thanks again in advance!
Issue: Do we need to implement `_prepare_split`?

Comment:
>  the requirement of a loading script has always seemed counterintuitive to me

This is a requirement only for datasets not stored in standard formats such as CSV, JSON, SQL, Parquet, ImageFolder, etc. 

>  if i have to provide a script with every dataset, what is the point of using datasets if we're doing all the work of loading it, I can just do that in my code and skip the datasets integration (this of course discounts other potential benefits around metadata management, etc., my example is just simplest use case though for the sake of discussion)

Our README/documentation lists the main features... 

One of the main ones is that our library makes it easy to work with datasets larger than RAM (thanks to Arrow and the caching mechanism), and this is not trivial to implement.

Regarding the step-by-step builder, this is the pattern:
```python
from datasets import load_dataset_builder
builder = load_dataset_builder("path/to/script") # or direct instantiation with MyDatasetBuilder(...)
builder.download_and_prepare()
dset = builder.as_dataset()
```
Issue: Do we need to implement `_prepare_split`?

Comment:
ok, that makes sense. thank you @mariosasko. I realized i'd never looked on the hub at any of the files associated with any datasets. just did that now and it appears that i'll need to have a script regardless _but_ that will just contain my custom config and builder classes, so without realizing it I was already making my script,  I just need to wrap that in a file that sits alongside my data (I looked at Glue and realized I was already doing what I thought didn't make sense to have to do, lol).

`download_and_prepare` isn't returning me a dataset though, but I'll look into that and open another issue if I can't figure it out.
Issue: Do we need to implement `_prepare_split`?

Comment:
`download_and_prepare` downloads and prepares the arrow files. You need to call `as_dataset` on the builder to get the dataset.
Issue: Do we need to implement `_prepare_split`?

Comment:
looks like my arrow files are all empty @mariosasko 

![image](https://user-images.githubusercontent.com/7530947/208179977-9ae62c9a-866c-472b-9a09-25d1191188fb.png)


i also see the `incomplete_info.lock` file a level up too. seems like the data isn't being persisted to disk when I call `download_and_prepare`. is there something else i need to do before then, perhaps?
Issue: Do we need to implement `_prepare_split`?

Comment:
quick update @mariosasko. i got it working! i had to downgrade to `datasets==2.4.0`. testing other versions now and will let you know the results.
Issue: Do we need to implement `_prepare_split`?

Comment:
I've tested with every version of `datasets>2.4.0` and i get the same error with all of them.
Issue: The data downloaded in the download folder of the cache does not respect `umask`

Comment:
note, that `datasets` already did some of that umask fixing in the past and also at the hub - the recent work on the hub about the same: https://github.com/huggingface/huggingface_hub/pull/1220

Also I noticed that each file has a .json counterpart and the latter always has the correct perms:

```
-rw------- 1 uue59kq cnw 173M Dec  9 01:37 537596e64721e2ae3d98785b91d30fda0360c196a8224e29658ad629e7303a4d
-rw-rw---- 1 uue59kq cnw  101 Dec  9 01:37 537596e64721e2ae3d98785b91d30fda0360c196a8224e29658ad629e7303a4d.json
```

so perhaps cheating is possible and syncing the perms between the 2 will do the trick.
Issue: Wrong dtype for array in audio features

Comment:
After some more investigation, this is due to [this line of code](https://github.com/huggingface/datasets/blob/main/src/datasets/features/audio.py#L279). The function `sf.read(file)` should be updated to `sf.read(file, dtype="float32")`

Indeed, the default value in soundfile is `float64` ([see here](https://pysoundfile.readthedocs.io/en/latest/#soundfile.read)).
Issue: Wrong dtype for array in audio features

Comment:
@qmeeus I agree, decoding of different audio formats should return the same dtypes indeed!

But note that here you are concatenating datasets with different sampling rates: 48000 for CommonVoice and 16000 for Voxpopuli. So you should cast them to the same sampling rate value before interleaving, for example:
```
cv = cv.cast_column("audio", Audio(sampling_rate=16000))
```
otherwise you would get the same error because features of the same column ("audio") are not the same.

Also, the error you get is unexpected. Could you please confirm that you use the latest main version of the `datasets`?  We had an issue that could lead to an error like this after using `rename_column` method, but it was fixed in https://github.com/huggingface/datasets/pull/5287
Issue: Wrong dtype for array in audio features

Comment:
Hi Polina,
Sorry for the late answer
It is possible that the issue was due to a bug that is now fixed. I installed an editable version of datasets from github, but I don't recall whether I had updated it at the time of the issue. My research led me to other directions so I did not follow through on the interleave datasets.
Issue: Emotion dataset cannot be downloaded

Comment:
Hi @cbarond there's already an open issue at https://github.com/dair-ai/emotion_dataset/issues/5, as the data seems to be missing now, so check that issue instead 👍🏻
Issue: Emotion dataset cannot be downloaded

Comment:
Thanks @cbarond for reporting and @alvarobartt for pointing to the issue we opened in the author's repo.

Indeed, this issue was first raised in the "emotion" dataset Community tab: https://huggingface.co/datasets/emotion/discussions/3

I'm closing this issue and leave the issue above for the subsequent updates.

Duplicate of: https://huggingface.co/datasets/emotion/discussions/3
Issue: Emotion dataset cannot be downloaded

Comment:
> try using "SetFit/emotion" instead

I' replaced "emotion" with "SetFit/Emotion", but the code is getting stuck at

`emotions = load_dataset("SetFit/emotion")`

I pause execution using the debugger, and it takes me to filelock.py:226

`with self._thread_lock:`

Do you know a way to get past this issue?
Issue: `map()` stops every 1000 steps

Comment:
Hi !

> It starts using all the cores (I am not sure why because I did not pass num_proc)

The tokenizer uses Rust code that is multithreaded. And maybe the `feature_extractor` might run some things in parallel as well - but I'm not super familiar with its internals.

> then progress bar stops at every 1k steps. (starts using a single core)

Every 1000 examples we flush the processed examples to disk. It is this way because Arrow is a columnar format: you must write data chunk by chunk. The processing in on hold while writing right now - maybe this can be improved in the future.
Issue: `map()` stops every 1000 steps

Comment:
Hi @lhoestq 
Thanks for the explanation! it was so helpful! Let me check why `feature_extractor` is running on multiple cpus.
Issue: Support webdataset format

Comment:
I like the idea of having `webdataset` as an optional dependency to ensure our loader generates web datasets the same way as the main project.
Issue: Support webdataset format

Comment:
Webdataset is the one of the most popular dataset formats for large scale computer vision tasks. Upvote for this issue.
Issue: Support webdataset format

Comment:
We haven't had the bandwidth to implement it so far, but if someone wants to give it a shot please don't hesitate ^^
Issue: Passing numpy array to ClassLabel names causes ValueError

Comment:
Should `datasets` allow `ClassLabel` input parameter to be an `np.array` even though internally we need to cast it to a Python list? @lhoestq @mariosasko
Issue: Passing numpy array to ClassLabel names causes ValueError

Comment:
Hi! No, I don't think so. The `names` parameter is [annotated](https://github.com/huggingface/datasets/blob/582236640b9109988e5f7a16a8353696ffa09a16/src/datasets/features/features.py#L892) as `List[str]` (**NumPy arrays are not lists**), and considering that type checking is not a common practice in Python, I think we can leave the code as-is.
Issue: Passing numpy array to ClassLabel names causes ValueError

Comment:
I appreciate it is the wrong type, and that type checking is not common, but I think there's a few circumstances that make it a good idea from a usability perspective.

It's quite a difficult error to debug because it comes from a utility function (so it's not immediately obvious which parameter caused it). What makes it even more difficult is the exception happens when the features instance is used to instantiate the dataset, **not** when when the wrong type is actually passed when the features is instantiated. When I was debugging the error, I didn't really consider it could be an issue with the features instance because it had instantiated fine. It's also not one of the more common exceptions caused by trying to use a non-list as a list.

It's also relatively easy to accidentally get a numpy array of class types (e.g. calling `unique()` on a pandas dataframe column). Additionally, passing in a `set` instead of the list (again, relatively easy because people may run `set(classes)` to generate uniques) causes an error when the features instance is used, albeit a slightly more obvious one.

The names list is already being processed and validated in the `__post_init__` method anyway, so it would not really be adding any complexity to check it is actually a list here too. I'm happy to contribute this change if you change your mind about whether it's worthwhile.
Issue: Passing numpy array to ClassLabel names causes ValueError

Comment:
I agree that it's not easy to debug this issue, so perhaps we could add some basic type checking (e.g. `not isinstance(names, list)` -> error) to make debugging easier. Feel free to submit a PR.

> Additionally, passing in a set instead of the list (again, relatively easy because people may run set(classes) to generate uniques) causes an error when the features instance is used, albeit a slightly more obvious one.

`set` is an unordered structure (it's ordered in Python 3.6+, but this is CPython's implementation detail), and the order of ClassLabel `names` matters, so this doesn't require a fix.
Issue: Passing numpy array to ClassLabel names causes ValueError

Comment:
What about checking for `Sequence` instead? I think users can pass a list or a tuple as well.