SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Config name / split name lost after map with multiproc

Comment:
This must be due to DatasetInfo.from_merge which drops them and is used in `concatenate_datasets`.

And you're experiencing this issue because multiprocessing does concatenate the resulting datasets from each process.

Maybe they should be kept if all the subdatasets share the same values for config_name and split
Issue: "Couldn't cast array of type" in complex datasets

Comment:
Thanks for reporting! 

Specifying the target features explicitly should avoid this error:
```python
dataset = dataset.map(
    batch_process,
    batched=True,
    batch_size=1,
    num_proc=1,
    remove_columns=dataset.column_names,
    features=datasets.Features({"texts": datasets.Sequence(datasets.Value("string"))})
)
```

This error stems from our type promotion not handling the nested case. But this promotion/casting allocates memory in most scenarios, which can be problematic for large datasets, so explicitly passing the features is the optimal solution.
Issue: "Couldn't cast array of type" in complex datasets

Comment:
Hi @mariosasko thanks for the context, this is helpful to know. Would it be worth having some logic to generate this explicit feature specification automatically if a type annotation for a .map returns a dataclass that can be inferred?

Feels like something that would be easy to implement and could save memory / deal with this case in a standardized way.
Issue: "Couldn't cast array of type" in complex datasets

Comment:
> . Would it be worth having some logic to generate this explicit feature specification automatically if a type annotation for a .map returns a dataclass that can be inferred?

Interesting proposal! Yes, we could consider doing this if the (return) type hint is `TypedDict`, and raise an error that type hints are incorrect if the cast using the inferred types fails.
Issue: "Couldn't cast array of type" in complex datasets

Comment:
@mariosasko Put up an initial PR to implement this proposal. Let me know your thoughts on direction and what else should be in-scope here.
Issue: Got an error  _pickle.PicklingError use Dataset.from_spark.

Comment:
i got error using method from_spark  when using multi-node Spark cluster. seems could only use "from_spark" in local?
Issue: Got an error  _pickle.PicklingError use Dataset.from_spark.

Comment:
cc @maddiedawson it looks like there an issue with `_validate_cache_dir` ?

It looks like the function passed to mapPartitions has a reference to the Spark dataset builder, and therefore contains the SparkContext itself.

I think it can be fixed by defining `create_cache_and_write_probe` outside the Spark dataset builder, and pass a `partial(create_cache_and_write_probe, cache_dir=self._cache_dir)` to `mapPartitions`
Issue: Got an error  _pickle.PicklingError use Dataset.from_spark.

Comment:
Just saw this; thanks for flagging! Your proposed solution sounds good. I can prepare a PR
Issue: Got an error  _pickle.PicklingError use Dataset.from_spark.

Comment:
@maddiedawson  can you show me the demo ,so i can test in local .before your PR
Issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway

Comment:
Does "number of shards" refer to the total number of data?

my config:
nproc_per_node=2
ds=ds['train'] = load_dataset(streaming=True).take(50000)

I'm test again: in prepare_data(),  data have the same for each GPU
Issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway

Comment:
The number of shards is `ds.n_shards`. It corresponds generally to the number of files the dataset is made of, to be able to distribute to several nodes.

**You don't end up with the same data per GPU**. But all the samples are going through your preprocessing function you pass to map. They are just skipped afterwards to only keep 1 sample out of n(GPUs)
Issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway

Comment:
For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end. 
Is my understanding correct?

Where can I print the actual training data for each GPU?
Issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway

Comment:
> For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
Is my understanding correct?

Yes exactly :)

> Where can I print the actual training data for each GPU?

You should call print in the data_collator
Issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway

Comment:
I print out n_shards, and under multiple GPUs, this value is always 1.
Is this value correct?
Issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway

Comment:
Yes it's correct, and it explains why you always have the same data passed to your map function (the data can't be split).

But after being passed to `map`, each GPU keeps one example out of n(GPUs) so that you don't end up with duplicate data across GPUs
Issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway

Comment:
> > For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
> > Is my understanding correct?
> 
> Yes exactly :)
> 
> > Where can I print the actual training data for each GPU?
> 
> You should call print in the data_collator

OK, when printing the train data in the data collator, each GPU sees different data.

Thanks for your reply
Issue: Strange bug in loading local JSON files, using load_dataset

Comment:
This is the actual error:
```
Failed to read file '/home/lakala/hjc/code/pycode/glm/temp.json' with error <class 'pyarrow.lib.ArrowInvalid'>: cannot mix list and non-list, non-null values
```
Which means some samples are incorrectly formatted.

PyArrow, a storage backend that we use under the hood, requires that all the list elements have the same level of nesting (same number of dimensions) or are `None`.
```python
import pyarrow as pa
pa.array([[1, 2, 3], 2]) # ArrowInvalid: cannot mix list and non-list, non-null values
pa.array([[1, 2, 3], [2]]) # works
```
Issue: Strange bug in loading local JSON files, using load_dataset

Comment:
@mariosasko 
I used the same operation to check the original data before and after slicing.
This is reflected in my code.
160000 is not a specific number.
I can also get output using 150000.
This doesn't seem to align very well with what you said.
Because if only some sample formats are incorrect.
So there should be an error in one of the front and back slices.
thank you for your reply.
Issue: Strange bug in loading local JSON files, using load_dataset

Comment:
Our JSON loader does the following in your case:

```python
import json
import pyarrow as pa

with open(file, encoding="utf-8") as f:
    dataset = json.load(f)
keys = set().union(*[row.keys() for row in dataset])
mapping = {col: [row.get(col) for row in dataset] for col in keys}
pa_table = pa.Table.from_pydict(mapping) # the ArrowInvalid error comes from here
```

So if this code throws an error with correctly-formatted JSON, then this is an Arrow bug and should be reported in their repo.

> I used the same operation to check the original data before and after slicing.
This is reflected in my code.
160000 is not a specific number.
I can also get output using 150000.
This doesn't seem to align very well with what you said.
Because if only some sample formats are incorrect.
So there should be an error in one of the front and back slices.

You should shuffle the data to make sure that's not the case
Issue: Bad error message when trying to download gated dataset

Comment:
cc @sanchit-gandhi @Vaibhavs10 @lhoestq - this is mainly for demos that use Common Voice datasets as done here: https://github.com/facebookresearch/fairseq/tree/main/examples/mms#-transformers
Issue: Bad error message when trying to download gated dataset

Comment:
Hi ! the error for me is

```
FileNotFoundError: Couldn't find a dataset script at /content/mozilla-foundation/common_voice_13_0/common_voice_13_0.py or any data file in the same directory. Couldn't find 'mozilla-foundation/common_voice_13_0' on the Hugging Face Hub either: FileNotFoundError: Dataset 'mozilla-foundation/common_voice_13_0' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

And tbh idk how you managed to get your error. "n_shards.json" is not even a thing in `datasets`
Issue: Bad error message when trying to download gated dataset

Comment:
Okay, I am able to reproduce @patrickvonplaten's original error: https://github.com/Vaibhavs10/scratchpad/blob/main/cv13_datasets_test.ipynb

Also not sure why it looks for `n_shards.json`
Issue: Bad error message when trying to download gated dataset

Comment:
Ok I see, this file is downloaded from the CV dataset script - let me investigate
Issue: Bad error message when trying to download gated dataset

Comment:
Ok I see: when you log out you no longer have access to the repository.

Therefore the dataset script is loaded from cache:
```
WARNING:datasets.load:Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_13_0/22809012aac1fc9803eaffc44122e4149043748e93933935d5ea19898587e4d7 (last modified on Wed Jun 14 10:13:17 2023) since it couldn't be found locally at mozilla-foundation/common_voice_13_0., or remotely on the Hugging Face Hub.
```

and the script tries to download the n_shards.json but fails
Issue: Support for data with instance-wise dictionary as features

Comment:
Hi ! We use the Arrow columnar format under the hood, which doesn't support such dictionaries: each field must have a fixed type and exist in each sample.

Instead you can restructure your data like
```
{
    "index": 0,
    "keys": ["2 * x + y >= 3"],
    "values": [["2 * x + y >= 3", "4 * x + 2 * y >= 6"]],
    }
},
...
{
    "index": 9999,
    "keys": ["x >= 6"],
    "values": [["x >= 6", "x >= 0", "x >= -1"]],
},
...
```
Issue: Return the audio filename when decoding fails due to corrupt files

Comment:
Hi ! The audio data don't always exist as files on disk - the blobs are often stored in the Arrow files. For now I'd suggest disabling decoding with `.cast_column("audio", Audio(decode=False))` and apply your own decoding that handles corrupted files (maybe to filter them out ?)

cc @sanchit-gandhi since it's related to our discussion about allowing users to make decoding return `None` and show a warning when there are corrupted files
Issue: Return the audio filename when decoding fails due to corrupt files

Comment:
Thanks @lhoestq, I wasn't aware of the decode flag. It makes more sense as you say to show a warning when there are corrupted files together with some metadata of the file that allows to filter them from the dataset.

My workaround was to catch the LibsndfileError and generate a dummy audio with an unsual sample rate to filter it later. However returning `None` seems better. 

`try:
    array, sampling_rate = sf.read(file)
except sf.LibsndfileError:
    print("bad file")
    array = np.array([0.0])
    sampling_rate = 99.000`
Issue: IndexError Not Solving -> IndexError: Invalid key: ?? is out of bounds for size 0 or ??

Comment:
> Looks related to https://discuss.huggingface.co/t/indexerror-invalid-key-16-is-out-of-bounds-for-size-0/14298/4?u=lhoestq

The problem has not been solved, I have tried this before, but the problem is the same
Issue: IndexError Not Solving -> IndexError: Invalid key: ?? is out of bounds for size 0 or ??

Comment:
data = data["train"].shuffle().map(generate_and_tokenize_prompt, batched = False) # change this line to -

data["train"] = data["train"].shuffle().map(generate_and_tokenize_prompt, batched = False)
After doing this change you code should run fine.
Issue: IndexError Not Solving -> IndexError: Invalid key: ?? is out of bounds for size 0 or ??

Comment:
> > 
> 
> @syngokhan did u solve it? I am desperate

refer to my earlier comment. you will find the solution.
Issue: Failing to upload dataset to the hub

Comment:
Hi ! Feel free to re-run your code later, it will resume automatically where you left
Issue: Failing to upload dataset to the hub

Comment:
Alternatively you can save your dataset in parquet files locally and upload them to the hub manually

```python
from tqdm import tqdm
num_shards = 60
for index in tqdm(range(num_shards)):
    ds.shard(num_shards=num_shards, index=index, contiguous=True).to_parquet(f"{index:05d}.parquet")
````
Issue: Load Data Sets Too Slow In Train Seq2seq Model

Comment:
already did，but not useful for step Generating train split，it works in step "Resolving data files" & "Downloading data files"
Issue: Load Data Sets Too Slow In Train Seq2seq Model

Comment:
We need more info about the issue to provide help. 

Can you interrupt the process (with `num_proc=None`) after the `load_dataset` call when the slowdown occurs? So we can know what part of the code is causing it.

The `audiofolder` \ `imagefolder` with metadata is not performant for large datasets. Luckily, we can make them much faster if drop the nested metadata files feature (not that useful). I plan to work on this soon.

In the meantime, it's better to use `Dataset.from_generator` (requires replacing the `load_dataset` calls in the transformers script with `Dataset.from_generator`) or write a dataset loading script for large datasets.
Issue: Load Data Sets Too Slow In Train Seq2seq Model

Comment:
Can you interrupt the process (with num_proc=None) after the load_dataset call when the slowdown occurs? So we can know what part of the code is causing it.
（I'll try this operation）
The audiofolder \ imagefolder with metadata is not performant for large datasets. Luckily, we can make them much faster if drop the nested metadata files feature (not that useful). I plan to work on this soon.
(My data is indeed a bit large, exceeding 10000 hours of audio data. Looking forward to your improvement work very much)

In the meantime, it's better to use Dataset.from_generator (requires replacing the load_dataset calls in the transformers script with Dataset.from_generator) or write a dataset loading script for large datasets.
（I want to use Dataset.from_generator instead of  load_dataset ，where can i found sample code to load audio&label dataset， I was to do asr task）
Issue: Load Data Sets Too Slow In Train Seq2seq Model

Comment:
Can you interrupt the process (with num_proc=None) after the load_dataset call when the slowdown occurs? So we can know what part of the code is causing it.
================================================================================
Here is the log：
[load_dataset.log](https://github.com/huggingface/datasets/files/12169362/load_dataset.log)
（The larger my training data, the slower it loads）
![image](https://github.com/huggingface/datasets/assets/19569322/381b73e4-0a54-4240-b95e-cb8164584047)
Issue: Load Data Sets Too Slow In Train Seq2seq Model

Comment:
In the meantime, it's better to use Dataset.from_generator (requires replacing the load_dataset calls in the transformers script with Dataset.from_generator) or write a dataset loading script for large datasets.
================================================================================
I tried ‘Dataset. from_generator’ implements data loading, but the testing results show no improvement
Issue: Load Data Sets Too Slow In Train Seq2seq Model

Comment:
I have already solved this problem, referring to #5990 : read audio frist, then use data_generator to change format .
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
Hi @AntreasAntoniou , sorry to know you are facing this issue. To help debugging it, could you tell me:
- What is the total dataset size?
- Is it always failing on the same shard or is the hanging problem happening randomly?
- Were you able to save the dataset as parquet locally? This would help us determine if the problem comes from the upload or the file generation.

I'm cc-ing @lhoestq who might have some insights from a `datasets` perspective.
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
One trick that can also help is to check the traceback when you kill your python process: it will show where in the code it was hanging
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
@Wauplin 

>What is the total dataset size?

There are three variants, and the random hanging happens on all three. The sizes are 2TB, 1TB, and 200GB. 

>Is it always failing on the same shard or is the hanging problem happening randomly?

It seems to be very much random, as restarting can help move past the previous hang, only to find a new one, or not. 

>Were you able to save the dataset as parquet locally? This would help us determine if the problem comes from the upload or the file generation.

Yes. The dataset seems to be locally stored as parquet.
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
Hmm it looks like an issue with TQDM lock. Maybe you can try updating TQDM ?
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
I am using the latest version of tqdm

```
⬢ [Docker] ❯ pip install tqdm --upgrade
Requirement already satisfied: tqdm in /opt/conda/envs/main/lib/python3.10/site-packages (4.65.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
```
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
I'm on my phone so can't help that much. What I'd advice to do is to [save_to_disk](https://huggingface.co/docs/datasets/package_reference/main_classes#save_to_disk) if it's not already done and then upload the files/folder to the Hub separately. You can find what you need in the [upload guide](https://huggingface.co/docs/huggingface_hub/guides/upload). It might not help finding the exact issue for now but at least it can unblock you.
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
In your last stacktrace it interrupted while embedding external content - in case your dataset in made of images or audio files that live on your disk. Is it the case ?
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
It's maybe related to https://github.com/apache/arrow/issues/34455: are you using ArrayND features ?

Also what's your `pyarrow` version ? Could you try updating to >= 12.0.1 ?
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
I was using pyarrow == 12.0.0

I am not explicitly using ArrayND features, unless the hub API automatically converts my files to such.
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
You can also try to reduce the `max_shard_size` - Sometimes parquet has a hard time working with data bigger than 2GB
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
So, updating the pyarrow seems to help. It can still throw errors here and there but I can retry when that happens. It's better than hanging. 

However, I am a bit confused about something. I have uploaded my datasets, but while earlier I could see all three sets, now I can only see 1. What's going on? 
https://huggingface.co/datasets/Antreas/TALI-base

I have seen this happen before as well, so I deleted and reuploaded, but this dataset is way too large for me to do this.
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
It's a bug on our side, I'll update the dataset viewer ;)

Thanks for reporting !
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
Apparently this happened because of bad modifications in the README.md split metadata.

I fixed them in this PR: https://huggingface.co/datasets/Antreas/TALI-base/discussions/1
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
@lhoestq It's a bit odd that when uploading a dataset, one set at a time "train", "val", "test", the push_to_hub function overwrites the readme and removes differently named sets from previous commits. i.e., you push "val", all is well. Then you push "test", and the "val" entry disappears from the readme, while the data remain intact.
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
> @lhoestq It's a bit odd that when uploading a dataset, one set at a time "train", "val", "test", the push_to_hub function overwrites the readme and removes differently named sets from previous commits. i.e., you push "val", all is well. Then you push "test", and the "val" entry disappears from the readme, while the data remain intact.

Hmm this shouldn't happen. What code did you run exactly ? Using which version of `datasets` ?
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
> I have a while loop that forces retries, but it seems that the progress itself is randomly getting lost as well. Any ideas on how to improve this? It has been blocking me for way too long.

Could you also print the cause of the error (`e.__cause__`) ? Or show the full stack trace when the error happens ?
This would give more details about why it failed and would help investigate.
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
> Should I build the parquet manually and then push manually as well? If I do things manually, how can I ensure my dataset works properly with "stream=True"?

Parquet is supported out of the box ^^

If you want to make sure it works as expected you can try locally first:
```python
ds = load_dataset("path/to/local", streaming=True)
```
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
@lhoestq @AntreasAntoniou I transferred this issue to the `datasets` repository as the questions and answers are more related to this repo. Hope it can help other users find the bug and fixes more easily (like updating [tqdm](https://github.com/huggingface/datasets/issues/5990#issuecomment-1607120204) and [pyarrow](https://github.com/huggingface/datasets/issues/5990#issuecomment-1607120278) or [setting a lower `max_shard_size`](https://github.com/huggingface/datasets/issues/5990#issuecomment-1607120328)).

~For the initial "pushing large dataset consistently hangs"-issue, I still think it's best to try to `save_to_disk` first and then upload it manually/with a script (see [upload_folder](https://huggingface.co/docs/huggingface_hub/guides/upload#upload-a-folder)). It's not the most satisfying solution but at least it would confirm from where the problem comes from.~

**EDIT:** removed suggestion about saving to disk first (see https://github.com/huggingface/datasets/issues/5990#issuecomment-1607186914).
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
> @lhoestq @AntreasAntoniou I transferred this issue to the datasets repository as the questions and answers are more related to this repo. Hope it can help other users find the bug and fixes more easily (like updating https://github.com/huggingface/datasets/issues/5990#issuecomment-1607120204 and https://github.com/huggingface/datasets/issues/5990#issuecomment-1607120278 or https://github.com/huggingface/datasets/issues/5990#issuecomment-1607120328).

thanks :)

> For the initial "pushing large dataset consistently hangs"-issue, I still think it's best to try to save_to_disk first and then upload it manually/with a script (see [upload_folder](https://huggingface.co/docs/huggingface_hub/guides/upload#upload-a-folder)). It's not the most satisfying solution but at least it would confirm from where the problem comes from.

As I've already said in other discussions, I would not recommend pushing files saved with `save_to_disk` to the Hub but save to parquet shards and upload them instead. The Hub does not support datasets saved with `save_to_disk`, which is meant for disk only.