SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Downloading via Apache Pipeline, client cancelled (org.apache.beam.vendor.grpc.v1p43p2.io.grpc.StatusRuntimeException)

Comment:
Hi @dan-the-meme-man, thanks for reporting.

We are investigating a similar issue but with Beam+Dataflow (instead of Beam+Flink): 
- #4525

In order to go deeper into the root cause, we need as much information as possible: logs from the main process + logs from the workers are very informative.

In the case of the issue with Beam+Dataflow, the logs from the workers report an out of memory issue.
Issue: Downloading via Apache Pipeline, client cancelled (org.apache.beam.vendor.grpc.v1p43p2.io.grpc.StatusRuntimeException)

Comment:
As I continued working on this today, I came to suspect that it is in fact an out of memory issue - I have a few more notebooks that I've left running, and if they produce the same error, I will try to get the logs. In the meantime, if there's any chance that there is a repo out there with those three languages already as .arrow files, or if you know about how much memory would be needed to actually download those sets, please let me know!
Issue: Datasets method `.map` not hashing

Comment:
Didn't realize it's a bug when I asked the question yesterday! Free free to post an answer if you are sure the cause has been addressed.

https://stackoverflow.com/questions/72664827/can-pickle-dill-foo-but-not-lambda-x-foox
Issue: Failure to  hash `dataclasses` - results in functions that cannot be hashed or cached in `.map`

Comment:
I think this has been fixed by #4516, let me know if you encounter this again :)

I re-ran your code in 3.7 and 3.9 and it works fine
Issue: cast_storage method from datasets.features

Comment:
Hi! We've recently added a check to the `ClassLabel` type to ensure the values are in the valid label range `-1, 0, ..., num_classes-1` (-1 is used for missing values). The error in your case happens only if the `labels` column is of type `Sequence(ClassLabel(...))` before the `map` call and can be avoided by calling `dataset = dataset.cast_column("labels", Sequence(Value("int")))` beforehand. The token-classification examples in Transformers introduce a new `labels` column, so their type is also `Sequence(Value("int"))`, which doesn't lead to an error as this type unbounded.
Issue: How to let `load_dataset` return a `Dataset` instead of `DatasetDict` in customized loading script

Comment:
Hi @liyucheng09.

Users can pass the `split` parameter to `load_dataset`. For example, if your split name is "train",
```python
ds = load_dataset("dataset_name", split="train")
```
will return a `Dataset` instance.
Issue: Failure to hash (and cache) a `.map(...)` (almost always) - using this method can produce incorrect results

Comment:
Important info:

As hashes are generated randomly for functions, it leads to **false identifying some results as already hashed** (mapping function is not executed after a method update) when there's a `pytorch_lightning.seed_everything(123)`
Issue: Failure to hash (and cache) a `.map(...)` (almost always) - using this method can produce incorrect results

Comment:
Hi ! Thanks for reporting. This bug seems to appear in python 3.9 using dill 3.5.1

As a workaround you can use an older version of dill:
```
pip install "dill<0.3.5"
```
Issue: Failure to hash (and cache) a `.map(...)` (almost always) - using this method can produce incorrect results

Comment:
installing `dill<0.3.5` after installing `datasets` by pip results in dependency conflict with the version required for `multiprocess`. It can be solved by installing `pip install datasets "dill<0.3.5"` (simultaneously) on a clean environment
Issue: Failure to hash (and cache) a `.map(...)` (almost always) - using this method can produce incorrect results

Comment:
This has been fixed in https://github.com/huggingface/datasets/pull/4516, we will do a new release soon to include the fix :)
Issue: Can you please add the Stanford dog dataset?

Comment:
would you like to give it a try, @dgrnd4? (maybe with the help of the dataset author?)
Issue: Can you please add the Stanford dog dataset?

Comment:
@julien-c i am sorry but I have no idea about how it works: can I add the dataset by myself, following "instructions to add a new dataset"?
Can I add a dataset even if it's not mine? (it's public in the link that I wrote on the post)
Issue: Can you please add the Stanford dog dataset?

Comment:
Hi! The [ADD NEW DATASET](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) instructions are indeed the best place to start. It's also perfectly fine to add a dataset if it's public, even if it's not yours. Let me know if you need some additional pointers.
Issue: Can you please add the Stanford dog dataset?

Comment:
@khushmeeet this is the [link](https://huggingface.co/datasets/dgrnd4/stanford_dog_dataset) where I added the dataset already. If you can I would ask you to do this:
1) The dataset it's all in TRAINING SET: can you please divide it in Training,Test and Validation Set? If you can for each class, take the 80% for the Training set and the 10% for Test and 10% Validation
2) The images has different size, can you please resize all the images in 224,224,3? Look even at the last dimension "3" because some images has dimension 4!

Thank you!!
Issue: Can you please add the Stanford dog dataset?

Comment:
Hi @khushmeeet! Thanks for the interest. You can self-assign the issue by commenting `#self-assign` on it. 

Also, I think we can skip @dgrnd4's steps as we try to avoid any custom processing on top of raw data. One can later copy the script and override `_post_process` in it to perform such processing on the generated dataset.
Issue: Can you please add the Stanford dog dataset?

Comment:
Thanks @mariosasko 

@dgrnd4 As dataset is there on Hub, and preprocessing is not recommended. I am not sure if there is any other task to do. However, I can't seem to find relevant `.py` files for this dataset in GitHub repo.
Issue: Can you please add the Stanford dog dataset?

Comment:
@khushmeeet @mariosasko The point is that the images must be processed and must have the same size in order to can be used for things for example "Training".
Issue: Can you please add the Stanford dog dataset?

Comment:
@dgrnd4 Yes, but this can be done after loading (`map` to resize images and `train_test_split` to create extra splits)

@khushmeeet The linked version is implemented as a no-code dataset and is generated directly from the ZIP archive, but our "GitHub" datasets (these are datasets without a user/org namespace on the Hub) need a generation script, and you can find one [here](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/image_classification/stanford_dogs.py). `datasets` started as a fork of TFDS, so we share similar script structure, which makes it trivial to adapt it.
Issue: Can you please add the Stanford dog dataset?

Comment:
@mariosasko The point is that if I use something like this:
x_train, x_test = train_test_split(dataset, test_size=0.1) 

to get Train 90% and Test 10%, and then to get the Validation Set (10% of the whole 100%):

```
train_ratio = 0.80
validation_ratio = 0.10
test_ratio = 0.10

x_train, x_test, y_train, y_test = train_test_split(dataX, dataY, test_size=1 - train_ratio)
x_val, x_test, y_val, y_test = train_test_split(x_test, y_test, test_size=test_ratio/(test_ratio + validation_ratio)) 

```

The point is that the structure of the data is:
```
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 20580
    })
})

```

So how to extract images and labels?

EDIT --> Split of the dataset in Train-Test-Validation:
```
import datasets
from datasets.dataset_dict import DatasetDict
from datasets import Dataset

percentage_divison_test = int(len(dataset['train'])/100 *10)       # 10%  --> 2058 
percentage_divison_validation = int(len(dataset['train'])/100 *20) # 20%  --> 4116

dataset_ = datasets.DatasetDict({"train": Dataset.from_dict({

                                  'image':  dataset['train'][0 : len(dataset['train']) ]['image'],    
                                  'labels': dataset['train'][0 : len(dataset['train']) ]['label'] }), 
                                 
                                 "test": Dataset.from_dict({  #20580-4116 (validation) ,20580-2058 (test)
                                  'image':  dataset['train'][len(dataset['train']) - percentage_divison_validation : len(dataset['train']) - percentage_divison_test]['image'], 
                                  'labels': dataset['train'][len(dataset['train']) - percentage_divison_validation : len(dataset['train']) - percentage_divison_test]['label'] }), 
                                 
                                  "validation": Dataset.from_dict({ # 20580-2058 (test)
                                  'image':  dataset['train'][len(dataset['train']) - percentage_divison_test : len(dataset['train'])]['image'], 
                                  'labels': dataset['train'][len(dataset['train']) - percentage_divison_test : len(dataset['train'])]['label'] }), 
                                })
```
Issue: Can you please add the Stanford dog dataset?

Comment:
@mariosasko in order to resize images I'm trying this method: 
```
for i in range(0,len(dataset['train'])): #len(dataset['train'])

  ex = dataset['train'][i] #i
  image = ex['image']
  image = image.convert("RGB") # <class 'PIL.Image.Image'> <PIL.Image.Image image mode=RGB size=500x333 at 0x7F84F1948150>
  image_resized = image.resize(size_to_resize) # <PIL.Image.Image image mode=RGB size=224x224 at 0x7F84F17885D0>

  dataset['train'][i]['image'] = image_resized 
```

Because the DatasetDict is formed by arrows that are immutable, the changing assignment in the last line of code, doesn't work!
Do you have any idea in order to get a valid result?
Issue: Can you please add the Stanford dog dataset?

Comment:
I have raised PR for adding stanford-dog dataset. I have not added any data preprocessing code. Only dataset generation script is there. Let me know any changes required, or anything to add to README.
Issue: Logic bug in arrow_writer?

Comment:
Hi @cccntu you're right, as when `batch_examples={}` the current if-statement won't be triggered as the condition won't be satisfied, I'll prepare a PR to address it as well as add the regression tests so that this issue is handled properly.
Issue: Logic bug in arrow_writer?

Comment:
Hi @alvarobartt ,
Thanks for answering. Do you know when and why an empty batch is passed to this function? This only happened to me when processing with multiple workers, while chunking examples, I think.
Issue: Logic bug in arrow_writer?

Comment:
> Hi @alvarobartt , Thanks for answering. Do you know when and why an empty batch is passed to this function? This only happened to me when processing with multiple workers, while chunking examples, I think.

So it depends on how you're actually chunking the data as if you're not handling empty chunks `batch_examples={}` or `batch_examples=None`, you may end up running into this issue. So you could check the chunks before you actually call `ArrowWriter.write_batch`, but anyway the fix you proposed I think improves the logic of `write_batch` to avoid running into these issues.
Issue: Logic bug in arrow_writer?

Comment:
Thanks, I added a if-print and I found it does return an empty examples in the chunking function that is passed to `.map()`.
Issue: Logic bug in arrow_writer?

Comment:
Hi ! We consider an empty batch to look like this:
```python
empty_batch = {
    "column_1": [],
    "column_2": [],
    ...
}
```

While `{}` corresponds to a batch with no columns.

Therefore calling this code should fail, because the two batches don't have the same columns:
```python
writer.write_batch({"a": [1, 2, 3]})
writer.write_batch({})
```

If you want to write an empty batch, you should do this instead:
```python
writer.write_batch({"a": [1, 2, 3]})
writer.write_batch({"a": []})
```
Issue: Logic bug in arrow_writer?

Comment:
Makes sense, then the if-statement should remain the same or is it better to handle both cases separately using `if not batch_examples or len(next(iter(batch_examples.values()))) == 0: ...`?

Updating the regressions tests with an empty batch formatted as `{"col_1": [], "col_2": []}` instead of `{}` works fine with the current if, and also with the one proposed by @cccntu.
Issue: Logic bug in arrow_writer?

Comment:
> Makes sense, then the if-statement should remain the same or is it better to handle both cases separately using if not batch_examples or len(next(iter(batch_examples.values()))) == 0: ...?

There's a check later in the code that makes sure that the columns are the right ones, so I don't think we need to check for `{}` here

In particular the check `if not batch_examples or len(next(iter(batch_examples.values()))) == 0:` doesn't raise an error while it should, that why the old `if` is fine IMO

> Updating the regressions tests with an empty batch formatted as {"col_1": [], "col_2": []} instead of {} works fine with the current if, and also with the one proposed by @cccntu.

Cool ! If you want you can update your PR to add the regression tests, to make sure that `{"col_1": [], "col_2": []}` works but not `{}`
Issue: Logic bug in arrow_writer?

Comment:
Great thanks for the response! So I'll just add that regression test and remove the current if-statement.
Issue: Logic bug in arrow_writer?

Comment:
Hi @lhoestq ,

Thanks for your explanation. Now I get it that `{}` means the columns are different. But wouldn't it be nice if the code can ignore it, like it ignores `{"a": []}`?


--- 
BTW, 
> There's a check later in the code that makes sure that the columns are the right ones, so I don't think we need to check for {} here

I remember the error happens around here:
https://github.com/huggingface/datasets/blob/88a902d6474fae8d793542d57a4f3b0d187f3c5b/src/datasets/arrow_writer.py#L506-L507
The error says something like `arrays` and `schema` doesn't have the same length. And it's not very clear I passed a `{}`.

edit: actual error message
```
File "site-packages/datasets/arrow_writer.py", line 595, in write_batch
    pa_table = pa.Table.from_arrays(arrays, schema=schema)
  File "pyarrow/table.pxi", line 3557, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 1401, in pyarrow.lib._sanitize_arrays
ValueError: Schema and number of arrays unequal
```
Issue: Logic bug in arrow_writer?

Comment:
> But wouldn't it be nice if the code can ignore it, like it ignores {"a": []}?

I think it would make things confusing because it doesn't follow our definition of a batch: "the columns of a batch = the keys of the dict". It would probably break certain behaviors as well. For example if you remove all the columns of a dataset (using `.remove_colums(...)` or `.map(..., remove_columns=...)`), the writer has to write 0 columns, and currently the only way to tell the writer to do so using `write_batch` is to pass `{}`.

> The error says something like arrays and schema doesn't have the same length. And it's not very clear I passed a {}.

Yea the message can actually be improved indeed, it's definitely not clear. Maybe we can add a line right before the call `pa.Table.from_arrays` to make sure the keys of the batch match the field names of the schema
Issue: WER and CER > 1

Comment:
WER can have values bigger than 1.0, this is expected when there are too many insertions

From [wikipedia](https://en.wikipedia.org/wiki/Word_error_rate):
> Note that since N is the number of words in the reference, the word error rate can be larger than 1.0
Issue: Dataset Viewer issue for Pavithree/test

Comment:
This issue can be resolved according to this post https://stackoverflow.com/questions/70566660/parquet-with-null-columns-on-pyarrow. It looks like first data entry in the json file must not have any null values as pyarrow uses this first file to infer schema for entire dataset.
Issue: Use `torch.nested_tensor` for arrays of varying length in torch formatter

Comment:
Currently, we return a list of Torch tensors if their shapes don't match. If they do, we consolidate them into a single Torch tensor.
Issue: Dataset.map throws pyarrow.lib.ArrowNotImplementedError when converting from list of empty lists

Comment:
Hi @sanderland ! Thanks for reporting :) This is a bug, I opened a PR to fix it. We'll do a new release soon

In the meantime you can fix it by specifying in advance that the "label" are integers:
```python
import numpy as np

ds = Dataset.from_dict(
    {
        "text": ["the lazy dog jumps over the quick fox", "another sentence"],
        "label": [[], []],
    }
)
# explicitly say that the "label" type is int64, even though it contains only null values
ds = ds.cast_column("label", Sequence(Value("int64")))

def mapper(features):
    features['label'] = [
        [0,0,0] for l in features['label']
    ]
    return features

ds_mapped = ds.map(mapper,batched=True)
```
Issue: Bigbench tensorflow GPU dependency

Comment:
Thanks for reporting ! :) cc @andersjohanandreassen can you take a look at this ?

Also @cceyda feel free to open an issue at [BIG-Bench](https://github.com/google/BIG-bench) as well regarding the `AttributeError`
Issue: Bigbench tensorflow GPU dependency

Comment:
I'm on vacation for the next week, so won't be able to do much debugging at the moment. Sorry for the inconvenience.
But I did quickly take a look:

**pypi**:
I managed to reproduce the above error with the pypi version begin out of date. 
The version on `https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz` should be up to date, but it was my understanding that there was some issue with the pypi upload, so I don't even understand why there is a version [on pypi from April 1](https://pypi.org/project/bigbench/0.0.1/). Perhaps @ethansdyer, who's handling the pypi upload, knows the answer to that?

**OOM error**:
But, I'm unable to reproduce the OOM error in a google colab with GPU enabled.
This is what I ran:
```
!pip install bigbench@https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz
!pip install datasets

from datasets import load_dataset
dataset = load_dataset("bigbench","swedish_to_german_proverbs")
``` 
The `swedish_to_german_proverbs`task is only 72 examples, so I don't understand what could be causing the OOM error. Loading the task has no effect on the RAM for me. @cceyda Can you confirm that this does not occur in a [colab](https://colab.research.google.com/)?
If the GPU is somehow causing issues on your system, disabling the GPU from TF might be an option too
```
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
```
Issue: Bigbench tensorflow GPU dependency

Comment:
Solved.
Yes it works on colab, and somehow magically on my machine too now. hmm not sure what was wrong before I had used a fresh venv both times with just the dataloading code, and tried multiple times. (maybe just a wrong tensorflow version got mixed up somehow) The tensorflow call seems to come from the bigbench side anyway.

about bigbench pypi version update, I opened an issue over there https://github.com/google/BIG-bench/issues/846

anyway closing this now. If anyone else has the same problem can re-open.
Issue: Dataset slow during model training

Comment:
Hi ! cc @Rocketknight1 maybe you know better ?

I'm not too familiar with `tf.data.experimental.save`. Note that `datasets` uses memory mapping, so depending on your hardware and the disk you are using you can expect performance differences with a dataset loaded in RAM
Issue: Dataset slow during model training

Comment:
Hi @lehrig, I suspect what's happening here is that our `to_tf_dataset()` method has some performance issues when streaming samples. This is usually not a problem, but they become apparent when streaming a vision dataset into a very small vision model, which will need a lot of sample throughput to saturate the GPU.

When you save a `tf.data.Dataset` with `tf.data.experimental.save`, all of the samples from the dataset (which are, in this case, batches of images), are saved to disk. When you load this saved dataset, you're effectively bypassing `to_tf_dataset()` entirely, which alleviates this performance bottleneck.

`to_tf_dataset()` is something we're actively working on overhauling right now - particularly for image datasets, we want to make it possible to access the underlying images with `tf.data` without going through the current layer of indirection with `Arrow`, which should massively improve simplicity and performance. 

However, if you just want this to work quickly but without needing your save/load hack, my advice would be to simply load the dataset into memory if it's small enough to fit. Since all your samples have the same dimensions, you can do this simply with:

```
dataset = load_from_disk(prep_data_dir)
dataset = dataset.with_format("numpy")
data_in_memory = dataset[:]
```

Then you can simply do something like:

```
model.fit(data_in_memory["pixel_values"], data_in_memory["labels"])
```
Issue: Dataset slow during model training

Comment:
Thanks for the information! 

I have now updated the training code like so:

```
dataset = load_from_disk(prep_data_dir)
train_dataset = dataset["train"][:]
validation_dataset = dataset["dev"][:]

...

model.fit(
    train_dataset["pixel_values"],
    train_dataset["label"],
    epochs=epochs,
    validation_data=(
        validation_dataset["pixel_values"],
        validation_dataset["label"]
    ),
    callbacks=[earlyStopping, mcp_save, reduce_lr_loss]
)
```

- Creating the in-memory dataset is quite quick
- But: There is now a long wait (~4-5 Minutes) before the training starts (why?)
- And: Training times have improved but the very first epoch leaves me wondering why it takes so long (why?)

**Epoch Breakdown:**
- Epoch 1/10
78s 12s/step - loss: 3.1307 - accuracy: 0.0737 - val_loss: 2.2827 - val_accuracy: 0.1273 - lr: 0.0010
- Epoch 2/10
1s 168ms/step - loss: 2.3616 - accuracy: 0.2350 - val_loss: 2.2679 - val_accuracy: 0.2182 - lr: 0.0010
- Epoch 3/10
1s 189ms/step - loss: 2.0221 - accuracy: 0.3180 - val_loss: 2.2670 - val_accuracy: 0.1818 - lr: 0.0010
- Epoch 4/10
0s 67ms/step - loss: 1.8895 - accuracy: 0.3548 - val_loss: 2.2771 - val_accuracy: 0.1273 - lr: 0.0010
- Epoch 5/10
0s 67ms/step - loss: 1.7846 - accuracy: 0.3963 - val_loss: 2.2860 - val_accuracy: 0.1455 - lr: 0.0010
- Epoch 6/10
0s 65ms/step - loss: 1.5946 - accuracy: 0.4516 - val_loss: 2.2938 - val_accuracy: 0.1636 - lr: 0.0010
- Epoch 7/10
0s 63ms/step - loss: 1.4217 - accuracy: 0.5115 - val_loss: 2.2968 - val_accuracy: 0.2182 - lr: 0.0010
- Epoch 8/10
0s 67ms/step - loss: 1.3089 - accuracy: 0.5438 - val_loss: 2.2842 - val_accuracy: 0.2182 - lr: 0.0010
- Epoch 9/10
1s 184ms/step - loss: 1.2480 - accuracy: 0.5806 - val_loss: 2.2652 - val_accuracy: 0.1818 - lr: 0.0010
- Epoch 10/10
0s 65ms/step - loss: 1.2699 - accuracy: 0.5622 - val_loss: 2.2670 - val_accuracy: 0.2000 - lr: 0.0010
Issue: Dataset slow during model training

Comment:
Regarding the new long ~5 min. wait introduced by the in-memory dataset update: this might be causing it? https://datascience.stackexchange.com/questions/33364/why-model-fit-generator-in-keras-is-taking-so-much-time-even-before-picking-the

For now, my save/load hack is still more performant, even though having more boiler-plate code :/
Issue: Dataset slow during model training

Comment:
That 5 minute wait is quite surprising! I don't have a good explanation for why it's happening, but it can't be an issue with `datasets` or `tf.data` because you're just fitting directly on Numpy arrays at this point. All I can suggest is seeing if you can isolate the issue - for example, does fitting on a smaller dataset containing only 10% of the original data reduce the wait? This might indicate the delay is caused by your data being copied or converted somehow. Alternatively, you could try removing things like callbacks and seeing if you could isolate the issue there.