SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Failure to hash function when using .map() 

Comment:
@mengban I cannot reproduce the issue even with these versions installed. It would help if you could provide info about your system and the `pip list` output.
Issue: Failure to hash function when using .map() 

Comment:
@mariosasko Please take a look at this
```python
from typing import Any
from datasets import Dataset
import tiktoken

dataset = Dataset.from_list([{"n": str(i)} for i in range(20)])
enc = tiktoken.get_encoding("gpt2")


class A:
    tokenizer = enc #tiktoken.get_encoding("gpt2")

    def __call__(self, example) -> Any:
        ids = self.tokenizer.encode(example["n"])
        example["len"] = len(ids)
        return example

a = A()

def process(example):
    ids = a.tokenizer.encode(example["n"])
    example["len"] = len(ids)
    return example

# success
tokenized = dataset.map(process, desc="tiktoken", num_proc=2)

# raise TypeError: cannot pickle 'builtins.CoreBPE' object
tokenized = dataset.map(a, desc="tiktoken", num_proc=2)
```

pip list
```
datasets                      2.14.4
tiktoken                      0.4.0
```
Issue: Failure to hash function when using .map() 

Comment:
Thanks @maxwellzh! Our `Hasher` works with this snippet, but the problem is running multiprocessing with a non-serializable `tiktoken.Encoding` object.

Inserting the following code before the `map` should fix this:
```python
import copyreg

def pickle_Encoding(enc):
  return (functools.partial(tiktoken.core.Encoding, enc.name, pat_str=enc._pat_str, mergeable_ranks=enc._mergeable_ranks, special_tokens=enc._special_tokens), ())

copyreg.pickle(tiktoken.core.Encoding, pickle_Encoding)
```

But the best fix would be implementing `__reduce__` for `tiktoken.Encoding` or `tiktoken.CoreBPE`. If I find time, I'll try to fix this in the `tiktoken` repo.
Issue: map() breaks at certain dataset size when using Array3D

Comment:
Hi! This code works for me locally or in Colab. What's the output of `python -c "import pyarrow as pa; print(pa.__version__)"` when you run it inside your environment?
Issue: map() breaks at certain dataset size when using Array3D

Comment:
Thanks for looking into this!
The output of  `python -c "import pyarrow as pa; print(pa.__version__)"` is:
```
11.0.0
```

I did the following to setup the environment:
```
conda create -n datasets_debug python=3.9
conda activate datasets_debug
pip install datasets==2.9.0
```

I just tested this on another machine (Ubuntu 18.04.6 LTS) with the same result as mentioned in the issue description.
Issue: train_test_split in arrow_dataset does not ensure to keep single classes in test set

Comment:
Hi! You can get this behavior by specifying `stratify_by_column="label"` in `train_test_split`.

This is the full example:
```python
import numpy as np
from datasets import Dataset, ClassLabel

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.cast_column("label", ClassLabel(num_classes=3))
    data_set = data_set.train_test_split(test_size=0.5, stratify_by_column="label")
    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test) 
```
Issue: TypeError: Couldn't cast array of type string to null

Comment:
Thanks for reporting, @TJ-Solergibert.

We cannot access your Colab notebook: `There was an error loading this notebook. Ensure that the file is accessible and try again.`
Could you please make it publicly accessible?
Issue: TypeError: Couldn't cast array of type string to null

Comment:
I swear it's public, I've checked the settings and I've been able to open it in incognito mode.

Notebook: https://colab.research.google.com/drive/1JCrS7FlGfu_kFqChMrwKZ_bpabnIMqbP?usp=sharing

Anyway, this is the code to reproduce the error:

```python3
from datasets import ClassLabel
from datasets import load_dataset

europarl_ds = load_dataset("tj-solergibert/Europarl-ST")

source_lang = "nl"
languages = list(europarl_ds["train"][0]["transcriptions"].keys())
ClassLabels = ClassLabel(num_classes = len(languages), names = languages)

def map_label2id(example):
    example['dest_lang'] = ClassLabels.str2int(example['dest_lang'])
    return example

def unfold_transcriptions(example):
  for lang in languages:
    example[lang] = example["transcriptions"][lang]
  return example

def unroll(batch, src_lang, dest_langs):
  source_t, dest_t, dest_l = [], [], []
  for lang in dest_langs: 
    source_t += batch[src_lang]
    dest_t += batch[lang]
    dest_l += [lang]
  return_dict = {"source_text": source_t, "dest_text": dest_t, "dest_lang": dest_l}
  return return_dict

def preprocess_split(ds_split, src_lang):
  dest_langs = [x for x in languages if x != src_lang]

  ds_split = ds_split.map(unroll, fn_kwargs= {"src_lang": src_lang, "dest_langs": dest_langs}, batched = True, batch_size = 1, remove_columns= list(languages))
  ds_split = ds_split.filter(lambda x: x["source_text"] != None and x["dest_text"] != None) # Remove incomplete translations
  ds_split = ds_split.filter(lambda x: x["source_text"] != "None" and x["dest_text"] != "None")
  ds_split = ds_split.map(map_label2id) 
  ds_split = ds_split.cast_column("dest_lang", ClassLabels)
  return ds_split

def reset_cortas(example):
  for lang in languages:
    if isinstance(example[lang], str):
      if example[lang].isnumeric () or len(example[lang]) <= 5:
        example[lang] = "None"
  return example

def clean_dataset(dataset):
  # Remove columns
  dataset = dataset.remove_columns(["original_speech", "original_language", "audio_path", "segment_start", "segment_end"])
  # Unfold
  dataset = dataset.map(unfold_transcriptions, remove_columns = ["transcriptions"])
  dataset = dataset.map(reset_cortas)
  return dataset

processed_europarl = clean_dataset(europarl_ds["test"])
new_train_ds = preprocess_split(processed_europarl, 'nl')
```
Issue: TypeError: Couldn't cast array of type string to null

Comment:
Thanks, @TJ-Solergibert. I can access your notebook now. Maybe it was just a temporary issue.

At first sight, it seems something related to your data: maybe some of the examples do not have all the transcriptions for all the languages. Then, some of them are null when unrolled. And when trying to concatenate with the other rows containing strings, the cast issue is raised (the arrays to be concatenated have different types).

Do you think this could be the case?
Issue: TypeError: Couldn't cast array of type string to null

Comment:
You can fix this issue by forcing the cast of None to str by hand:
- If you replace this line:
```python
source_t += batch[src_lang]
```
- With this line (because the batch size is 1):
```python
source_t += [str(batch[src_lang][0])]
```
- Or with this line (if the batch size were larger than 1):
```python
source_t += [str(text) for text in batch[src_lang]]
```
Issue: TypeError: Couldn't cast array of type string to null

Comment:
Problem solved! Thanks @albertvillanova, now I have even increased the batch size and it's crazy fast :rocket: !
Issue: `with_format("numpy")` silently downcasts float64 to float32 features

Comment:
Hi! This behavior stems from these lines:

https://github.com/huggingface/datasets/blob/b065547654efa0ec633cf373ac1512884c68b2e1/src/datasets/formatting/np_formatter.py#L45-L46

I agree we should preserve the original type whenever possible and downcast explicitly with a warning.

@lhoestq Do you remember why we need this "default dtype" logic in our formatters?
Issue: `with_format("numpy")` silently downcasts float64 to float32 features

Comment:
I was also wondering why the default type logic is needed. Me just deleting it is probably too naive of a solution.
Issue: `with_format("numpy")` silently downcasts float64 to float32 features

Comment:
Hmm I think the idea was to end up with the usual default precision for deep learning models - no matter how the data was stored or where it comes from.

For example in NLP we store tokens using an optimized low precision to save disk space, but when we set the format to `torch` we actually need to get `int64`. Although the need for a default for integers also comes from numpy not returning the same integer precision depending on your machine. Finally I guess we added a default for floats as well for consistency.

I'm a bit embarrassed by this though, as a user I'd have expected to get the same precision indeed as well and get a zero copy view.
Issue: `with_format("numpy")` silently downcasts float64 to float32 features

Comment:
Unfortunately removing it for integers is a breaking change for most `transformers` + `datasets` users for NLP (which is a common case). Removing it for floats is a breaking change for `transformers` + `datasets` for ASR as well. And it also is a breaking change for the other users relying on this behavior.

Therefore I think that the only short term solution is for the user to provide `dtype=` manually and document better this behavior. We could also extend `dtype` to accept a value that means "return the same dtype as the underlying storage" and make it easier to do zero copy.
Issue: `with_format("numpy")` silently downcasts float64 to float32 features

Comment:
@lhoestq It should be fine to remove this conversion in Datasets 3.0, no? For now, we can warn the user (with a log message) about the future change when the default type is changed.
Issue: `with_format("numpy")` silently downcasts float64 to float32 features

Comment:
Let's see with the transformers team if it sounds reasonable ? We'd have to fix multiple example scripts though.

If it's not ok we can also explore keeping this behavior only for tokens and audio data.
Issue: `with_format("numpy")` silently downcasts float64 to float32 features

Comment:
IMO being coupled with Transformers can lead to unexpected behavior when one tries to use our lib without pairing it with Transformers, so I think it's still important to "fix" this, even if it means we will need to update Transformers' example scripts afterward.
Issue: `with_format("numpy")` silently downcasts float64 to float32 features

Comment:
For others that run into the same issue: A temporary workaround for me is this:
```python
def numpy_transform(batch):
    return {key: np.asarray(val) for key, val in batch.items()}

dataset = dataset.with_transform(numpy_transform)
```
Issue: Improve inconsistency of `Dataset.map` interface for `load_from_cache_file`

Comment:
Hi, thanks for noticing this! We can't just remove the cache control as this allows us to control where the arrow files generated by the ops are written (cached on disk if enabled or a temporary directory if disabled). The right way to address this inconsistency would be by having `load_from_cache_file=None` by default everywhere.
Issue: Improve inconsistency of `Dataset.map` interface for `load_from_cache_file`

Comment:
Hi! Yes, this seems more plausible. I can implement that. One last thing is the type annotation `load_from_cache_file: bool = None`. Which I then would change to `load_from_cache_file: Optional[bool] = None`.
Issue: Some functions use a param named `type` shouldn't that be avoided since it's a Python reserved name?

Comment:
Hi! Let's not do this - renaming it would be a breaking change, and going through the deprecation cycle is only worth it if it improves user experience.
Issue: Some functions use a param named `type` shouldn't that be avoided since it's a Python reserved name?

Comment:
Hi @mariosasko, ok it makes sense. Anyway, don't you think it's worth it at some point to start a deprecation cycle e.g. `fs` in `load_from_disk`? It doesn't affect user experience but it's for sure a bad practice IMO, but's up to you 😄 Feel free to close this issue otherwise!
Issue: Some functions use a param named `type` shouldn't that be avoided since it's a Python reserved name?

Comment:
I don't think deprecating a param name in this particular instance is worth the hassle, so I'm closing the issue 🙂.
Issue: Creating a dummy dataset from a bigger one

Comment:
Update `datasets` or downgrade `huggingface-hub` ;)

The `huggingface-hub` lib did a breaking change a few months ago, and you're using an old version of `datasets` that does't support it
Issue: Creating a dummy dataset from a bigger one

Comment:
Getting same error with latest versions.


```shell
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[99], line 1
----> 1 dataset.push_to_hub("mirfan899/kids_phoneme_asr")

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:3538, in Dataset.push_to_hub(self, repo_id, split, private, token, branch, shard_size, embed_external_files)
   3493 def push_to_hub(
   3494     self,
   3495     repo_id: str,
   (...)
   3501     embed_external_files: bool = True,
   3502 ):
   3503     """Pushes the dataset to the hub.
   3504     The dataset is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
   3505 
   (...)
   3536     ```
   3537     """
-> 3538     repo_id, split, uploaded_size, dataset_nbytes = self._push_parquet_shards_to_hub(
   3539         repo_id=repo_id,
   3540         split=split,
   3541         private=private,
   3542         token=token,
   3543         branch=branch,
   3544         shard_size=shard_size,
   3545         embed_external_files=embed_external_files,
   3546     )
   3547     organization, dataset_name = repo_id.split("/")
   3548     info_to_dump = self.info.copy()

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:3474, in Dataset._push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, shard_size, embed_external_files)
   3472     shard.to_parquet(buffer)
   3473     uploaded_size += buffer.tell()
-> 3474     _retry(
   3475         api.upload_file,
   3476         func_kwargs=dict(
   3477             path_or_fileobj=buffer.getvalue(),
   3478             path_in_repo=path_in_repo(index),
   3479             repo_id=repo_id,
   3480             token=token,
   3481             repo_type="dataset",
   3482             revision=branch,
   3483             identical_ok=True,
   3484         ),
   3485         exceptions=HTTPError,
   3486         status_codes=[504],
   3487         base_wait_time=2.0,
   3488         max_retries=5,
   3489         max_wait_time=20.0,
   3490     )
   3491 return repo_id, split, uploaded_size, dataset_nbytes

File /opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py:330, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    328 while True:
    329     try:
--> 330         return func(*func_args, **func_kwargs)
    331     except exceptions as err:
    332         if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):

File /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:120, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    117 if check_use_auth_token:
    118     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 120 return fn(*args, **kwargs)

TypeError: HfApi.upload_file() got an unexpected keyword argument 'identical_ok'
```
Issue: Saving a dataset after setting format to torch doesn't work, but only if filtering

Comment:
Hey, I'm a research engineer working on language modelling wanting to contribute to open source. I was wondering if I could give it a shot?
Issue: IterableDataset and Dataset return different batch sizes when using Trainer with multiple GPUs

Comment:
Hi ! `datasets` doesn't do batching - the PyTorch DataLoader does and is created by the `Trainer`. Do you pass other arguments to training_args with respect to data loading ?

Also we recently released `.to_iterable_dataset` that does pretty much what you implemented, but using contiguous shards to get a better speed:
```python
if use_iterable_dataset:
    num_shards = 100
    dataset = dataset.to_iterable_dataset(num_shards=num_shards)
```
Issue: IterableDataset and Dataset return different batch sizes when using Trainer with multiple GPUs

Comment:
This is the full set of training args passed. No training args were changed when switching dataset types.

```python
training_args = TrainingArguments(
    output_dir="./checkpoints",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=256,
    save_steps=2000,
    save_total_limit=4,
    prediction_loss_only=True,
    report_to='none',
    gradient_accumulation_steps=6,
    fp16=True,
    max_steps=60000,
    lr_scheduler_type='linear',
    warmup_ratio=0.1,
    logging_steps=100,
    weight_decay=0.01,
    adam_beta1=0.9,
    adam_beta2=0.98,
    adam_epsilon=1e-6,
    learning_rate=1e-4
)
```
Issue: IterableDataset and Dataset return different batch sizes when using Trainer with multiple GPUs

Comment:
Makes sense. Given that it's a `transformers` issue and already being tracked, I'll close this out.
Issue: PyTorch BatchSampler still loads from Dataset one-by-one

Comment:
This change seems to come from a few months ago in the PyTorch side. That's good news and it means we may not need to pass a batch_sampler as soon as we add `Dataset.__getitems__` to get the optimal speed :)

Thanks for reporting ! Would you like to open a PR to add `__getitems__` and remove this outdated documentation ?
Issue: PyTorch BatchSampler still loads from Dataset one-by-one

Comment:
Yeah I figured this was the sort of thing that probably once worked. I can confirm that you no longer need the batch sampler, just `batch_size=n` in the `DataLoader`.

I'll pass on the PR, I'm flat out right now, sorry.
Issue: `load_dataset` has ~4 seconds of overhead for cached data

Comment:
Hi ! To skip the verification step that checks if newer data exist, you can enable offline mode with `HF_DATASETS_OFFLINE=1`.

Although I agree this step should be much faster for datasets hosted on the HF Hub - we could just compare the commit hash from the local data and the remote git repository. We're not been leveraging the git commit hashes, since the library was built before we even had git repositories for each dataset on HF.
Issue: TypeError: 'bool' object is not iterable when filtering a datasets.arrow_dataset.Dataset

Comment:
Hi! Instead of a single boolean, your filter function should return an iterable (of booleans) in the batched mode like so:
```python
train_dataset = train_dataset.filter(
            function=lambda batch: [image is not None for image in batch["image"]], 
            batched=True,
            batch_size=10)
```

PS: You can make this operation much faster by operating directly on the arrow data to skip the decoding part:
```python
train_dataset = train_dataset.with_format("arrow")
train_dataset = train_dataset.filter(
            function=lambda table: table["image"].is_valid().to_pylist(), 
            batched=True,
            batch_size=100)
train_dataset = train_dataset.with_format(None)
```
Issue: Add a `reduce` method

Comment:
Hi! Sure, feel free to open a PR, so we can see the API you have in mind.
Issue: to_tf_dataset fails with datetime UTC columns even if not included in columns argument

Comment:
Hi! This is indeed a bug in our zero-copy logic.

To fix it, instead of the line:
https://github.com/huggingface/datasets/blob/7cfac43b980ab9e4a69c2328f085770996323005/src/datasets/features/features.py#L702

we should have:
```python
return pa.types.is_primitive(pa_type) and not (pa.types.is_boolean(pa_type) or pa.types.is_temporal(pa_type))
```
Issue: Update audio installation doc page

Comment:
Totally agree, the docs should be in sync with our code.

Indeed to avoid confusing users, I think we should have updated the docs at the same time as this PR:
- #5167
Issue: Update audio installation doc page

Comment:
No, @polinaeterna, nothing to be sorry about.

My comment was for all of us datasets team, as a reminder: when making a PR, but also when reviewing some other's PR, we should not forget to update the corresponding docstring and doc pages. It is something we can improve if we help each other in reminding about it... :hugs:
Issue: Update audio installation doc page

Comment:
@polinaeterna I think we can close this issue now as we no longer use `torchaudio` for decoding.
Issue: Push_to_hub in a pull request

Comment:
Assigned to myself and will get to it in the next week, but if someone finds this issue annoying and wants to submit a PR before I do, just ping me here and I'll reassign :).
Issue: Error loading MP3 files from CommonVoice

Comment:
Hi @kradonneoh, thanks for reporting.

Please note that to work with audio datasets (and specifically with MP3 files) we have detailed installation instructions in our docs: https://huggingface.co/docs/datasets/installation#audio
- one of the requirements is torchaudio<0.12.0

Let us know if the problem persists after having followed them.
Issue: Error loading MP3 files from CommonVoice

Comment:
I saw that and have followed it (hence the Expected Behavior section of the bug report). 

Is there no intention of updating to the latest version? It does limit the version of `torch` I can use, which isn’t ideal.
Issue: Error loading MP3 files from CommonVoice

Comment:
@kradonneoh hey! actually with `ffmpeg4` loading of mp3 files should work, so this is a not expected behavior and we need to investigate it. It works on my side with `torchaudio==0.13` and `ffmpeg==4.2.7`. Which `torchaudio` version do you use?

`datasets` should support decoding of mp3 files with `torchaudio` when its version is `>0.12` but as you noted it requires `ffmpeg>4`, we need to fix this in the documentation, thank you for pointing to this! 

But according to your traceback it seems that it tries to use [`libsndfile`](https://github.com/libsndfile/libsndfile) backend for mp3 decoding. And `libsndfile` library supports mp3 decoding starting from version 1.1.0 which on Linux has to be compiled from source for now afaik. 

fyi - we are aiming at getting rid of `torchaudio` dependency at all by the next major library release in favor of `libsndfile` too.
Issue: Incorrect filepath for dill module

Comment:
Hi! The correct path is still `dill._dill.XXXX` in the latest release. What do you get when you run  `python -c "import dill; print(dill.__version__)"` in your environment?
Issue: Incorrect filepath for dill module

Comment:
`0.3.6` I feel like that's bad news, because it's probably not the issue.

My mistake, about the wrong path guess.  I think I didn't notice that the first `dill` in the path isn't supposed to be included in the path specification in python.
<img width="146" alt="Screen Shot 2023-01-31 at 12 58 32 PM" src="https://user-images.githubusercontent.com/35349273/215844209-74af6a8f-9bff-4c75-9495-44c658c8e9f7.png">
Issue: Incorrect filepath for dill module

Comment:
Hi, @avivbrokman, this issue you report appeared only with old versions of dill. See:
- #288

Are you sure you are in the right Python environment?
- Please note that Jupyter (where I guess you get the error) may have multiple execution backends (IPython kernels) that might be different from the Python environment your are using to get the dill version
  - Have you run `import dill; print(dill.__version__)` in the same Jupyter/IPython that you were using when you got the error while executing `import datasets`?