SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: `Dataset.class_encode` should support custom class names

Comment:
Hi @Dref360, thanks a lot for your proposal.

It totally makes sense to have more flexibility when class encoding, I agree.

You could even further customize the class encoding by passing an instance of `ClassLabel` itself (instead of replicating `ClassLabel` instantiation arguments as `Dataset.class_encode_column` arguments).

And the latter made me think of `Dataset.cast_column`...

Maybe better to have some others' opinions @lhoestq @mariosasko
Issue: `Dataset.class_encode` should support custom class names

Comment:
Hi @Dref360! You can use [`Dataset.align_labels_with_mapping`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.Dataset.align_labels_with_mapping) after `Dataset.class_encode_column` to assign a different mapping of labels to ids.

@albertvillanova I'd like to avoid adding more complexity to the API where it's not (absolutely) needed, so I don't think introducing a new param in `Dataset.class_encode_column` is a good idea.
Issue: Know your data for HF hub 

Comment:
Hi @Muhtasham you should take a look at https://huggingface.co/blog/data-measurements-tool and accompanying demo app at https://huggingface.co/spaces/huggingface/data-measurements-tool

We would be interested in your feedback. cc @meg-huggingface @sashavor @yjernite
Issue: Unable to view the Gradio flagged call back dataset

Comment:
Hi @kingabzpro.

I think you need to create a loading script that creates the dataset from the CSV file and the image paths.

As example, you could have a look at the Food-101 dataset: https://huggingface.co/datasets/food101
- Loading script: https://huggingface.co/datasets/food101/blob/main/food101.py

Once the loading script is created, the viewer will show a previsualization of your dataset.
Issue: Unable to view the Gradio flagged call back dataset

Comment:
@albertvillanova I don't think this is the issue. I have created another dataset with similar files and format and it works. https://huggingface.co/datasets/kingabzpro/savtadepth-flags-V2
Issue: Unable to view the Gradio flagged call back dataset

Comment:
Yes, you are right, that was not the issue.

Just take into account that sometimes the viewer can take some time until it shows the preview of the dataset.
After some time, yours is finally properly shown: https://huggingface.co/datasets/kingabzpro/savtadepth-flags
Issue: Unable to view the Gradio flagged call back dataset

Comment:
The problem was resolved by deleted the dataset and creating new one with similar name and then clicking on flag button.
Issue: head_qa file missing

Comment:
We usually find issues with files hosted at Google Drive...

In this case we download the Google Drive Virus scan warning instead of the data file.
Issue: Images get decoded when using `map()` with `input_columns` argument on a dataset

Comment:
Hi! If I'm not mistaken, this behavior is intentional, but I agree it could be more intuitive.

@albertvillanova Do you remember why you decided not to decode columns in the `Audio` feature PR when `input_columns` is not `None`? IMO we should decode those columns, and we don't even have to use lazy structures here because the user explicitly requires them in the map transform. 

cc @lhoestq for visibility
Issue: Images get decoded when using `map()` with `input_columns` argument on a dataset

Comment:
I think I excluded to decorate the function when `input_columns` were passed as a quick fix for some non-passing tests: 
- https://github.com/huggingface/datasets/pull/2324/commits/9d7c3e8fa53e23ec636859b4407eeec904b1b3f9

That PR was quite complex and I decided to focus on the main feature requests, leaving refinements for subsequent PRs.

Note that when `input_columns` are passed, the signature of the function is effectively changed, while the decorated function expects an item (whether an example or a batch) as first arg (which is not the case when passing `input_columns`.

I agree we should consider supporting the case when `input_columns` are passed.
Issue: Cannot preview dataset

Comment:
Thanks for reporting. The dataset viewer depends on some backend treatments, and for now, they might take some hours to get processed. We're working on improving it.
Issue: Expanding streaming capabilities

Comment:
Cool ! `filter` will be very useful. There can be a filter that you can apply on a streaming dataset:
```python
load_dataset(..., streaming=True).filter(lambda x: x["lang"] == "sw")
```

Otherwise if you want to apply a filter on the source files that are going to be used for streaming, the logic has to be impIemented directly in the dataset script, or if there's no dataset script this can be done with pattern matching
```python
load_dataset(..., lang="sw")                    # if the dataset script supports this parameter
load_dataset(..., data_files="data/lang=sw/*")  # if there's no dataset script, but only data files
```

--------------

Here are also some additional ideas of API to convert from iterable to map-style dataset:
```python
on_disk_dataset = streaming_dataset.to_disk()
on_disk_dataset = streaming_dataset.to_disk(path="path/to/my/dataset/dir")

in_memory_dataset = streaming_dataset.take(100).to_memory()  # to experiment without having to write files
```
--------------

Finally regarding `push_to_hub`, we can replace `batch_size` by `shard_size` (same API as for on-disk datasets). The default is 500MB per file

Let me know what you think !
Issue: Expanding streaming capabilities

Comment:
Regarding conversion, I'd also ask for some kind of equivalent to `save_to_disk` for an `IterableDataset`.

Similarly to the streaming to hub idea, my use case would be to define a sequence of dataset transforms via `.map()`, using an `IterableDataset` as the input (so processing could start without doing whole download up-front), but streaming the resultant processed dataset just to disk.
Issue: Expanding streaming capabilities

Comment:
That makes sense @athewsey , thanks for the suggestion :)

Maybe instead of the `to_disk` we could simply have `save_to_disk` instead:
```python
streaming_dataset.save_to_disk("path/to/my/dataset/dir")
on_disk_dataset = load_from_disk("path/to/my/dataset/dir")

in_memory_dataset = Dataset.from_list(list(streaming_dataset.take(100)))  # to experiment without having to write files
```
Issue: Pubmed dataset does not work in streaming mode

Comment:
Thanks for reporting, @abhi-mosaic (related to #3655).

Please note that `xml.etree.ElementTree.parse` already supports streaming:
- #3476

No need to refactor to use `open`/`xopen`. Is is enough with importing the package `as ET` (instead of `as etree`).
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
Note that we might change the heuristic and create a different config per file, at least in that case.
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
Hi @severo, thanks for reporting.

Yes, this happens because when non-streaming, a cast of all data is done in order to "concatenate" it all into a single dataset (thus the error), while this casting is not done while yielding item by item in streaming mode.

Maybe in streaming mode we should keep the schema (inferred from the first item) and throw an exception if a subsequent item does not conform to the inferred schema?
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
Why do we want to concatenate the files? Is it the expected behavior for most datasets that lack a script and dataset info?
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
These files are two different dataset configurations since they don't share the same schema.

IMO the streaming mode should fail in this case, as @albertvillanova said.

There is one challenge though: inferring the schema from the first example is not robust enough in the general case - especially if some fields are nullable. I guess we can at least make sure that no new columns are added
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
OK. So, if we make the streaming also fail, the dataset https://huggingface.co/datasets/huggingface/transformers-metadata will never be [viewable](https://github.com/huggingface/datasets-preview-backend/issues/144) (be it using streaming or fallback to downloading the files), right?
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
Yes, until we have a way for the user to specify explicitly that those two files are different configurations.

We can maybe have some rule to detect this automatically, maybe checking the first line of each file ? That would mean that for dataset of 10,000+ files we would have to verify every single one of them just to know if there is one ore more configurations, so I'm not sure if this is a good idea
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
i think requiring the user to specify that those two files are different configurations is in that case perfectly reasonable.

(Maybe at some point we could however detect this type of case and prompt them to define a config mapping etc)
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
OK, so, before closing the issue, what do you think should be done?

> Maybe in streaming mode we should keep the schema (inferred from the first item) and throw an exception if a subsequent item does not conform to the inferred schema?

or nothing?
Issue: For data-only datasets, streaming and non-streaming don't behave the same

Comment:
We should at least raise an error if a new sample has column names that are missing, or if it has extra columns. No need to check for the type for now.

I'm in favor of having an error especially because we want to avoid silent issues as much as possible - i.e. when something goes wrong (when schemas don't match or some data are missing) and no errors/warnings are raised.

Consistency between streaming and non-streaming is also important.
Issue: Performance of `datasets` at scale

Comment:
The most surprising part to me is the saving time. Wondering if it could be due to compression (`ParquetWriter` uses SNAPPY  compression by default; it can be turned off with `to_parquet(..., compression=None)`).
Issue: Performance of `datasets` at scale

Comment:
+1 to what @mariosasko mentioned. Also, @lvwerra did you parallelize `to_parquet` using similar approach in #2747? (we used multiprocessing at the shard level). I'm working on a similar PR to add multi_proc in `to_parquet` which might give you further speed up. 
Stas benchmarked his approach and mine in this [gist](https://gist.github.com/stas00/dc1597a1e245c5915cfeefa0eee6902c) for `lama` dataset when we were working on adding multi_proc support for `to_json`.
Issue: Performance of `datasets` at scale

Comment:
@mariosasko I did not turn it off but I can try the next time - I have to run the pipeline again, anyway. 

@bhavitvyamalik Yes, I also sharded the dataset and used multiprocessing to save each shard. I'll have a closer look at your approach, too.
Issue: Checksum Error when loading multi-news dataset

Comment:
Thanks for reporting @byw2.
We are fixing it.
In the meantime, you can load the dataset by passing `ignore_verifications=True`:
 ```python
dataset = load_dataset("multi_news", ignore_verifications=True)
Issue: Wrong number of examples when loading a text dataset

Comment:
Hi @kg-nlp, thanks for reporting.

That is weird... I guess we would need some sample data file where this behavior appears to reproduce the bug for further investigation...
Issue: Wrong number of examples when loading a text dataset

Comment:
ok, I found the reason why that two results are not same.
there is /u2029 in the text, the datasets will split sentence according to the /u2029,but when I use open function will not do that .
so I want to know which function shell do that
thanks
Issue: Builder Configuration Update Required on Common Voice Dataset

Comment:
Hi @aasem, thanks for reporting.

Please note that currently Commom Voice is hosted on our Hub as a community dataset by the Mozilla Foundation. See all Common Voice versions here: https://huggingface.co/mozilla-foundation

Maybe we should add an explaining note in our "legacy" Common Voice canonical script? What do you think @lhoestq @mariosasko ?
Issue: Builder Configuration Update Required on Common Voice Dataset

Comment:
Thank you, @albertvillanova, for the quick response. I am not sure about the exact flow but I guess adding the following lines under the `_Languages` dictionary definition in [common_voice.py](https://github.com/huggingface/datasets/blob/master/datasets/common_voice/common_voice.py) might resolve the issue. I guess the dataset is recently made available so the file needs updating.

```
"ur": {
        "Language": "Urdu",
        "Date": "2022-01-19",
        "Size": "68 MB",
        "Version": "ur_3h_2022-01-19",
        "Validated_Hr_Total": 1,
        "Overall_Hr_Total": 3,
        "Number_Of_Voice": 48,
    },
```
Issue: Builder Configuration Update Required on Common Voice Dataset

Comment:
@aasem for compliance reasons, we are no longer updating the `common_voice.py` script.

We agreed with Mozilla Foundation to use their community datasets instead, which will ask you to accept their terms of use:
```
You need to share your contact information to access this dataset.

This repository is publicly accessible, but you have to register to access its content — don't worry, it's just one click!

By clicking on “Access repository” below, you accept that your contact information (email address and username) can be shared with the repository authors. This will let the authors get in touch for instance if some parts of the repository's contents need to be taken down for licensing reasons.

By clicking on “Access repository” below, you also agree to not attempt to determine the identity of speakers in the Common Voice dataset.

You will immediately be granted access to the contents of the dataset. 
```

In order to use e.g. their Common Voice dataset version 8.0, please:
- First visit their dataset page: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
- Accept their term of use by clicking "Access repository"
- You can then load their dataset with:
  ```python
  load_dataset("mozilla-foundation/common_voice_8_0", "ur", split="train+validation")
  ```
Issue: Builder Configuration Update Required on Common Voice Dataset

Comment:
@albertvillanova 
>Maybe we should add an explaining note in our "legacy" Common Voice canonical script?

Yes, I agree we should have a deprecation notice in the canonical script to redirect users to the new script.
Issue: Builder Configuration Update Required on Common Voice Dataset

Comment:
@albertvillanova, 
I now get the following error after downloading my access token from the huggingface and passing it to `load_dataset` call:

`AttributeError: 'DownloadManager' object has no attribute 'download_config'`

Any quick pointer on how it might be resolved?
Issue: Builder Configuration Update Required on Common Voice Dataset

Comment:
@aasem What version of `datasets` are you using? We renamed that attribute from `_download_config` to `download_conig` fairly recently, so updating to the newest version should resolve the issue:
```
pip install -U datasets
```
Issue: wrong condition in `Features ClassLabel encode_example`

Comment:
Hi @Tudyx, 

Please note that in Python, the boolean NOT operator (`not`) has lower precedence than comparison operators (`<=`, `<`), thus the expression you mention is equivalent to:
```python
 not (-1 <= example_data < self.num_classes)
```

Also note that as expected, the exception is raised if:
- `example_data < -1`
- or `example_data >= self.num_classes`

The raise of the exception is expected when `example_data` equals 4 and `self.num_classes` equals 4 too.
Issue: Loading JSON gets stuck with many workers/threads

Comment:
Hi ! Note that it does `block_size *= 2` until `block_size > len(batch)`, so it doesn't loop indefinitely. What do you mean by "get stuck indefinitely" then ? Is this the actual call to `paj.read_json` that hangs ?

> increasing the `chunksize` argument decreases the chance of getting stuck

Could you share the values of chunksize that you're using to observe this ? And maybe the order of magnitude of number of bytes per line of JSON ?
Issue: Loading JSON gets stuck with many workers/threads

Comment:
To clarify, I don't think it loops indefinitely but the `paj.read_json` gets stuck after the first try. That's why I think it could be an issue with a lock somewhere. 

Using `load_dataset(..., chunksize=40<<20)` worked without errors.
Issue: Loading JSON gets stuck with many workers/threads

Comment:
@lhoestq  I encountered another related issue. I use load_dataset() for my json data and set_transform() for preprocessing. But it hangs at the end of the epoch if `dataloader_num_workers>=1`.  It appears to be working fine with num_worker=0, but it's slow.
```
train_dataset = datasets.load_dataset("json", 
                                      data_files=corpus_jsonl_path,
                                      keep_in_memory=False,
                                      cache_dir=model_args.cache_dir,
                                      streaming=False)
train_dataset.set_transform(psg_parse_fn)
```
Issue: Loading JSON gets stuck with many workers/threads

Comment:
I couldn't I think your problem is unrelated to this issue @memray
Indeed this issue discusses a bug when doing `load_dataset`, while your case has to do with the dataloader in a multiprocessing setup. Can you open a new issue and provide more details (share your env and what psg_parse_fn does) ?
Issue: Loading JSON gets stuck with many workers/threads

Comment:
I also encountered a similar issue when loading a 190GB dataset of jsonl files (255 files with less than 1Gb) where it got stuck for over 20h at tables generation (fig below), increasing the `chunksize` with `load_dataset(..., chunksize=40<<20)` fixed the issue

<img width="560" alt="image" src="https://user-images.githubusercontent.com/44069155/195605603-548a106e-7ad3-4269-8cdd-2ad3e975bf16.png">
Issue: Loading JSON gets stuck with many workers/threads

Comment:
> @lhoestq I encountered another related issue. I use load_dataset() for my json data and set_transform() for preprocessing. But it hangs at the end of the epoch if `dataloader_num_workers>=1`. It appears to be working fine with num_worker=0, but it's slow.
> 
> ```
> train_dataset = datasets.load_dataset("json", 
>                                       data_files=corpus_jsonl_path,
>                                       keep_in_memory=False,
>                                       cache_dir=model_args.cache_dir,
>                                       streaming=False)
> train_dataset.set_transform(psg_parse_fn)
> ```

In case people also get this problem, I found a way to fix it by adding `persistent_workers=True` when initializing DataLoader, like:
`train_loader = DataLoader(
            train_dataset,
            batch_size=self._train_batch_size,
            sampler=train_sampler,
            collate_fn=data_collator,
            num_workers=self.args.dataloader_num_workers,
            persistent_workers=True
        )`

The error was `CUDA error: initialization error Exception raised from insert_events at ../c10/cuda/CUDACachingAllocator.cpp:1266` after the 1st epoch, I guess it's because the data_loader worker is killed after each epoch and the data supply is cut off. This error only occurs when num_workers>1.
Issue: Loading JSON gets stuck with many workers/threads

Comment:
I can confirm the issue using datasets (2.12.0) with the following code and Accelerate (0.20.3) env:

````
trainDataloader = DataLoader(trainSplit, batch_size=args.train_batch_size, shuffle=True)
evalDataloader = DataLoader(validSplit, batch_size=args.valid_batch_size) // Here is where it gets stuck.
````
````
- `Accelerate` version: 0.20.3
- Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Numpy version: 1.24.3
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- PyTorch XPU available: False
- System RAM: 503.28 GB
- GPU type: Tesla V100-SXM2-32GB
- `Accelerate` default config:
	- compute_environment: LOCAL_MACHINE
	- distributed_type: MULTI_GPU
	- mixed_precision: fp16
	- use_cpu: False
	- num_processes: 2
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: 0,1
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
````

Notable that with Accelerate configured for one GPU only, **it doesn't get stuck.** 

The suggestion made by @memray worked in my case. This is how it was applied: 
````
trainDataloader = DataLoader(trainSplit, batch_size=args.train_batch_size, shuffle=True, num_workers=2, persistent_workers=True)
evalDataloader = DataLoader(validSplit, batch_size=args.valid_batch_size, num_workers=2, persistent_workers=True)
````
Issue: Loading JSON gets stuck with many workers/threads

Comment:
I think your issue is related to `accelerate`, feel free to open an issue there: https://github.com/huggingface/accelerate/issues

`Dataset` objects generally work fine with the torch DataLoader, idk what `accelerate` does that could make it get stuck.
Issue: `.select`: unexpected behavior with `indices`

Comment:
Hi! Currently, we compute the final index as `index % len(dset)`. I agree this behavior is somewhat unexpected and that it would be more appropriate to raise an error instead (this is what `df.iloc` in Pandas does, for instance).

@albertvillanova @lhoestq wdyt?
Issue: `.select`: unexpected behavior with `indices`

Comment:
I agree. I think `index % len(dset)` was used to support negative indices.

I think this needs to be fixed in `datasets.formatting.formatting._check_valid_index_key` if I'm not mistaken
Issue: Unable to load dataset 'big_patent'

Comment:
Hi @ankitk2109,

Have you tried passing the split name with the keyword `split=`? See e.g. an example in our Quick Start docs: https://huggingface.co/docs/datasets/quickstart.html#load-the-dataset-and-model
```python
 ds = load_dataset("big_patent", "d", split="validation")
Issue: Unable to load dataset 'big_patent'

Comment:
Hi @albertvillanova,

Thanks for your response.

Yes, I tried the `split='validation'` as well. But getting the same issue.
Issue: Unable to load dataset 'big_patent'

Comment:
I'm sorry, but I can't reproduce your problem:
```python
In [5]: ds = load_dataset("big_patent", "d", split="validation")
Downloading and preparing dataset big_patent/d (download: 6.01 GiB, generated: 169.61 MiB, post-processed: Unknown size, total: 6.17 GiB) to .../.cache/big_patent/d/1.0.0/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.45G/6.45G [27:36<00:00, 3.89MB/s]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:18<00:00, 66.08s/it]
Dataset big_patent downloaded and prepared to .../.cache/big_patent/d/1.0.0/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c. Subsequent calls will reuse this data.    

In [6]: ds
Out[6]: 
Dataset({
    features: ['description', 'abstract'],
    num_rows: 565
})
Issue: Unable to load dataset 'big_patent'

Comment:
Maybe you had a connection issue while downloading the file and this was corrupted?
Our cache system uses the file you downloaded first time.
If so, you could try forcing redownload of the file with:
```python
ds = load_dataset("big_patent", "d", split="validation", download_mode="force_redownload")