SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: irc_disentagle viewer error

Comment:
DUPLICATED comment from https://github.com/huggingface/datasets/issues/3807:

my code:
```
from datasets import load_dataset

dataset = load_dataset("irc_disentangle", download_mode="force_redownload")
```
however, it produces the same error
```
[38](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=37) if len(bad_urls) > 0:
     [39](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=38)     error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> [40](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=39)     raise NonMatchingChecksumError(error_msg + str(bad_urls))
     [41](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=40) logger.info("All the checksums matched successfully" + for_verification_name)

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/jkkummerfeld/irc-disentanglement/tarball/master']
```
I attempted to use the `ignore_verifications' as such:

```
ds = datasets.load_dataset('irc_disentangle', download_mode="force_redownload", ignore_verifications=True)

Downloading builder script: 12.0kB [00:00, 5.92MB/s]                   
Downloading metadata: 7.58kB [00:00, 3.48MB/s]                   
No config specified, defaulting to: irc_disentangle/ubuntu
Downloading and preparing dataset irc_disentangle/ubuntu (download: 112.98 MiB, generated: 60.05 MiB, post-processed: Unknown size, total: 173.03 MiB) to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5...
Downloading data: 118MB [00:09, 12.1MB/s] 
                                                                             
Dataset irc_disentangle downloaded and prepared to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5. Subsequent calls will reuse this data.
100%|██████████| 3/3 [00:00<00:00, 675.38it/s]
```
but, this returns an empty set?
```
DatasetDict({
    train: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    test: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
})
```
not sure what else to try at this point?
Thanks in advanced🤗
Issue: irc_disentagle viewer error

Comment:
The issue with checksum and empty dataset has been fixed by:
- #4377

To load the dataset, you should force the re-generation of the dataset from the downloaded file by passing `download_mode="reuse_cache_if_exists"` to `load_dataset`.

In relation with the issue with the dataset viewer, first the dataset should be refactored to support streaming.
Issue: irc_disentagle viewer error

Comment:
Hi there, 
I see this issue is closed, but I am wondering if there is any chance the source files have been moved since this fix? I am stumbling into the same NonMatchingChecksumError noted by lebouz's second post once 118MB of data has been downloaded, and have tried the solutions noted in the various fix checksum posts linked here and in other posts regarding passing in "reuse_cache_if_exists" to download_mode. Any suggestions? Thank you!
Issue: extremely slow processing when using a custom dataset 

Comment:
Hi !

My guess is that some examples in your dataset are bigger than your RAM, and therefore loading them in RAM to pass them to `remove_non_indic_sentences` takes forever because it might use SWAP memory.

Maybe several examples in your dataset are grouped together, can you check `len(lang_dataset["train"])` and `lang_dataset["train"].data.nbytes` of both datasets please ? It can also be helpful to check the distribution of lengths of each examples in your dataset.
Issue: The dataset preview is not available for this split.

Comment:
Hi!  A dataset has to be streamable to work with the viewer. I did a quick test, and yours is, so this might be a bug in the viewer. cc @severo
Issue: The dataset preview is not available for this split.

Comment:
Looking at it. The message is now:

```
Message:       cannot cache function '__shear_dense': no locator available for file '/src/services/worker/.venv/lib/python3.9/site-packages/librosa/util/utils.py'
```

so possibly it's related to the libraries versions?
Issue: Missing dataset tags and sections in some dataset cards

Comment:
@lhoestq I can take this issue. Please can you point out to me where I can find the other positional arguments?
Issue: Missing dataset tags and sections in some dataset cards

Comment:
Hi @RohitRathore1 :)

You can find all the YAML tags in the tagging app here: https://hf.co/spaces/huggingface/datasets-tagging). They're all passed as arguments to a DatasetMetadata object used to validate the tags.
Issue: Problems with WMT dataset

Comment:
Hi! Yes, the docs are outdated. Expect this to be fixed soon. 

In the meantime, you can try to fix the issue yourself.

These are the configs/language pairs supported by `wmt15` from which you can choose:
* `cs-en` (Czech - English)
* `de-en` (German - English)
* `fi-en` (Finnish- English)
* `fr-en` (French - English)
* `ru-en` (Russian - English)

And the current implementation always uses all the subsets available for a language, so to define custom subsets, you'll have to clone the repo from the Hub and replace the line https://huggingface.co/datasets/wmt15/blob/main/wmt_utils.py#L688 with:
`for split, ss_names in (self._subsets if self.config.subsets is None else self.config.subsets).items()`

Then, you can load the dataset as follows:
```python
from datasets import load_dataset
dset = load_dataset("path/to/local/wmt15_folder", "<one of 5 available configs>", subsets=...)
Issue: Problems with WMT dataset

Comment:
Hi @mariosasko 

Are the docs updated? If not, I would like to get on it. I am new around here, would we helpful, if you can guide.

Thanks
Issue: Problems with WMT dataset

Comment:
Hi @mariosasko ,

I have made the changes as suggested by you and have opened a PR #4537.

Thanks
Issue: When using `dataset.map()` if passed `Features` types do not match what is returned from the mapped function, execution does not except in an obvious way

Comment:
Hi ! Thanks for reporting :) `datasets` usually returns a `pa.lib.ArrowInvalid` error if the feature types don't match.

It would be awesome if we had a way to reproduce the `OverflowError` in this case, to better understand what happened and be able to provide the best error message
Issue: Add optional progress bar for .save_to_disk(..) and .load_from_disk(..) when working with remote filesystems

Comment:
Hi! I like this idea. For consistency with `load_dataset`, we can use `fsspec`'s `TqdmCallback` in  `.load_from_disk` to monitor the number of bytes downloaded, and in `.save_to_disk`, we can track the number of saved shards for consistency with `push_to_hub` (after we implement https://github.com/huggingface/datasets/issues/4196).
Issue: Dataset.map()'s fails at any value of parameter writer_batch_size 

Comment:
Note that this same issue occurs even if i preprocess with the more default way of tokenizing that uses LayoutLMv2Processor's internal OCR:

```python
        feature_extractor = LayoutLMv2FeatureExtractor()
        tokenizer = LayoutLMv2Tokenizer.from_pretrained("microsoft/layoutlmv2-base-uncased")
        processor = LayoutLMv2Processor(feature_extractor, tokenizer)
        encoded_inputs = processor(images, padding="max_length", truncation=True)
        encoded_inputs["image"] = np.array(encoded_inputs["image"])
        encoded_inputs["label"] = examples['label_id']
```
Issue: Dataset.map()'s fails at any value of parameter writer_batch_size 

Comment:
Did you close it because you found that it was due to the incorrect Feature types ?
Issue: Dataset.map()'s fails at any value of parameter writer_batch_size 

Comment:
Yeah-- my analysis of the issue was wrong in this one so I just closed it while linking to the new issue
Issue: Dataset.map()'s fails at any value of parameter writer_batch_size 

Comment:
I met with the same problem when doing some experiments about layoutlm. I tried to set the writer_batch_size to 1, and the error still exists. Is there any solutions to this problem?
Issue: Dataset.map()'s fails at any value of parameter writer_batch_size 

Comment:
The problem lies in how your Features are defined. It's erroring out when it actually goes to write them to disk
Issue: `inspect` functions can't fetch dataset script from the Hub

Comment:
Hi, thanks for reporting! `git bisect` points to #2986 as the PR that introduced the bug. Since then, there have been some additional changes to the loading logic, and in the current state, `force_local_path` (set via `local_path`) forbids pulling a script from the internet instead of downloading it: https://github.com/huggingface/datasets/blob/cfae0545b2ba05452e16136cacc7d370b4b186a1/src/datasets/inspect.py#L89-L91

cc @lhoestq: `force_local_path` is only used in `inspect_dataset` and `inspect_metric`. Is it OK if we revert the behavior to match the old one?
Issue: Metrics documentation is not accessible in the datasets doc UI

Comment:
Hey @fxmarty :) Yes we are working on showing the docs of all the metrics on the Hugging face website. If you want to follow the advancements you can check the [evaluate](https://github.com/huggingface/evaluate) repository cc @lvwerra @sashavor
Issue: `wikipedia` pre-processed datasets

Comment:
Hi @vpj, thanks for reporting.

I'm sorry, but I can't reproduce your bug: I load "20220301.simple"in 9 seconds:
```shell
time python -c "from datasets import load_dataset; load_dataset('wikipedia', '20220301.simple')"

Downloading and preparing dataset wikipedia/20220301.simple (download: 228.58 MiB, generated: 224.18 MiB, post-processed: Unknown size, total: 452.76 MiB) to .../.cache/huggingface/datasets/wikipedia/20220301.simple/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.66k/1.66k [00:00<00:00, 1.02MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 235M/235M [00:02<00:00, 82.8MB/s]
Dataset wikipedia downloaded and prepared to .../.cache/huggingface/datasets/wikipedia/20220301.simple/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 290.75it/s]

real	0m9.693s
user	0m6.002s
sys	0m3.260s
```

Could you please check your environment info, as requested when opening this issue?
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```
Maybe you are using an old version of `datasets`...
Issue: `wikipedia` pre-processed datasets

Comment:
Downloading and processing `wikipedia simple` dataset completed in under 11sec on M1 Mac. Could you please check `dataset` version as mentioned by @albertvillanova? Also check system specs, if system is under load processing could take some time I guess.
Issue: Dataset Viewer issue for strombergnlp/offenseval_2020, strombergnlp/polstance

Comment:
Not sure if it's related... I was going to raise an issue for https://huggingface.co/datasets/domenicrosati/TruthfulQA which also has the same issue... https://huggingface.co/datasets/domenicrosati/TruthfulQA/viewer/domenicrosati--TruthfulQA/train
Issue: Dataset Viewer issue for strombergnlp/offenseval_2020, strombergnlp/polstance

Comment:
Yes, it's related. The backend behind the dataset viewer is currently under too much load, and these datasets are still in the jobs queue. We're actively working on this issue, and we expect to fix the issue permanently soon. Thanks for your patience 🙏
Issue: Dataset Viewer issue for strombergnlp/offenseval_2020, strombergnlp/polstance

Comment:
Thanks @severo and no worries! - a suggestion for a UI usability thing maybe is to indicate that the dataset processing is in the job queue (rather than no data?)
Issue: Audio can not find value["bytes"]

Comment:
![image](https://user-images.githubusercontent.com/34292279/168063684-fff5c12a-8b1e-4c65-b18b-36100ab8a1af.png)

that is reason my bytes`s empty
but i have some confused why path prior is higher than bytes?

if you can make bytes in _generate_examples , you don`t have to make bytes to path?
because we have path and bytes already
Issue: Audio can not find value["bytes"]

Comment:
> but i have some confused why path prior is higher than bytes?

If the audio file is already available locally, we don't need to store the bytes again.

If you don't specify a "path" to a local file, then the bytes are stored. You can set "path" to None for example.

> if you can make bytes in _generate_examples , you don`t have to make bytes to path?
> because we have path and bytes already

It's useful to pass both "path" and "bytes" in `_generate_examples`:
- when the dataset has been downloaded, then the "path" to the audio files are stored and we can ignore "bytes" in order to save disk space.
- when the dataset is loaded in streaming mode, the audio files are not available on your disk and therefore we use the "bytes"
Issue: Audio can not find value["bytes"]

Comment:
@lhoestq 
First of all, thx for reply

but, if i put in "bytes" and "path"
ex) {"bytes":"blah blah~", "path":"blah blah~"}

that source working that my bytes to empty first,
and then, re-calculate my bytes!
![image](https://user-images.githubusercontent.com/34292279/168534687-1fb60d8c-d369-47d2-a4bb-db68f95194b4.png)

if you have some pcm file, pcm is can read bytes.
so, i put in bytes and paths.
but bytes is been None why encode_example func make None
and then, on decode_example func, we no have bytes. so, calculate bytes to path.
pcm is not support librosa or soundfile, error occured!

the most important thing is not announced anywhere this situation can be reproduced

is that truly right process flow?
Issue: Audio can not find value["bytes"]

Comment:
I don't think we support PCM files, feel free to convert your data to WAV for now.

It would be awesome to support PCM files though, let me know if you'd like to contribute this feature, I'd be happy to help
Issue: Audio can not find value["bytes"]

Comment:
You can clone the repository (see the guide on [how to contribute](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-create-a-pull-request)) and see how we can make the `Image.encode_example` method work with PCM data.

There might be other ways to approach this problem, but here is what I think is a reasonable one:

I think `Image.encode_example` should be able to take PCM bytes as input and the sampling rate, and return the WAV bytes (built by combining the PCM bytes and the sampling rate info), so that `Image.decode_example` can read it.

To check if the input bytes are PCM data, you can just check if the extension of the `path` is ".pcm".
Issue: Multi-news dataset loader attempts to strip wrong character from beginning of summaries

Comment:
Hi ! Thanks for reporting :)

This dataset was simply converted from [tensorflow datasets](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/multi_news.py)

I think we can just remove the `.strip("- ")` and keep this character
Issue: Language code search does direct matches

Comment:
Thanks for reporting ! I forwarded the issue to the front-end team :)

Will keep you posted !

I also changed the tagging app to suggest two letters code for now.
Issue: Normalise license names

Comment:
we'll add the same server-side metadata validation system as for hf.co/models soon-ish

(you can check on hf.co/models that licenses are "clean")
Issue: Dataset Viewer issue for strombergnlp/ipm_nel : preview is empty, no error message

Comment:
Hi @leondz, thanks for reporting.

Indeed, the dataset viewer relies on the dataset being streamable (passing `streaming=True` to `load_dataset`). Whereas most of the datastes are streamable out of the box (thanks to our implementation of streaming), there are still some exceptions.

In particular, in your case, that is due to the data file being TAR. This format is not streamable out of the box (it does not allow random access to the archived files), but we use a trick to allow streaming: using `dl_manager.iter_archive`.

Let me know if you need some help: I could push a commit to your repo with the fix.
Issue: Dataset Viewer issue for strombergnlp/ipm_nel : preview is empty, no error message

Comment:
Ah, right! The preview is working now, but this explanation is good to know, thank you. I'll prefer formats with random file access supported in datasets.utils.extract in future, and try out this fix for the tarfiles :)
Issue: "NameError: name 'faiss' is not defined" on `.add_faiss_index` when `device` is not None

Comment:
So I managed to solve this by adding a missing `import faiss` in the `@staticmethod` defined in https://github.com/huggingface/datasets/blob/f51b6994db27ea69261ef919fb7775928f9ec10b/src/datasets/search.py#L305, triggered from https://github.com/huggingface/datasets/blob/f51b6994db27ea69261ef919fb7775928f9ec10b/src/datasets/search.py#L249 when trying to `ds_with_embeddings.add_faiss_index(column='embeddings', device=0)` with the code above.

As it seems that the `@staticmethod` doesn't recognize the `import faiss` defined in https://github.com/huggingface/datasets/blob/f51b6994db27ea69261ef919fb7775928f9ec10b/src/datasets/search.py#L261, so whenever the value of `device` is not None in https://github.com/huggingface/datasets/blob/71f76e0bdeaddadedc4f9c8d15cfff5a36d62f66/src/datasets/search.py#L438, that exception is triggered.

So on, adding `import faiss` inside https://github.com/huggingface/datasets/blob/71f76e0bdeaddadedc4f9c8d15cfff5a36d62f66/src/datasets/search.py#L305 right after the check of `device`'s value, solves the issue and lets you calculate the indices in GPU.

I'll add the code in a PR linked to this issue in case you want to merge it!
Issue: "NameError: name 'faiss' is not defined" on `.add_faiss_index` when `device` is not None

Comment:
Adding here the complete error traceback!

```
Traceback (most recent call last):
  File "/home/alvarobartt/lol.py", line 12, in <module>
    ds_with_embeddings.add_faiss_index(column='embeddings', device=0) # default `device=None`
  File "/home/alvarobartt/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3656, in add_faiss_index
    super().add_faiss_index(
  File "/home/alvarobartt/.local/lib/python3.9/site-packages/datasets/search.py", line 478, in add_faiss_index
    faiss_index.add_vectors(self, column=column, train_size=train_size, faiss_verbose=True)
  File "/home/alvarobartt/.local/lib/python3.9/site-packages/datasets/search.py", line 281, in add_vectors
    self.faiss_index = self._faiss_index_to_device(index, self.device)
  File "/home/alvarobartt/.local/lib/python3.9/site-packages/datasets/search.py", line 327, in _faiss_index_to_device
    faiss_res = faiss.StandardGpuResources()
NameError: name 'faiss' is not defined
```
Issue: Issues in processing very large datasets

Comment:
Hi ! `datasets` doesn't load the dataset in memory. Instead it uses memory mapping to load your dataset from your disk (it is stored as arrow files). Do you know at what point you have RAM issues exactly ?

How big are your graph_data_train dictionaries btw ?
Issue: OpenBookQA has missing and inconsistent field names

Comment:
Thanks for reporting, @vblagoje.

Indeed, I noticed some of these issues while reviewing this PR:
- #4259 

This is in my TODO list.
Issue: OpenBookQA has missing and inconsistent field names

Comment:
On the other hand, I am not sure if we should always preserve the original nested structure. I think we should also consider other factors as convenience or consistency.

For example, other datasets also flatten "question.stem" into "question":
- ai2_arc:
  ```python
  question = data["question"]["stem"]
  choices = data["question"]["choices"]
  text_choices = [choice["text"] for choice in choices]
  label_choices = [choice["label"] for choice in choices]
  yield id_, {
      "id": id_,
      "answerKey": answerkey,
      "question": question,
      "choices": {"text": text_choices, "label": label_choices},
  }
  ```
- commonsense_qa:
  ```python
  question = data["question"]
  stem = question["stem"]
  yield id_, {
      "answerKey": answerkey,
      "question": stem,
      "choices": {"label": labels, "text": texts},
  }
  ```
- cos_e:
  ```python
  "question": cqa["question"]["stem"],
  ```
- qasc
- quartz
- wiqa

Exceptions:
- exams

I think we should agree on a CONVENIENT format for QA and use always CONSISTENTLY the same.
Issue: OpenBookQA has missing and inconsistent field names

Comment:
@albertvillanova I agree that we should be consistent. In the last month, I have come across tons of code that deals with OpenBookQA and CommonSenseQA and all of that code relies on the original data format structure. We can't expect users to adopt HF Datasets if we arbitrarily change the structure of the format just because we think something makes more sense. I am in that position now (downloading original data rather than using HF Datasets) and undoubtedly it hinders HF Datasets' widespread use and adoption. Missing fields like in the case of #4275 is definitely bad and not even up for a discussion IMHO! cc @lhoestq
Issue: OpenBookQA has missing and inconsistent field names

Comment:
I'm opening a PR that adds the missing fields.

Let's agree on the feature structure: @lhoestq @mariosasko @polinaeterna
Issue: OpenBookQA has missing and inconsistent field names

Comment:
IMO we should always try to preserve the original structure unless there is a good reason not to (and I don't see one in this case).
Issue: OpenBookQA has missing and inconsistent field names

Comment:
I agree with @mariosasko . The transition to the original format could be done in one PR for the next minor release, clearly documenting all dataset changes just as @albertvillanova outlined them above and perhaps even providing a per dataset util method to convert the new valid format to the old for backward compatibility. Users who relied on the old format will update their code with either the util method for a quick fix or slightly more elaborate for the new.