SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Dataset Viewer issue for wmt16

Comment:
Thanks for reporting, @lewtun.

~We can't load the dataset locally, so I think this is an issue with the loading script (not the viewer).~

 We are investigating...
Issue: Dataset Viewer issue for wmt16

Comment:
Recently, there was a merged PR related to this dataset:
- #4554

We are looking at this...
Issue: Dataset Viewer issue for wmt16

Comment:
Indeed, the above mentioned PR fixed the loading script (it was not working before).

I'm forcing the refresh of the Viewer.
Issue: Dataset Viewer issue for wmt16

Comment:
Please note that the above mentioned PR also made an enhancement in the `datasets` library, required by this loading script. This enhancement will only be available to the Viewer once we make our next release.
Issue: Can't extract files from `.7z` zipfile using `download_and_extract`

Comment:
Hi @bhavitvyamalik, thanks for reporting.

Yes, currently we do not support 7zip archive compression: I think we should.

As a workaround, you could uncompress it explicitly, like done in e.g. `samsum` dataset: 

https://github.com/huggingface/datasets/blob/fedf891a08bfc77041d575fad6c26091bc0fce52/datasets/samsum/samsum.py#L106-L110
Issue: Can't extract files from `.7z` zipfile using `download_and_extract`

Comment:
Sure, let me look into and check what can be done. Will keep you guys updated here!
Issue: Can't extract files from `.7z` zipfile using `download_and_extract`

Comment:
Initially, I thought of solving this without any external dependency. Almost everywhere I saw `lzma` can be used for this but there is a caveat that lzma doesn’t work with 7z archives but only single files. In my case the 7z archive has multiple files so it didn't work. Is it fine to use external library here?
Issue: Can't extract files from `.7z` zipfile using `download_and_extract`

Comment:
Hi @bhavitvyamalik, thanks for your investigation.

On Monday, I started a PR that will eventually close this issue as well: I'm linking it to this.
- #4672

Let me know what you think.
Issue: Dataset Viewer issue for hungnm/multilingual-amazon-review-sentiment-processed

Comment:
It seems like a private dataset. The viewer is currently not supported on the private datasets.
Issue: Issues with concatenating datasets

Comment:
Hi! I agree we should improve the features equality checks to account for this particular case. However, your code fails due to `answer_start` having the dtype `int64` instead of `int32` after loading from JSON (it's not possible to embed type precision info into a JSON file; `save_to_disk` does that for arrow files), which would lead to the concatenation error as PyArrow does not support this sort of type promotion. This can be fixed as follows:
```python
temp = load_dataset("json", data_files={"train": "output.jsonl"}, features=squad["train"].features)
```
Issue: Issues with concatenating datasets

Comment:
That makes sense. I totally missed the `int64` and `int32` part. Thanks for pointing it out! Will close this issue for now.
Issue: Unable to create dataset having Python dataset script only

Comment:
Hi @aleSuglia, thanks for reporting.

We are having a look at it. 

We transfer this issue to the Community tab of the corresponding Hub dataset: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/discussions
Issue: Concurrency bug when using same cache among several jobs

Comment:
I can confirm that if I run one job first that processes the dataset, then I can run any jobs in parallel with no problem (no write-concurrency anymore...).
Issue: Concurrency bug when using same cache among several jobs

Comment:
Hi! That's weird. It seems like the error points to the `mkstemp` function, but the official docs state the following:
```
There are no race conditions in the file’s creation, assuming that the platform properly implements the [os.O_EXCL](https://docs.python.org/3/library/os.html#os.O_EXCL) flag for [os.open()](https://docs.python.org/3/library/os.html#os.open)
```
So this could mean your platform doesn't support that flag.

~~Can you please check if wrapping the temp file creation (the line `tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)` in `_map_single`) with the `multiprocess.Lock` fixes the issue?~~
Perhaps wrapping the temp file creation in `_map_single`  with `filelock` could work:
```python
with FileLock(lock_path):
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
```
Can you please check if that helps?
Issue: Streaming issue for ccdv/pubmed-summarization

Comment:
Thanks for reporting @lewtun.

I confirm there is an issue with streaming: it does not stream locally.
Issue: Streaming issue for ccdv/pubmed-summarization

Comment:
Oh, after investigation, the source of the issue is in the Hub dataset loading script.

I'm opening a PR on the Hub dataset.
Issue: Dataset Viewer issue for kmfoda/booksum

Comment:
Thanks for reporting, @lewtun.

It works locally in streaming mode:
```
{'bid': 27681,
 'is_aggregate': True,
 'source': 'cliffnotes',
 'chapter_path': 'all_chapterized_books/27681-chapters/chapters_1_to_2.txt',
 'summary_path': 'finished_summaries/cliffnotes/The Last of the Mohicans/section_1_part_0.txt',
 'book_id': 'The Last of the Mohicans.chapters 1-2',
 'summary_id': 'chapters 1-2',
 'content': None,
 'summary': '{"name": "Chapters 1-2", "url": "https://web.archive.org/web/20201101053205/https://www.cliffsnotes.com/literature/l/the-last-of-the-mohicans/summary-and-analysis/chapters-12", "summary": "Before any characters appear, the time and geography are made clear. Though it is the last war that England and France waged for a country that neither would retain, the wilderness between the forces still has to be...
```

I'm forcing the refresh of the preview.
Issue: The "all" split breaks streaming

Comment:
@albertvillanova Nice! Let me know if it's something I can fix my self; would love to contribtue!
Issue: The "all" split breaks streaming

Comment:
@cakiki I was working on this but if you would like to contribute, go ahead. I will close my PR. ;)

For the moment I just pushed the test (to see if it impacts other tests).
Issue: The "all" split breaks streaming

Comment:
It impacted the test `test_generator_based_download_and_prepare` and I have fixed this.

So that you can copy the test I implemented in my PR and then implement a fix for this issue that passes the test `tests/test_builder.py::test_builder_as_streaming_dataset`.
Issue: The "all" split breaks streaming

Comment:
Hi @cakiki are you still interested in working on this? Are you planning to open a PR?
Issue: The "all" split breaks streaming

Comment:
Hi @albertvillanova ! Sorry it took so long; I wanted to spend this weekend working on it.
Issue: Dataset Viewer issue for vadis/sv-ident

Comment:
Thanks for reporting, @e-tornike 

Some context:
- #4527 

The dataset loads locally in streaming mode:
```python
In [2]: from datasets import load_dataset; ds = load_dataset("vadis/sv-ident", split="validation", streaming=True); item = next(iter(ds)); item
Using custom data configuration default
Out[2]: 
{'sentence': 'Im Falle von Umweltbelastungen kann selten eindeutig entschieden werden, ob Unbedenklichkeitswerte bereits erreicht oder überschritten sind, die die menschliche Gesundheit oder andere Wohlfahrts»güter« beeinträchtigen.',
 'is_variable': 0,
 'variable': [],
 'research_data': [],
 'doc_id': '51971',
 'uuid': 'ee3d7f88-1a3e-4a59-997f-e986b544a604',
 'lang': 'de'}
```
Issue: Dataset Viewer issue for vadis/sv-ident

Comment:
~~I have forced the refresh of the split in the preview without success.~~

I have forced the refresh of the split in the preview, and now it works.
Issue: Dataset Viewer issue for vadis/sv-ident

Comment:
OK, thank you @e-tornike.

Apparently, after forcing the refresh, we just had to wait a little until it is effectively refreshed.
Issue: Dataset Viewer issue for vadis/sv-ident

Comment:
I'm closing this issue as it was solved after forcing the refresh of the split in the preview.
Issue: Can't load the Hausa audio dataset

Comment:
Could you provide the error details. It is difficult to debug otherwise. Also try other config. `ha` is not a valid.
Issue: 'sort' method sorts one column only

Comment:
Hi ! `ds.sort()` does sort the full dataset, not just one column:
```python
from datasets import *

ds = Dataset.from_dict({"foo": [3, 2, 1], "bar": ["c", "b", "a"]})
print(d.sort("foo").to_pandas()
#    foo bar
# 0    1   a
# 1    2   b
# 2    3   c
```

What made you think it was not the case ? Did you experience a situation where it was only sorting one column ?
Issue: 'sort' method sorts one column only

Comment:
Hi! thank you for your quick reply!
I wanted to sort the `cnn_dailymail` dataset by the length of the labels (num of characters). I added a new column to the dataset (`ds.add_column`) with the lengths and then sorted by this new column. Only the new length column was sorted, the reset left in their original order.
Issue: Loading MNIST as Pytorch Dataset

Comment:
Hi ! We haven't implemented the conversion from images data to PyTorch tensors yet I think

cc @mariosasko
Issue: Loading MNIST as Pytorch Dataset

Comment:
So I understand:

set_format() does not properly do the conversion to pytorch tensors from PIL images.

So that someone who stumbles on this can use the package:

```python
dataset = load_dataset("mnist", split="train")
def transform_func(examples):
    examples["image"] = [np.array(img) for img in examples["image"]]
    return examples
dataset = dataset.with_transform(transform_func)
dataset[0]
```
Issue: Loading MNIST as Pytorch Dataset

Comment:
This then appears to work with pytorch dataloaders as:
```
dataloader=torch.utils.data.DataLoader(dataset,batch_size=1)
```

and tensorflow as:
```
dataset=dataset.to_tf_dataset(batch_size=1)
```
Issue: Loading MNIST as Pytorch Dataset

Comment:
Hi! `set_transform`/`with_transform` is indeed the correct solution for the conversion. Improving this part of the API is one of the things I'm working on currently, so stay tuned!
Issue: np arrays get turned into native lists

Comment:
If you add the line `dataset2.set_format('np')` before calling `dataset2[0]['tmp']` it should return `np.ndarray`.
I believe internally it will not store it as a list, it is only returning a list when you index it.

```
In [1]: import datasets, numpy as np
In [2]: dataset = datasets.load_dataset("glue", "mrpc")["validation"]
In [3]: dataset2 = dataset.map(lambda x: {"tmp": np.array([0.5])}, batched=False)
In [4]: dataset2[0]["tmp"]
Out[4]: [0.5]

In [5]: dataset2.set_format('np')

In [6]: dataset2[0]["tmp"]
Out[6]: array([0.5])
```
Issue: np arrays get turned into native lists

Comment:
I see, thanks! Any idea if the default numpy → list conversion might cause precision loss?
Issue: np arrays get turned into native lists

Comment:
I'm not super familiar with our datasets works internally, but I think your `np` array will be stored in a `pyarrow` format, and then you take a view of this as a python array. In which case, I think the precision should be preserved.
Issue: contribute data loading for object detection datasets with yolo data format

Comment:
Hi! The `imagefolder` script is already quite complex, so a standalone script sounds better. Also, I suggest we create an org on the Hub (e.g. `hf-loaders`) and store such scripts there for easier maintenance rather than having them as packaged modules (IMO only very generic loaders should be packaged). WDYT @lhoestq @albertvillanova @polinaeterna?
Issue: contribute data loading for object detection datasets with yolo data format

Comment:
Thank you for the suggestion @mariosasko . I agree with the point, but I have a few doubts

1. How would the user access the script if it's not a part of the core codebase?
2. Could you direct me as to what will be the tasks I have to do to contribute to the code? As per my understanding, it would be like
    1. Create a new org "hf-loaders" and add you (and more HF people) to the org
    2. Add data loader script as a (model?)
    3. Test it with a dataset on HF hub
3. We should maybe brainstorm as to which public datasets have this format (YOLO type) and are the most important ones to test the script with. We can even add the datasets on HF Hub alongside the script
Issue: contribute data loading for object detection datasets with yolo data format

Comment:
1. Like this: `load_dataset("hf-loaders/yolo", data_files=...)`
2. The steps would be:
    1. Create a new org `hf-community-loaders` (IMO a better name than "hf-loaders") and add me (as an admin)
    2. Create a new dataset repo `yolo` and add the loading script to it (`yolo.py`)
    3. Open a discussion to request our review
4. I like this idea. Another option is to add snippets that describe how to load such datasets using the `yolo` loader.
Issue: Release 2.3.0 broke custom iterable datasets

Comment:
Apparently, `fsspec` does not allow access to attribute-based modules anymore, such as `fsspec.async`.

However, this is a fairly simple fix:
- Change the import to: `from fsspec import asyn`;
- Change line 18 to: `asyn.iothread[0] = None`;
- Change line 19 to `asyn.loop[0] = None`.
Issue: Release 2.3.0 broke custom iterable datasets

Comment:
Hi! I think it's easier to replace `import fsspec` with `import fsspec.asyn` and leave the rest unchanged. @gugarosa Are you interested in submitting a PR?
Issue: codeparrot/github-code failing to load 

Comment:
I believe the issue is in `codeparrot/github-code`. `base_path` param is missing - https://huggingface.co/datasets/codeparrot/github-code/blob/main/github-code.py#L169

Function definition has changed.
https://github.com/huggingface/datasets/blob/0e1c629cfb9f9ba124537ba294a0ec451584da5f/src/datasets/data_files.py#L547

@mariosasko could you please confirm my finding? And are there any changes that need to be done from my side?
Issue: codeparrot/github-code failing to load 

Comment:
Good catch ! We recently did a breaking change in `get_patterns_in_dataset_repository`, I think we can revert it
Issue: codeparrot/github-code failing to load 

Comment:
> Good catch ! We recently did a breaking change in `get_patterns_in_dataset_repository`, I think we can revert it

I can't wait for that releasee. Broke my application
Issue: codeparrot/github-code failing to load 

Comment:
This simple workaround should fix: https://huggingface.co/datasets/codeparrot/github-code/discussions/2

`get_patterns_in_dataset_repository` can treat whether `base_path=None`, so we just need to make sure that codeparrot/github-code `_split_generators` calls with such an argument.
Issue: codeparrot/github-code failing to load 

Comment:
I am afraid your suggested change @gugarosa will break compatibility with older datasets versions that don't have `base_path` argument in `get_patterns_in_dataset_repository`, as a workaround while the issue gets resolved in `datasets` can you downgrade your datasets version  to `<=2.1.0` ? 
@lvwerra do you think we should adapt the script to check the datasets version before calling `get_patterns_in_dataset_repository`?
Issue: codeparrot/github-code failing to load 

Comment:
Actually I think it's just simpler to fix it in the dataset itself, let me open a PR

EDIT: PR opened here: https://huggingface.co/datasets/codeparrot/github-code/discussions/3
Issue: codeparrot/github-code failing to load 

Comment:
> I am afraid your suggested change @gugarosa will break compatibility with older datasets versions that don't have `base_path` argument in `get_patterns_in_dataset_repository`, as a workaround while the issue gets resolved in `datasets` can you downgrade your datasets version to `<=2.1.0` ?
> @lvwerra do you think we should adapt the script to check the datasets version before calling `get_patterns_in_dataset_repository`?

You are definitely right, sorry about it. I always keep forgetting that we need to keep in mind users from past versions, my bad.
Issue: librispeech dataset has to download whole subset when specifing the split to use

Comment:
Hi! You can use streaming to fetch only a subset of the data:
```python
raw_dataset = load_dataset("librispeech_asr", "clean", split="train.100", streaming=True)
```
Also, we plan to make it possible to download a particular split in the non-streaming mode, but this task is not easy due to how our dataset scripts are structured.
Issue: evaluation result changes after `datasets` version change

Comment:
Hi! The GH/no-namespace datasets versioning is synced with the version of the `datasets` lib, which means that the `wikiann` script was modified between the two compared versions. In this scenario, you can ensure reproducibility by pinning the script version, which is done by passing `revision="x.y.z"` (e.g. `revision="2.2.0"`) to `load_dataset.`
Issue: Dataset Viewer issue for boris/gis_filtered

Comment:
Yes, this dataset is "gated": you first have to go to https://huggingface.co/datasets/boris/gis_filtered and click "Access repository" (if you accept to share your contact information with the repository authors).
Issue: Dataset Viewer issue for boris/gis_filtered

Comment:
I could reproduce the error, even though I provided my token and accepted the gate form. It looks like an error from `datasets`
Issue: Dataset Viewer issue for boris/gis_filtered

Comment:
This is indeed a bug in `datasets`. Parquet datasets in gated/private repositories can't be streamed properly, which caused the viewer to fail. I opened a PR at https://github.com/huggingface/datasets/pull/4608
Issue: Streaming issue for financial_phrasebank

Comment:
cc @huggingface/datasets: it seems like https://www.researchgate.net/ is flaky for datasets hosting (I put the "hosted-on-google-drive" tag since it's the same kind of issue I think)
Issue: Streaming issue for financial_phrasebank

Comment:
License is Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0).

We can host their data on the Hub.
Issue: Dataset Viewer issue for universal_dependencies

Comment:
Finally fixed! We updated the dataset viewer and it fixed the issue.

https://huggingface.co/datasets/universal_dependencies/viewer/aqz_tudet/train

<img width="1561" alt="Capture d’écran 2022-09-07 à 13 29 18" src="https://user-images.githubusercontent.com/1676121/188867795-4f7dd438-d4f2-46cd-8a92-20a37fb2d6bc.png">
Issue: Dataset Viewer issue with False positive PII redaction

Comment:
The value is in the data, it's not an issue with the "dataset-viewer".

<img width="1161" alt="Capture d’écran 2022-06-29 à 10 25 51" src="https://user-images.githubusercontent.com/1676121/176389325-4d2a9a7f-1583-45b8-aa7a-960ffaa6a36a.png">

 Maybe open a PR: https://huggingface.co/datasets/cakiki/rosetta-code/discussions
Issue: Dataset Viewer issue with False positive PII redaction

Comment:
This was indeed a scraping issue which I assumed was a display issue; sorry about that!
Issue: Issue with jalFaizy/detect_chess_pieces when running datasets-cli test

Comment:
Hi @faizankshaikh

Please note that we have recently launched the Community feature, specifically targeted to create Discussions (about issues/questions/asking-for-help) on each Dataset on the Hub:
- Blog post: https://huggingface.co/blog/community-update
- Docs: https://huggingface.co/docs/hub/repositories-pull-requests-discussions

The Discussion tab for your "jalFaizy/detect_chess_pieces" dataset is here: https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/discussions
You can use it to ask for help by pinging the Datasets maintainers: see our docs here: https://huggingface.co/docs/datasets/master/en/share#ask-for-a-help-and-reviews

I'm transferring this discussion to your Discussion tab and trying to address it: https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/discussions/1
Issue: Issue with jalFaizy/detect_chess_pieces when running datasets-cli test

Comment:
Thank you @albertvillanova , I will keep that in mind.

Just a quick note - I posted the issue on Github because the dataset viewer suggested me to "open an issue for direct support". Maybe it can be updated with your suggestion

![image](https://user-images.githubusercontent.com/8406903/176397633-7b077d81-2044-4487-b58e-6346b05be5cf.png)
Issue: Issue with jalFaizy/detect_chess_pieces when running datasets-cli test

Comment:
Thank you pointing this out: yes, definitely, we should fix the error message. We are working on this.
Issue: Can't push Images to hub with manual Dataset

Comment:
Hi, thanks for reporting! This issue stems from the changes introduced in https://github.com/huggingface/datasets/pull/4282 (cc @lhoestq), in which list casts are ignored if they don't change the list type (required to preserve `null` values). And `push_to_hub` does a special cast to embed external image files but doesn't change the types, hence the failure.
Issue: Dataset Viewer issue for pn_summary

Comment:
Note that I refreshed twice this dataset, and I still have (another) error on one of the splits

```
Status code:   400
Exception:     ClientResponseError
Message:       403, message='Forbidden', url=URL('https://doc-14-4c-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/pgotjmcuh77q0lk7p44rparfrhv459kp/1656403650000/11771870722949762109/*/16OgJ_OrfzUF_i3ftLjFn9kpcyoi7UJeO?e=download')
```

Like the three splits are processed in parallel by the workers, I imagine that the Google hosting is rate-limiting us.

cc @albertvillanova
Issue: Dataset Viewer issue for multi_news

Comment:
Thanks for reporting, @lewtun.

I forced the refreshing of the preview and it worked OK for train and validation splits.

I guess the error has to do with the data files being hosted at Google Drive: this gives errors when requested automatically using scripts.
We should host them to fix the error. Let's see if the license allows that.
Issue: [Multi Configs] Use directories to differentiate between subsets/configurations

Comment:
How to set new split names, instead of train/test/validation? For example, I have a local dataset, consists of several subsets, named "A", "B", and "C". How can I create a huggingface dataset, with splits A/B/C ?

The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?
Issue: [Multi Configs] Use directories to differentiate between subsets/configurations

Comment:
> The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

It works the same - you just need to use local paths instead of URLs
Issue: Problem about wmt17 zh-en dataset

Comment:
@winterfell2021 Hi, I wonder where the code you provided should be added. I tried to add them in the `datasets/table.py` in `array_cast` function, however, the 'zh' item is none.
Issue: Problem about wmt17 zh-en dataset

Comment:
I found some 'zh' item is none while 'c[hn]' is not.
So the code may change to:
```python
if 'c[hn]' in str(array.type):
        py_array = array.to_pylist()
        data_list = []
        for vo in py_array:
            tmp = {
                'en': vo['en'],
            }
            if vo.get('zh'):
                tmp['zh'] = vo['zh']
            else:
                tmp['zh'] = vo['c[hn]']
            data_list.append(tmp)
        array = pa.array(data_list, type=pa.struct([
            pa.field('en', pa.string()),
            pa.field('zh', pa.string()),
        ]))
```
Issue: Problem about wmt17 zh-en dataset

Comment:
I just pushed a fix, we'll do a new release of `datasets` soon to include this fix. In the meantime you can use the fixed dataset by passing `revision="main"` to `load_dataset`
Issue: Dataset Viewer issue for mlsum

Comment:
Thanks for reporting, @lewtun.

After investigation, it seems that the server https://gitlab.lip6.fr does not allow HTTP Range requests.

We are trying to find a workaround...