NyssaW/github-issues huggingface dataset:
Cannot preview "hazal/Turkish-Biomedical-corpus-trM"

## Dataset viewer issue for '*hazal/Turkish-Biomedical-corpus-trM'

**Link:** *https://huggingface.co/datasets/hazal/Turkish-Biomedical-corpus-trM*

*I cannot see the dataset preview.*

```
Server Error

Status code:   400
Exception:     HTTPError
Message:       403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/hazal/Turkish-Biomedical-corpus-trM?full=true
```


Am I the one who added this dataset ? Yes


Comments:
Hi @hazalturkmen, thanks for reporting.

Note that your dataset repository does not contain any loading script; it only contains a data file named `tr_article_2`.

When there is no loading script but only data files, the `datasets` library tries to infer how to load the data by looking at the data file extensions. However, your data file does not have any extension.

Note that current supported data file extensions are: 'csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip'.

You have more info on our docs: [How to share a dataset](https://huggingface.co/docs/datasets/share).

thanks for reply :)
Fix None issue with Sequence of dict

`Features.encode_example` currently fails if it contains a sequence if dict like `Sequence({"subcolumn": Value("int32")})` and if `None` is passed instead of the dict.

```python
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 1310, in encode_example
    return encode_nested_example(self, example)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 973, in encode_nested_example
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 973, in <dictcomp>
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 998, in encode_nested_example
    for k, (sub_schema, sub_objs) in zip_dict(schema.feature, obj):
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/py_utils.py", line 207, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/py_utils.py", line 207, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
TypeError: 'NoneType' object is not subscriptable
```

I fixed this issue and updated the tests (this case was missing in the tests)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merging since I'd like do do a patch release soon for this one
AMI load_dataset error: sndfile library not found

## Describe the bug
Getting error message when loading AMI dataset.

## Steps to reproduce the bug
`python3 -c "from datasets import load_dataset; print(load_dataset('ami', 'headset-single', split='validation')[0])"

`

## Expected results
A clear and concise description of the expected results.

## Actual results
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/load.py", line 1707, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/builder.py", line 595, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/builder.py", line 690, in _download_and_prepare
    ) from None
OSError: Cannot find data file. 
Original error:
sndfile library not found

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.3
- PyArrow version: 7.0.0


Comments:
Issue unresolved, see [4000](https://github.com/huggingface/datasets/issues/4009#issue-1179658611)
set_format does not work with multi dimension tensor

## Describe the bug
set_format only transforms the last dimension of a multi-dimension list to tensor

## Steps to reproduce the bug
```python
import torch
from datasets import Dataset

ds = Dataset.from_dict({"A": [torch.rand((2, 2))]})
# ds = Dataset.from_dict({"A": [np.random.rand(2, 2)]})  # => same result
ds = ds.with_format("torch")
print(ds[0])
```

## Expected results
```
{'A': [tensor([[0.6689, 0.1516], [0.1403, 0.5567]])]}
```

## Actual results
```
{'A': [tensor([0.6689, 0.1516]), tensor([0.1403, 0.5567])]}
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- datasets version: 2.0.0
- Platform: Mac OSX
- Python version: 3.8.12
- PyArrow version: 7.0.0

Comments:
Hi! Use the `ArrayXD` feature type (where X is the number of dimensions) to get correctly formated tensors. So in your case, define the dataset as follows :
```python
ds = Dataset.from_dict({"A": [torch.rand((2, 2))]}, features=Features({"A": Array2D(shape=(2, 2), dtype="float32")}))
```


Hi @mariosasko I'm facing the same issue and the only work around I've found so far is to convert my `DatasetDict` to a dictionary and then create new objects with `Dataset.from_dict`.
```
dataset = load_dataset("my_dataset.py")
dataset = dataset.map(lambda example: blabla(example))
dict_dataset_test = dataset["test"].to_dict()
...
dataset_test = Dataset.from_dict(dict_dataset_test, features=Features(features))
```
However, converting a `Dataset` object to a dict takes quite a lot of time and memory... Is there a way to directly create an `Array2D` without having to transform the original `Dataset` to a dict?

Hi! Yes, you can directly pass the `Features` dictionary as `features` in `map` to cast the column to `Array2D`:
```python
dataset = dataset.map(lambda example: blabla(example), features=Features(features))
```
Or you can use `cast` after `map` to do that:
```python
dataset = dataset.map(lambda example: blabla(example))
dataset = dataset.cast(Features(features))
```

Fantastic thank you @mariosasko
The first option you suggested is indeed way faster 😃
Use audio feature in ASR task template

The AutomaticSpeechRecognition task template is outdated: it still uses the file path column as input instead of the audio column.

I changed that and updated all the datasets as well as the tests.

The only community dataset that will need to be updated is `facebook/multilingual_librispeech`. It has almost zero usage unfortunately (probably because users load the duplicate `multilingual_librispeech` directly instead), but it means we can update it.

(this makes me think that we should deprecate `multilingual_librispeech` it and redirect users to `facebook/multilingual_librispeech`).

This PR is also useful for the AudioFolder in https://github.com/huggingface/datasets/pull/3963

Comments:
_The documentation is not available anymore as the PR was closed or merged._
How to use generate this multitask dataset for SQUAD? I am getting a value error. 

## Dataset viewer issue for 'squad_multitask*'

**Link:** https://huggingface.co/datasets/vershasaxena91/squad_multitask

*short description of the issue*

I am trying to generate the multitask dataset for squad dataset. However, gives the error in dataset explorer as well as my local machine. 
I tried the command: dataset = load_dataset("vershasaxena91/squad_multitask", 'highlight_qg_format')
Error:
Status code:   400
Exception:     TypeError
Message:       argument of type 'Value' is not iterable

Kindly advice. 


Comments:
Hi! Replacing `nlp.<obj>` with `datasets.<obj>` in the script should fix the problem. `nlp` has been renamed to `datasets` more than a year ago, so please use `datasets` instead to avoid weird issues.

Thank You! Was able to solve with the help of this.

But I request you to please fix the same in the dataset hub explorer as well...

May I ask how to get this dataset?
Fix Audio.encode_example() when writing an array

Closes  #3996

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@albertvillanova do you think [this line](https://github.com/huggingface/datasets/pull/3998/files#diff-58e348f6e4deaa5f3119e420a5d48ebb82875a78c28628831748fb54f59b2c78R67) is enough? that's why we missed this bug, we didn't check this case
Sync Features dictionaries

This PR adds a wrapper to the `Features` class to keep the secondary dict, `_column_requires_decoding`, aligned with the main dict (as discussed in https://github.com/huggingface/datasets/pull/3723#discussion_r806912731).

A more elegant approach would be to subclass `UserDict` and override `__setitem__` and `__delitem__`, but this PR doesn't implement it for the following reasons:
* it requires replacing all occurrences of `isinstance(obj, dict)` with `isinstance(obj, Mapping)`, which is five times slower than `isinstance(obj, dict)` on my machine, in `features.py`
* is a breaking change, i.e., `isinstance(Features(...), dict)` would return `False` after it
* IMO, it makes sense to be consistent in the user-facing API and subclass either `dict` or `UserDict`. The problem with the latter is that it can't be used for `DatasetDict` because `DatasetDict` exposes the `data` property, which is also used by `UserDict`, so this would result in a collision.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Close `PIL.Image` file handler in `Image.decode_example`

Closes the file handler of the PIL image object in `Image.decode_example` to avoid the `Too many open files` error.

To pass [the image equality checks](https://app.circleci.com/pipelines/github/huggingface/datasets/10774/workflows/d56670e6-16bb-4c64-b601-a152c5acf5ed/jobs/65825) in CI, `Image.decode_example` calls `image.load()` regardless of how the image object is created (not only for the `PIL.Image.open(local_path)` case). This is needed because `load()` sets the `readonly` attribute of a `PIL.Image` object to 0 (it's 1 after `PIL.Image.open(file_like)`), and in the older PIL versions (only fixed on main), that attribute is considered in `PIL.Image.__eq__`. More info can be found here: https://github.com/python-pillow/Pillow/issues/5926.

Fix #3985 





Comments:
_The documentation is not available anymore as the PR was closed or merged._
Streaming dataset + interleave + DataLoader hangs with multiple workers

## Describe the bug
Interleaving multiple iterable datasets that use `load_dataset` on streaming mode hangs when passed to `torch.utils.data.DataLoader` with multiple workers.

## Steps to reproduce the bug
```python
from datasets import interleave_datasets, load_dataset
from torch.utils.data import DataLoader

en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)
fr_dataset = load_dataset('oscar', "unshuffled_deduplicated_fr", split='train', streaming=True)
it_dataset = load_dataset('oscar', "unshuffled_deduplicated_it", split='train', streaming=True)
de_dataset = load_dataset('oscar', "unshuffled_deduplicated_de", split='train', streaming=True)
multilingual_dataset = interleave_datasets([en_dataset, fr_dataset, de_dataset, it_dataset])
multilingual_dataset = multilingual_dataset.with_format('torch')

next(iter(multilingual_dataset))  # works fairly fast

dataloader = DataLoader(multilingual_dataset, batch_size=8, num_workers=4)
for batch in dataloader:
    print(len(batch))  # prints nothing after 30 min of waiting

dataloader = DataLoader(multilingual_dataset, batch_size=8, num_workers=0)
for batch in dataloader:
    print(len(batch))  # prints right away

```

## Expected results
It should be able to iterate the dataset with multiple workers.

## Actual results
Prints with results with `next(iter(multilingual_dataset)) ` and `num_workers=0`  but it prints nothing with `num_workers=4` or any number above 0.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.1.dev0
- `pytorch` version: 1.10.0+cu113
- Python version: 3.7
- PyArrow version: 6.0.1


Comments:
Same thing occurs when streaming files loaded from disk.

Hi ! Thanks for reporting, could this be related to https://github.com/huggingface/datasets/issues/3950 ?

Currently streaming datasets only works in single process, but we're working on having in work in distributed setups as well :) (EDIT: done)

Hi, thanks for your reply. It seems related :)

+1

Please update `datasets` if you're having this issue. What version are you using ?
Image column is not decoded in map when using with with_transform

## Describe the bug
Image column is not _decoded_ in **map** when using with `with_transform`

## Steps to reproduce the bug
```python
from datasets import Image, Dataset

def add_C(batch):
    batch["C"] = batch["A"]
    return batch

ds = Dataset.from_dict({"A": ["image.png"]}).cast_column("A",  Image())
ds = ds.with_transform(lambda x: x)  # <= This line causes the problem
ds = ds.map(add_C, batched=True)
print(ds[0])
```

## Expected results
```
{'C': <PIL.PngImagePlugin.PngImageFile>,  ...}
```

## Actual results
```
{'C': {'bytes': None, 'path': 'image.png'}, ...}
```

If we remove the `with_transform` line, we get the expected result.

## Environment info
- `datasets` version: 2.0.0
- Platform: Mac OSX
- Python version: 3.8.12
- PyArrow version: 7.0.0


Comments:
Hi! This behavior stems from this line: https://github.com/huggingface/datasets/blob/799b817d97590ddc97cbd38d07469403e030de8c/src/datasets/arrow_dataset.py#L1919
Basically, the `Image`/`Audio` columns are decoded only if the `format_type` attribute is `None` (`set_format`/`with_format` and `set_transform`/`with_transform` assign a non-`None` value to it) and the  `input_columns` param is not specified (see https://github.com/huggingface/datasets/issues/3756). We will remove these limitations soon.
Add Lung Image Database Consortium image collection (LIDC-IDRI) dataset

## Adding a Dataset

- **Name:** *Lung Image Database Consortium image collection (LIDC-IDRI)*
- **Description:** *Consists of diagnostic and lung cancer screening thoracic computed tomography (CT) scans with marked-up annotated lesions. It is a web-accessible international resource for development, training, and evaluation of computer-assisted diagnostic (CAD) methods for lung cancer detection and diagnosis. Initiated by the National Cancer Institute (NCI), further advanced by the Foundation for the National Institutes of Health (FNIH), and accompanied by the Food and Drug Administration (FDA) through active participation, this public-private partnership demonstrates the success of a consortium founded on a consensus-based process.*
- **Data:** *[link to the Github repository or current dataset location](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI)*
- **Motivation:** *Key dataset in the healthcare community*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

FYI @osanseviero @abidlabs 

Comments:
Improve AutomaticSpeechRecognition task template

**Is your feature request related to a problem? Please describe.**
[AutomaticSpeechRecognition task template](https://github.com/huggingface/datasets/blob/master/src/datasets/tasks/automatic_speech_recognition.py) is outdated as it uses path to audiofile as an audio column instead of a Audio feature itself (I guess it's because Audio feature didn't exist at the time this template was created).

**Describe the solution you'd like**
Change audio columns from string path to Audio feature.


Comments:
There is an open PR to do that: #3364. I just haven't had time to finish it... 

> There is an open PR to do that: #3364. I just haven't had time to finish it...

😬 thanks...
Remove old wikipedia leftovers

After updating Wikipedia dataset, remove old wikipedia leftovers from doc.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

> This makes me think we shouldn't advise the use of load_dataset in dataset scripts, since it doesn't guarantee that the cache will work as expected (the cache directory is not set correctly, and the required disk space for downloaded files is not recorded)

@lhoestq, do you think it could be a good idea to add a comment in this script WARNING that using load_dataset in a script is not good practice and that people should avoid using that script as a template to create other scripts? 

good idea ! :)
[image feature] Too many files open error when image feature is returned as a path

## Describe the bug

PR in context: #3967. If I load the dataset in this PR (TextVQA), and do a simple list comprehension on the dataset, I get `Too many open files error`. This is happening due to the way we are loading the image feature when a str path is returned from the `_generate_examples`. Specifically at https://github.com/huggingface/datasets/blob/508eb4ab5d52f590baa677b4f64b1cc069139f7b/src/datasets/features/image.py#L110, we are open the file handle to the image but never closing it. This in my understanding is causing the issue.

## Steps to reproduce the bug
Pull the PR locally and run the following code
```python
from datasets import load_dataset

dataset = load_dataset("./datasets/textvqa")["train"]
data = [item for item in dataset]

# Error happens
```

## Expected results

List comprehension should work smoothly

## Actual results

`Too many open files error`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.1.dev0
- Platform: macOS-12.2-arm64-arm-64bit
- Python version: 3.10.0
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Comments:
Exclude Google Drive tests of the CI

These tests make the CI spam the Google Drive API, the CI now gets banned by Google Drive very often.

I think we can just skip these tests from the CI for now.

In the future we could have a CI job that runs only once a day or once a week for such cases

cc @albertvillanova @mariosasko @severo 

Close #3415

![image](https://user-images.githubusercontent.com/42851186/159283608-fdeca1ac-b57f-4fa3-bf09-6fa5361c494f.png)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

I was thinking exactly the same: running unit tests that request continuously a third-party API is not a good idea.
Add tip on how to speed up loading with ImageFolder

This PR does two things:
* adds a tip on how to speed up loading of a large number of files with ImageFolder (motivated by [this issue](https://github.com/huggingface/datasets/issues/3960))
* replaces the current references to the `Dataset` methods in the Image Processing doc with their fully qualified counterparts (to align it with the Audio Processing doc)

cc @stevhliu 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for adding that tip! 👍 

For the docs syntax, it might be better if we hide the package name/full path to the class or function and only show the name of it. I think it's easier for users to read the function name (eg,`cast_column`) instead of the full path which can be a bit lengthy for some functions like `datasets.IterableDataset.remove_columns` (and if we like this idea, we can align the rest of the docs on it). 

> For the docs syntax, it might be better if we hide the package name/full path to the class or function and only show the name of it. I think it's easier for users to read the function name (eg,cast_column) instead of the full path which can be a bit lengthy for some functions like datasets.IterableDataset.remove_columns (and if we like this idea, we can align the rest of the docs on it).

That's also OK, as long as we are consistent.

@lhoestq @albertvillanova @polinaeterna Which one of these two styles do you prefer?

Agree on hiding `datasets` name. Not sure about hiding class name as it's anyway not visible for users if they use `Dataset.cast_column` or  `IterableDataset.cast_column` when working with their datasets. But I agree that the most important thing is to be consistent :)

Good points! :)

I think it'll be good to show the class name since some functions have different parameters. For example, if users click on `IterableDataset.map` and then `Dataset.map`, they'll see different parameters and have to figure out why (which isn't too difficult I guess lol). But showing the class name avoids any confusion upfront.
Fix google drive streaming for small files

Google drive did another change recently, following #3787 #3843 .

In particular Google Drive now returns 403 for GET requests with `confirm=t` when a files doesn't have a virus warning message. I fixed this by passing `confirm=t` if and only if when there is one (i.e. when status code is 200 for HEAD)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Actually the CI fails because of this
![image](https://user-images.githubusercontent.com/42851186/159281771-78e611b1-6b04-4a87-8324-b6ba2d8c6a6a.png)

It looks like we can't have a proper way to test google drive in the CI right now. Though it seems to work locally if you're not banned. I think I'll just disable those tests for now

this fix will not be included?

No we can't do anything except stop using google drive when possible
Fix main classes reference in docs

Currently the section index (on the page's right side) of the [main classes reference](https://huggingface.co/docs/datasets/master/en/package_reference/main_classes) incorrectly displays `Tensor returned:`, this PR fixes this issue by wrapping code examples in this page with markdown code block.

There are other examples in datasets library having this issue.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3976). All of your documentation changes will be reflected on that endpoint.

Not sure why some section titles end with `[[datasets.xxx]]`, like this: https://moon-ci-docs.huggingface.co/docs/datasets/pr_3976/en/package_reference/main_classes#datasetdict[[datasets.datasetdict]]

Thanks ! I think this has been fixed already in https://github.com/huggingface/datasets/pull/3925 though

I'm closing this one then if it's fine for you
Update many missing tags to dataset README's

I've started to go through the datasets available and noticed that there are 127 datasets that does not have all the tags so I started filling them in; starting with some of the most common and QA datasets

Not 100% certain that the task_id is correct for SuperGLUE

If anyone is browsing the issues and would like to help make Hugging face datasets even more feature complete and awesome, feel free to use this tool I wrote to find the missing tags in the [datacards](https://github.com/Hugging-Face-Supporter/datacards)

Comments:
Add XFUN dataset

This PR adds XFUN dataset.

Home page and repository: https://github.com/doc-analysis/XFUND

Source code: https://github.com/microsoft/unilm/blob/master/layoutlmft/layoutlmft/data/datasets/xfun.py

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Not sure how to generate dummy data.

The downloaded file structure is 

- document file paths
    - (a json file containing all documents info, document images folder)
    - (a json file containing all documents info, document images folder)
    - ...

Hey @mariosasko, thanks for the review. I'm not sure how to suggest these changes to the owner @ranpox, and I did spend some time to write the model card and hope to get it on the official repo. Is that possible?

Since the author is not responding, maybe we can go ahead with this PR ?

Go for it!

On Tue, Apr 12, 2022 at 10:24 AM Quentin Lhoest ***@***.***>
wrote:

> Since the author is not responding, maybe we can go ahead with this PR ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/3974#issuecomment-1096797650>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ATFNL66EVUFWS3P2FOAS7SLVEWBP3ANCNFSM5RFH3MXA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


@qqaatw Do you plan to finish this PR? I can give you some pointers and help you with the code if needed.

@mariosasko Yes, I'll apply all of the suggestions when I have some time.

Thanks for your contribution, @qqaatw.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you propose this changes there to the original repo. Please, feel free to tell us if you need some help.
Adding Roman Urdu Hate Speech dataset

This Pull request will add the Roman Urdu Hate speech Dataset. 

Comments:
@lhoestq can you review when you have some time? Also were the previous CI fails due to the Google Drive tests which were excluded by #3982 ?

> were the previous CI fails due to the Google Drive tests which were excluded by https://github.com/huggingface/datasets/pull/3982 ?

Yes exactly, merging `master` into your branch fixed the CI ;)

_The documentation is not available anymore as the PR was closed or merged._
Cannot preview cnn_dailymail dataset

## Dataset viewer issue for '*cnn_dailymail*'

**Link:** https://huggingface.co/datasets/cnn_dailymail

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Comments:
I guess the cache got corrupted due to a previous issue with Google Drive service.

The cache should be regenerated, e.g. by passing `download_mode="force_redownload"`.

CC: @severo 

Note that the dataset preview uses its own cache, not `datasets`' cache. So `download_mode="force_redownload"` doesn't help. But yes indeed the cache must be refreshed.

The CNN Dailymail dataste is currently hosted on Google Drive, which is an unreliable host and we've had many issues with it. Unless we found another most reliable host for the data, we will keep running into issues from time to time.

At Hugging Face we're not allowed to host the CNN Dailymail data by ourselves AFAIK

Yes @lhoestq, I didn't explain myself well: my previous message was addressed to @severo. 

I remove the tag dataset-viewer, since it's more an issue with the hosting on Google Drive

Sounds good. I was looking for another host of this dataset but couldn't find any (yet)

It seems like the issue is with the streaming mode, not with the hosting:

```python
>>> import datasets
>>> dataset = datasets.load_dataset('cnn_dailymail', name="3.0.0", split="train", streaming=True, download_mode="force_redownload")
Downloading builder script: 9.35kB [00:00, 10.2MB/s]
Downloading metadata: 9.50kB [00:00, 12.2MB/s]
>>> len(list(dataset))
0
>>> dataset = datasets.load_dataset('cnn_dailymail', name="3.0.0", split="train", streaming=False)
Reusing dataset cnn_dailymail (/home/slesage/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234)
>>> len(dataset)
287113
```

Note, in particular, that the streaming mode is failing silently, returning 0 row while I would have expected an exception instead. The result is that the dataset viewer shows `No data` instead of a detailed error.

<img width="1511" alt="Capture d’écran 2022-04-12 à 11 50 46" src="https://user-images.githubusercontent.com/1676121/162935341-d50f1e73-d053-41d4-917f-e79708a0ca23.png">


Well this is because the host (Google Drive) returns a document that is not the actual data, but an error page

Do you think that `datasets` should detect this anyway and throw an exception?

Yes it definitely should ! I don't have the bandwidth to work on this right now though

Indeed, streaming was not supported: tgz archives were not properly iterated.

I've opened a PR to support streaming.

However, keep in mind that Google Drive will keep generating issues from time to time, like 403,...
Cannot preview 'indonesian-nlp/eli5_id' dataset

## Dataset viewer issue for '*indonesian-nlp/eli5_id*'

**Link:** https://huggingface.co/datasets/indonesian-nlp/eli5_id

I can not see the dataset preview.
```
Server Error
Status code:   400
Exception:     Status400Error
Message:       Not found. Maybe the cache is missing, or maybe the dataset does not exist.
```
Am I the one who added this dataset ? Yes


Comments:
Hi @cahya-wirawan, thanks for reporting.

Your dataset is working OK in streaming mode:
```python
In [1]: from datasets import load_dataset
   ...: ds = load_dataset("indonesian-nlp/eli5_id", split="train", streaming=True)
   ...: item = next(iter(ds))
   ...: item
Using custom data configuration indonesian-nlp--eli5_id-9fe728a7e760fb7b

Out[1]: 
{'q_id': '1oy5tc',
 'title': 'dalam sepak bola apa gunanya menyia-nyiakan dua permainan pertama dengan terburu-buru - di tengah - bukan permainan terburu-buru biasa saya mendapatkannya',
 'selftext': '',
 'document': '',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['ccwtgnz', 'ccwtmho', 'ccwt946', 'ccwvj0u'],
  'text': ['Jaga pertahanan tetap jujur, rasakan operan terburu-buru, buka permainan yang lewat. Pelanggaran yang terlalu satu dimensi akan gagal. Dan mereka yang bergegas ke tengah kadang-kadang dapat dibuka lebar-lebar untuk ukuran yard yang besar.',
   'Jika Anda melempar bola sepanjang waktu, maka pertahanan akan beradaptasi untuk selalu menutupi umpan. Dengan melakukan permainan lari sederhana sesekali, Anda memaksa pertahanan untuk tetap dekat dan menjaga dari lari. Terkadang, pelanggaran dapat membuat pertahanan lengah dengan berpura-pura berlari dan membebaskan penerima mereka. Selain itu, Anda tidak perlu mendapatkan yard besar di setiap permainan. Terkadang, paling baik mendapatkan beberapa yard sekaligus. Selama Anda mendapatkan yang pertama, Anda dalam kondisi yang baik.',
   'Dalam kebanyakan kasus, O-Line seharusnya membuat lubang untuk dilalui kembali. Jika Anda menjalankan terlalu banyak permainan ke luar / melempar, pertahanan akan mengejar. Juga, 2 permainan 5 yard memberi Anda satu set down baru.',
   'Saya Anda tidak suka jenis drama itu, tonton CFL. Kami hanya mendapatkan 3 down sehingga Anda tidak bisa menyia-nyiakannya. Lebih banyak lagi yang lewat.'],
  'score': [3, 2, 2, 2]},
 'title_urls': {'url': []},
 'selftext_urls': {'url': []},
 'answers_urls': {'url': []}}
```
Therefore, it should be properly rendered in the previewer. Let me ping @severo to have a look at it.

Thanks @albertvillanova for checking it. Btw, I have another dataset indonesian-nlp/lfqa_id which has the same issue. However, this dataset is still private, is it the reason why the preview doesn't work?

Yes, preview is not supported on private datasets yet. We are working on that though...

Thanks for the confirmation 

Fixed. Thanks for your feedback.
[feat] Add TextVQA dataset

This would be the first classification-based vision-and-language dataset in the datasets library.

Currently, the dataset downloads everything you need beforehand. See the [paper](https://arxiv.org/abs/1904.08920) for more details.

Test Plan:
- Ran the full and the dummy data test locally

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey :) Have you had a chance to continue this PR ? Let me know if you have questions or if I can help

Hey @lhoestq, let me wrap this up soon. I will resolve your comments in next push.
Scores from Index at extra positions are not filtered out

If a FAISS index has fewer records than the requested number of top results (k), then it returns -1 in indices for the additional positions. The get_nearest_examples method only filters out the extra results from the dataset samples. It would be better to filter out extra scores too.

Reference: https://github.com/huggingface/datasets/blob/2.0.0/src/datasets/search.py#L693


Comments:
Hi! Yes, that makes sense! Would you like to submit a PR to fix this?

Created PR https://github.com/huggingface/datasets/pull/3971
The dataset preview is not available for tdklab/Hebrew_Squad_v1.1 dataset

## Dataset viewer issue for 'tdklab/Hebrew_Squad_v1.1'

**Link:**  https://huggingface.co/api/datasets/tdklab/Hebrew_Squad_v1.1?full=true

The dataset preview is not available for this dataset.

Am I the one who added this dataset ? Yes

Comments:
Hi @MatanBenChorin, thanks for reporting.

Please, take into account that the preview may take some time until it properly renders (we are working to reduce this time).

Maybe @severo can give more details on this.

Hi, 
Thank you

Thanks for reporting. We are looking at it and will give updates here.

I imagine the dataset has been moved to https://huggingface.co/datasets/tdklab/Hebrew_Squad_v1, which still has an issue:

```
Server Error

Status code:   400
Exception:     NameError
Message:       name 'HebrewSquad' is not defined
```

The issue is not related to the dataset viewer but to the loading script (cc @albertvillanova @lhoestq @mariosasko)

```python
>>> import datasets as ds
>>> hf_token = "hf_..." # <- required because the dataset is gated
>>> d = ds.load_dataset('tdklab/Hebrew_Squad_v1', use_auth_token=hf_token)
...
NameError: name 'HebrewSquad' is not defined
```

Yes indeed there is an error in [Hebrew_Squad_v1.py:L40](https://huggingface.co/datasets/tdklab/Hebrew_Squad_v1/blob/main/Hebrew_Squad_v1.py#L40)

Here is the fix @MatanBenChorin :

```diff
-         HebrewSquad(
+         HebrewSquadConfig(
```
Add ImageNet Sketch

## Adding a Dataset
- **Name:** ImageNet Sketch
- **Description:** ImageNet-Sketch is a dataset consisting of sketch-like images, that matches the ImageNet classification validation set in categories and scale.
- **Paper:** [Learning Robust Global Representations by Penalizing Local Predictive Power](https://arxiv.org/abs/1905.13549)
- **Data:** https://github.com/HaohanWang/ImageNet-Sketch
- **Motivation:** Allows for evaluating the robustness of vision models.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Can you assign this task to me? @nreimers @mariosasko 

Hi! Sure! Let us know if you need any pointers.
Google BLEU Metric Card

Add metric card for Google BLEU (GLEU) metric

One thing I noticed while writing this up is that, while this metric was made specifically to be better than BLEU at the sentence level instead of the corpus level, the current implementation only allows the calculation of the corpus-level statistic. I think changing this would be a good thing to put on the to do list for the future.

Comments:
A few things that aren't clear for me:
- "Because it performs better on individual sentence pairs as compared to BLEU, Google BLEU has also been used in RL experiments." -- why is this the case? why would that make it more usable for RL? (also, you should put "Reinforcement Learning" explicitly, not just the acronym)
- (Minor issue) -- I put inputs before the first example code, I think that's clearer somehow

Otherwise, it looks great, good job @emibaylor !
BLEU metric card

Add BLEU metric card

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Some thoughts:
- For values, e.g. "Defaults to False", I would put False in code: `False`. Same for : "Defaults to `4`."
- I would put the following remark in "Limitations": 
> "BLEU's output is always a number between 0 and 1. This value indicates how similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts. Few human translations will attain a score of 1, since this would indicate that the candidate is identical to one of the reference translations. For this reason, it is not necessary to attain a score of 1. Because there are more opportunities to match, adding additional reference translations will increase the BLEU score."

- Add some values from the original BLEU paper (https://aclanthology.org/P02-1040.pdf)
Fix comet metric

The COMET metric has been broken for a while since big breaking changes happened. We did not catch them in the CI because the slow test mocks the download_model function that was changed.

This PR fixes the metric, updates the download_model mock and updates the doctest.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Finally I'm done updating the dependencies ^^'

cc @sashavor can you review my changes in the metric card please ?

Looks good to me! Just fixed a tiny typo :wink: 

Thanks !
reddit_tifu dataset: Checksums didn't match for dataset source files

## Describe the bug
When loading the reddit_tifu dataset, it throws the exception "Checksums didn't match for dataset source files"


## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset

print(datasets.__version__)

# load_dataset('billsum')

load_dataset('reddit_tifu', 'short')
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0
- Platform: mac os
- Python version: Python 3.7.6
- PyArrow version: 3.0.0


Comments:
Hi @XingxingZhang, 

We have already fixed this. You should update `datasets` version to at least 1.18.4:
```shell
pip install -U datasets
```
And then force the redownload:
```python
load_dataset("...", download_mode="force_redownload")
```

Duplicate of:
- #3773

thanks @albertvillanova . by upgrading to 1.18.4 and using `load_dataset("...", download_mode="force_redownload")` fixed 
 the bug.

using the following as you suggested in another thread can also fixed the bug
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
```


The latter solution (installing from GitHub) was proposed because the fix was not released yet. But last week we made the 1.18.4 patch release (with the fix), so no longer necessary to install from GitHub.

You can now install from PyPI, as usual:
```shell
pip install -U datasets
```
billsum dataset: Checksums didn't match for dataset source files:

## Describe the bug

When loading the `billsum` dataset, it throws the exception "Checksums didn't match for dataset source files"

```
  File "virtualenv_projects/codex/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1g89WgFHMRbr4QrvA0ngh26PY081Nv3lx']
```

## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset

print(datasets.__version__)

load_dataset('billsum')
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0

- Platform: mac os
- Python version: Python 3.7.6
- PyArrow version: 3.0.0






Comments:
Hi @XingxingZhang, thanks for reporting.

This was due to a change in Google Drive service:
- #3786 

We have already fixed it:
- #3787

You should update `datasets` version to at least 1.18.4:
```shell
pip install -U datasets
```
And then force the redownload:
```python
load_dataset("...", download_mode="force_redownload")
```

thanks @albertvillanova 

@albertvillanova 
YOU Said: pip install git+ https://github.com/huggingface/datasets.git Then set: dataset=load_dataset ("multinews", download_mode="force-redownload"). I changed the ’datautils‘ file according to this setting: traindata=load_dataset (path='wikitext ', name='wikitext-2-raw v1', split='train ', download_mode="force-redownload")
Testdata=load_dataset (path='wikitext ', name='wikitext-2-raw v1', split='test ', download_mode="force-redownload")
then the bug is
![image](https://github.com/huggingface/datasets/assets/149936473/ee956e8f-e6f1-46bf-b514-0a7a0a0e0e37)
![image](https://github.com/huggingface/datasets/assets/149936473/f1318686-942c-4341-a61d-9be7a4b5747a)
I have tried both versions
![image](https://github.com/huggingface/datasets/assets/149936473/d3f0d786-304b-4596-83c6-49c3cac58aad)
Missing languages in lvwerra/github-code dataset

Hi,

I'm working with the github-code dataset. First of all, thank you for creating this amazing dataset!
I've noticed that two languages are missing from the dataset: TypeScript and Scala. 
Looks like they're also omitted from the query you used to get the original code.
Are there any plans to add them in the future?

Thanks!

Comments:
Thanks for contacting @Eytan-S.

I think @lvwerra could better answer this. 

That seems to be an oversight - I originally planned to include them in the dataset and for some reason they were in the list of languages but not in the query. Since there is an issue with the deduplication step I'll rerun the pipeline anyway and will double check the query.

Thanks for reporting this @Eytan-S!

Can confirm that the two languages are indeed missing from the dataset. Here are the file counts per language:
```Python
{'Assembly': 82847,
 'Batchfile': 236755,
 'C': 14127969,
 'C#': 6793439,
 'C++': 7368473,
 'CMake': 175076,
 'CSS': 1733625,
 'Dockerfile': 331966,
 'FORTRAN': 141963,
 'GO': 2259363,
 'Haskell': 340521,
 'HTML': 11165464,
 'Java': 19515696,
 'JavaScript': 11829024,
 'Julia': 58177,
 'Lua': 576279,
 'Makefile': 679338,
 'Markdown': 8454049,
 'PHP': 11181930,
 'Perl': 497490,
 'PowerShell': 136827,
 'Python': 7203553,
 'Ruby': 4479767,
 'Rust': 321765,
 'SQL': 655657,
 'Scala': 0,
 'Shell': 1382786,
 'TypeScript': 0,
 'TeX': 250764,
 'Visual Basic': 155371}
 ```

@Eytan-S check out v1.1 of the `github-code` dataset where issue should be fixed:

|    | Language     |File Count| Size (GB)|
|---:|:-------------|---------:|-------:|
|  0 | Java         | 19548190 | 107.7  |
|  1 | C            | 14143113 | 183.83 |
|  2 | JavaScript   | 11839883 |  87.82 |
|  3 | HTML         | 11178557 | 118.12 |
|  4 | PHP          | 11177610 |  61.41 |
|  5 | Markdown     |  8464626 |  23.09 |
|  6 | C++          |  7380520 |  87.73 |
|  7 | Python       |  7226626 |  52.03 |
|  8 | C#           |  6811652 |  36.83 |
|  9 | Ruby         |  4473331 |  10.95 |
| 10 | GO           |  2265436 |  19.28 |
| 11 | TypeScript   |  1940406 |  24.59 |
| 12 | CSS          |  1734406 |  22.67 |
| 13 | Shell        |  1385648 |   3.01 |
| 14 | Scala        |   835755 |   3.87 |
| 15 | Makefile     |   679430 |   2.92 |
| 16 | SQL          |   656671 |   5.67 |
| 17 | Lua          |   578554 |   2.81 |
| 18 | Perl         |   497949 |   4.7  |
| 19 | Dockerfile   |   366505 |   0.71 |
| 20 | Haskell      |   340623 |   1.85 |
| 21 | Rust         |   322431 |   2.68 |
| 22 | TeX          |   251015 |   2.15 |
| 23 | Batchfile    |   236945 |   0.7  |
| 24 | CMake        |   175282 |   0.54 |
| 25 | Visual Basic |   155652 |   1.91 |
| 26 | FORTRAN      |   142038 |   1.62 |
| 27 | PowerShell   |   136846 |   0.69 |
| 28 | Assembly     |    82905 |   0.78 |
| 29 | Julia        |    58317 |   0.29 |

Thanks @lvwerra.
Fix Wikipedia version and re-add tests

To keep backward compatibility when loading using "wikipedia" dataset ID (https://huggingface.co/datasets/wikipedia), we have created the pre-processed data for the same languages we were offering before, but with updated date "20220301":
- de
- en
- fr
- frr
- it
- simple

These pre-processed data can be accessed, e.g.:
```python
ds = load_dataset("wikipedia", "20220301.frr", split="train")
```

The next step will be to offer the pre-processed data for many other languages, but when loading using "wikimedia/wikipedia": https://huggingface.co/datasets/wikimedia/wikipedia

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3936). All of your documentation changes will be reflected on that endpoint.
Add align_labels_with_mapping docs

This PR documents the `align_labels_with_mapping` function to ensure predicted labels are aligned with the dataset, or to assign a different mapping of labels to ids (requested by @mariosasko 🎉 ).

For this specific code sample, the current dataset has a `mixed` label that the original [dataset](https://huggingface.co/datasets/poem_sentiment#data-fields) didn't. Is there a way to remove this label so it is completely aligned with the original dataset mappings? Otherwise, I'll just leave it as it is.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Load a local dataset twice

## Describe the bug
Load a local "dataset" composed of two csv files twice.

## Steps to reproduce the bug
Put the two joined files in a repository named "Data".
Then in python:

import datasets as ds
ds.load_dataset('Data', data_files = {'file1.csv', 'file2.csv'})

## Expected results
Should give something like (because files have only one data row):
Title, clicks
Truc et astuce, 123
Machin, 12

## Actual results
Gives 
Title, clicks
Truc et astuce, 123
Machin, 12
Truc et astuce, 123
Machin, 12

## Environment info
[file1.csv](https://github.com/huggingface/datasets/files/8256322/file1.csv)
[file2.csv](https://github.com/huggingface/datasets/files/8256323/file2.csv)
- `datasets` version: 2.0.0
- Platform: Linux-5.4.0-65-generic-x86_64-with-glibc2.10
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.1

Comments:
Hi @caush, thanks for reporting:

In order to load local CSV files, you can use our "csv" loading script: https://huggingface.co/docs/datasets/loading#csv
```python
dataset = load_dataset("csv", data_files=["data/file1.csv", "data/file2.csv"])
```
OR:
```python
dataset = load_dataset("csv", data_dir="data")
```

Alternatively, you may also use:
```python
dataset = load_dataset("data")