Add raw data files to the Hub with GitHub LFS for canonical dataset

Hi @zlucia, I would actually suggest hosting the dataset as a huggingface.co-hosted dataset.

The only difference with a "canonical"/legacy dataset is that it's nested under an organization (here `stanford` or `stanfordnlp` for instance – completely up to you) but then you can upload your data using git-lfs (unlike "canonical" datasets where we don't host the data)

Let me know if this fits your use case!

cc'ing @osanseviero @lhoestq and rest of the team 🤗

Hi @zlucia,

As @julien-c pointed out, the way to store/host raw data files in our Hub is by using what we call "community" datasets:
- either at your personal namespace: `load_dataset("zlucia/casehold")`
- or at an organization namespace: for example, if you create the organization `reglab`, then `load_dataset("reglab/casehold")`

Please note that "canonical" datasets do not normally store/host their raw data at our Hub, but in a third-party server. For "canonical" datasets, we just host the "loading script", that is, a Python script that downloads the raw data from a third-party server, creates the HuggingFace dataset from it and caches it locally.

In order to create an organization namespace in our Hub, please follow this link: https://huggingface.co/organizations/new

There are already many organizations at our Hub (complete list here: https://huggingface.co/organizations), such as:
- Stanford CRFM: https://huggingface.co/stanford-crfm
- Stanford NLP: https://huggingface.co/stanfordnlp
- Stanford CS329S: Machine Learning Systems Design: https://huggingface.co/stanford-cs329s

Also note that you in your organization namespace:
- you can add any number of members
- you can store both raw datasets and models, and those can be immediately accessed using `datasets` and `transformers`

Once you have created an organization, these are the steps to upload/host a raw dataset:
- The no-code procedure: https://huggingface.co/docs/datasets/upload_dataset.html
- Using the command line (terminal): https://huggingface.co/docs/datasets/share.html#add-a-community-dataset

Please, feel free to ping me if you have any further questions or need help.

Ah I see, I think I was unclear whether there were benefits to uploading a canonical dataset vs. a community provided dataset. Thanks for clarifying. I'll see if we want to create an organization namespace and otherwise, will upload the dataset under my personal namespace.

I'm interested in sharing the CaseHOLD dataset (https://arxiv.org/abs/2104.08671) as a canonical dataset on the HuggingFace Hub and would like to add the raw data files to the Hub with GitHub LFS, since it seems like a more sustainable long term storage solution, compared to other storage solutions available to my team. From what I can tell, this option is not immediately supported if one follows the sharing steps detailed here: [https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset](https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset), since GitHub LFS is not supported for public forks. Is there a way to request this? Thanks!


Add Image feature

Few additional comments:
* the current API doesn't meet the requirements mentioned in #3145 (e.g. image mime-type). However, this will be doable soon as we also plan to store image bytes alongside paths in arrow files (see https://github.com/huggingface/datasets/pull/3129#discussion_r738426187). Then, PIL can return the correct mime-type:
   ```python
   from PIL import Image
   import io

   mimetype = Image.open(io.BytesIO(image_bytes)).get_format_mimetype()
   ```
   I plan to add this change in a separate PR.
*  currently, I'm returning an `np.ndarray` object after decoding for consistency with the Audio feature. However, the vision models from Transformers prefer an `Image` object to avoid the `Image.fromarray` call in the corresponding feature extractors (see [this warning](https://huggingface.co/transformers/master/model_doc/vit.html#transformers.ViTFeatureExtractor.__call__) in the Transformers docs) cc @NielsRogge

So I'm not entirely sure whether to return only a NumPy array, only a PIL Image, or both when decoding. The last point worries me because we shouldn't provide an API that leads to a warning in Transformers (in the docs, not in code :)). At the same time, it makes sense to preserve consistency with the Audio feature and return a NumPy array.

That's why I would appreciate your opinions on this.

That is a good question. Also pinging @nateraw .

Currently we only support returning numpy arrays because of numpy/tf/torch/jax formatting features that we have, and to keep things simple. See the [set_format docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.set_format) for more info

I don't think centering the discussion on what ViT expects is good, as the vision Transformers model are still in an experimental stage and we can adapt those depending on what you do here :-).

IMO, the discussion should revolve on what a user will want to do with a vision dataset, and they will want to:
- lazily decode their images
- maybe apply data augmentation (for the training set)
- resize to a fixed shape for batching

The libraries that provide step 2 and 3 either use PIL (thinking torchvision) or cv2 (thinking albumentations). NumPy does not have any function to resize an image or do basic data augmentation (like a rotate) so I think it shouldn't be the default format for an image dataset, PIL or cv2 (in an ideal world with the ability to switch between the two depending on what the users prefer) would be better.

Side note: I will work on the vision integration in Transformers with Niels next month so please keep me in the loop for those awesome new vision features!

@sgugger I completely agree with you, especially after trying to convert the `run_image_classification` script from Transformers to use this feature. The current API doesn't seem intuitive there due to the torchvision transforms, which, as you say, prefer PIL over NumPy arrays.

So the default format would return `Image` (PIL) / `np.ndarray` (cv2) and `set_format(numpy/tf/pt)` would return image tensors if I understand you correctly. IMO this makes a lot more sense (and flexibility) than the current API.

Also, one additional library worth mentioning here is AugLy which supports image file paths and `PIL.Image.Image` objects.

That's so nice !

Also I couldn't help myself so I've played with it already ^^
I was agreeably surprised that with minor additions I managed to even allow this, which I find very satisfactory:
```python
import PIL.Image
from datasets import Dataset

path = "docs/source/imgs/datasets_logo_name.jpg"

dataset = Dataset.from_dict({"img": [PIL.Image.open(path)]})
print(dataset.features)
# {'img': Image(id=None)}
print(dataset[0]["img"])
# <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1200x300 at 0x129DE4AC8>
```

Let me know if that's a behavior you'd also like to see

EDIT: just pushed my changes on a branch, you can see the diff [here](https://github.com/mariosasko/datasets-1/compare/add-image-feature...huggingface:image-type-inference) if you want

Thanks, @lhoestq! I like your change. Very elegant indeed.

P.S. I have to write a big comment that explains all the changes/things left to consider. Will do that in the next few days!

I'm marking this PR as ready for review.

Thanks to @sgugger's comment, the API is much more flexible now as it decodes images (lazily) as `PIL.Image.Image` objects and supports transforms directly on them.

Also, we no longer return paths explicitly (previously, we would return `{"path": image_path, "image": pil_image}`) for the following reasons:
* what to return when reading an image from an URL or a NumPy array. We could set `path` to `None` in these situations, but IMO we should avoid redundant information.
* returning a dict doesn't match nicely with the requirement of supporting image modifications - what to do if the user modifies both the image path and the image

(Btw, for the images stored locally, you can access their paths with `dset[idx]["image"].filename`, or by avoiding decoding with `paths = [ex["path"] for ex in dset]`. @lhoestq @albertvillanova WDYT about having an option to skip decoding for complex features, e. g. `Audio(decode=False)`? This way, the user can easily access the underlying data.)

Examples of what you can do:
```python
# load local images
dset = Dataset.from_dict("image": [local_image_path], features=Features({"images": Image()}))
# load remote images (we got this for free by adding support for streaming)
dset = Dataset.from_dict("image": [image_url], features=Features({"images": Image()}))
# from np.ndarray
dset = Dataset.from_dict({"image": [np.array(...)]}, features=Features({"images": Image()}))
# cast column
dset = Dataset.from_dict({"image": [local_image_path]})
dset.cast_column("image", Image())

# automatic type inference
dset = Dataset.from_dict({"image": [PIL.Image.open(local_image_path)]})

# transforms
def img_transform(example):
     ...
     example["image"] = transformed_pil_image_or_np_ndarray
     return example
dset.map(img_trnasform)

# transform that adds a new column with images (automatic inference of the feature type)
dset.map(lambda ex: {"image_resized": ex["image"].resize((100, 100))})
print(dset.features["image_resized"]) # will print Image()
```

Some more cool features:
* We store the image filename (`pil_image.filename`) whenever possible to avoid costly conversion to bytes
* if possible, we use native compression when encoding images. Otherwise, we fall back to the lossless PNG format (e.g. after image ops or when storing NumPy arrays)

Hints to make reviewing easier:
* feel free to ignore the extension type part because it's related to PyArrow internals.
* also, let me know if we are too strict/ too flexible in terms of types the Image feature can encode/decode. Hints:
   * `encode_example` handles encoding during dataset generation (you can think of it as `yield key, features.encode_example(example)`)
   * `objects_to_list_of_image_dicts` handles encoding of returned examples in `map`

P.S. I'll fork the PR branch and start adding the Image feature to the existing image datasets (will also update the `ImageClassification` template while doing that).

> WDYT about having an option to skip decoding for complex features, e. g. Audio(decode=False)?

Yes definitely, also I think it could be useful for the dataset viewer to not decode the data but instead return either the bytes or the (possibly chained) URL. cc @severo

We want to merge this today/tomorrow, so I'd really appreciate your reviews @sgugger @nateraw.

Also, you can test this feature on the existing image datasets (MNIST, beans, food101, ...) by installing `datasets` from the PR branch:
```
pip install git+https://github.com/huggingface/datasets.git@adapt-image-datasets
```

Thanks for the review @nateraw!

1. This is a copy of your notebook with the fixed map call: https://colab.research.google.com/gist/mariosasko/e351a717682a9392ca03908e65a2600e/image-feature-demo.ipynb
   (Sorry for misleading you with the map call in my un-updated notebook)
   Also, we can avoid this cast by trying to infer the type of the column (`"pixel_values"`) returned by the image feature extractor (we are already doing something similar for the columns with names: `"attention_mask"`, `"input_ids"`, ...). I plan to add this QOL improvement soon.
2. It should work OK even without updating Pillow and PyArrow (these two libraries are pre-installed in Colab, so updating them requires a restart of the runtime).
     > I noticed an error that I'm guessing you ran into when I tried using the older version

    Do you recall which type of error it was because everything works fine on my side if I run the notebooks with the lowest supported version of Pillow (`6.2.1`)?

Thanks for playing with it @nateraw and for sharing your notebook, this is useful :)

I think this is ready now, congrats @mariosasko !

Love this feature and hope to release soon!

Adds the Image feature. This feature is heavily inspired by the recently added Audio feature (#2324). Currently, this PR is pretty simple.

Some considerations that need further discussion:
* I've decided to use `Pillow`/`PIL` as the image decoding library. Another candidate I considered is `torchvision`, mostly because of its `accimage` backend, which should be faster for loading `jpeg` images than `Pillow`. However, `torchvision`'s io module only supports png and jpeg images, has `torch` as a hard dependency, and requires magic to work with image bytes ( `torch.ByteTensor(torch.ByteStorage.from_buffer(image_bytes)))`).
* Currently, I'm converting `PIL`'s `Image` type to `np.ndarray`. The vision models in Transformers such as ViT prefer the raw `Image` type and not the decoded tensors, so there is a small overhead due to [this conversion](https://github.com/huggingface/transformers/blob/3e8761ab8077e3bb243fe2f78b2a682bd2257cf1/src/transformers/image_utils.py#L62-L73). IMO this is justified to keep this part aligned with the Audio feature, which also returns `np.ndarray`. What do you think?
* Still have to work on the channel decoding logic:
    * PyTorch prefers the channel-first ordering (C, H, W); TF and Flax the channel-last ordering (H, W, C). One cool feature would be adjusting the channel order based on the selected formatter (`torch`, `tf`, `jax`).
    * By default, `Image.open` returns images of shape (H, W, C). However, ViT's feature extractor expects the format (C, H, W) if the image is passed as an array (explained [here](https://huggingface.co/transformers/model_doc/vit.html#transformers.ViTFeatureExtractor.__call__)), so I'm more inclined to the format (C, H, W). Which one do you prefer, (C, H, W) or (H, W, C)?
* Are there any options you'd like to see? (the user could change those via `cast_column`, such as `sampling_rate` in the Audio feature)

TODOs:
* [x] tests
* in subsequent PRs:
  * docs - a section in the docs, which gives some additional info on the Image and Audio feature and compares them to
  `ArrayND`
  * streaming (waiting for #3129 and #3133 to get merged first)
  * update the image tasks and the datasets to use the new feature
  * Image/Audio formatting

[Colab Notebook](https://colab.research.google.com/drive/1mIrTnqTVkWLJWoBzT1ABSe-LFelIep1c?usp=sharing) where you can play with this feature.

I'm also adding a link to the [Image](https://github.com/tensorflow/datasets/blob/7ac7d506488d46038a5854961d068926b3f93c7f/tensorflow_datasets/core/features/image_feature.py#L155) feature in TFDS because one of our goals is to parse TFDS scripts eventually, so our Image feature has to (at least) support all the formats theirs does.
Feel free to cc anyone who might be interested.

P.S. Please ignore the changes in the `datasets/**/*.py` files 😄.


`datasets-cli test` should work with datasets without scripts

enhancement

New feature or request

> It would be really useful to be able to run `datasets-cli test`for datasets that don't have scripts attached to them (whether the datasets are private or not).
>
> I wasn't able to run the script for a private test dataset that I had created on the hub (https://huggingface.co/datasets/huggingface/DataMeasurementsTest/tree/main) -- although @lhoestq came to save the day!

why don't you try to share that info with people, so you can also save some days.

Hi ! You can run the command if you download the repository
```
git clone https://huggingface.co/datasets/huggingface/DataMeasurementsTest
```
and run the command
```
datasets-cli test DataMeasurementsTest/DataMeasurementsTest.py
```

(though on my side it doesn't manage to download the data since the dataset is private ^^)

> Hi ! You can run the command if you download the repository
>
> ```
> git clone https://huggingface.co/datasets/huggingface/DataMeasurementsTest
> ```
>
> and run the command
>
> ```
> datasets-cli test DataMeasurementsTest/DataMeasurementsTest.py
> ```
>
> (though on my side it doesn't manage to download the data since the dataset is private ^^)

Hi! Thanks for the info.
git cannot find the repository. Do you know if they have depreciated these tests and created a new one?

> I think it's become private, but feel free to try with any other dataset like `lhoestq/test` for example at `https://huggingface.co/datasets/lhoestq/test`

your example repo and this page `https://huggingface.co/docs/datasets/add_dataset.html` helped me to solve.. thanks a lot

It would be really useful to be able to run `datasets-cli test`for datasets that don't have scripts attached  to them (whether the datasets are private or not).

I wasn't able to run the script for a private test dataset that I had created on the hub (https://huggingface.co/datasets/huggingface/DataMeasurementsTest/tree/main) -- although @lhoestq came to save the day!


Add riddle_sense dataset

@lhoestq
I address all the comments, I think. Thanks!

The five test fails are unrelated to this PR and fixed on master so we can ignore them

Adding a new dataset for QA with riddles. I'm confused about the tagging process because it looks like the streamlit app loads data from the current repo, so is it something that should be done after merging or off my fork?


Better error msg if `len(predictions)` doesn't match `len(references)` in metrics

Can't test this now but it may be a good improvement indeed.

I added a function, but it only works with the `list` type. For arrays/tensors, we delegate formatting to the frameworks.

Improve the error message in `Metric.add_batch` if `len(predictions)` doesn't match `len(references)`.

cc: @BramVanroy (feel free to test this code on your examples and review this PR)


Make inspect.get_dataset_config_names always return a non-empty list

This PR is already working (although not very beautiful; see below): the idea was to have the `DatasetModule.builder_kwargs` accessible from the `builder_cls`, so that this can generate the default builder config (at the class level, without requiring the builder to be instantiated).

I have a plan for a follow-up refactoring (same functionality, better implementation, much nicer), but I think we could already merge this, so that @severo can test it in the datasets previewer and report any potential issues.

Yes @lhoestq you are completely right. Indeed I was exclusively using `builder_cls.kwargs` to get the community dataset `name` (nothing else): "lhoestq___demo1"

See et: https://github.com/huggingface/datasets/pull/3159/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R413-R415

In your example, the `name` I was getting from `builder_cls.kwargs` was:
```python
{"name": "lhoestq___demo1",...}
```

I'm going to refactor all the approach... as I only need the name for this specific case ;)

I think this makes more sense now, @lhoestq @severo 😅

Make all named configs cases, so that no special unnamed config case needs to be handled differently.

Fix #3135.


Fix string encoding for Value type

Some metrics have `string` features but currently it fails if users pass integers instead. Indeed feature encoding that handles the conversion of the user's objects to the right python type is missing a case for `string`, while it already works as expected for integers, floats and booleans

Here is an example code that didn't work previously, but that works with this fix:
```python
import datasets

# Note that 'id' is an integer while the SQuAD metric uses strings
predictions = [{'prediction_text': '1976', 'id': 5}]
references = [{'answers': {'answer_start': [97], 'text': ['1976']}, 'id': 5}]

squad_metric = datasets.load_metric("squad")
squad_metric.add_batch(predictions=predictions, references=references)
results = squad_metric.compute()
# {'exact_match': 100.0, 'f1': 100.0}
```

cc @sgugger @philschmid


Fixed: duplicate parameter and missing parameter in docstring

changing duplicate parameter `data_files` in `DatasetBuilder.__init__` to the missing parameter `data_dir`


Rouge and Meteor for multiple references

bug

Something isn't working

Hi @avinashsai ,

currently, multiple references are not supported. However, we could add a `multiref` config to fix that. When working with multiple references, we can accumulate them by either taking an average or the best score. Would you like to work on that?

Hi,

Currently rogue and meteor supports only single references. Can we use these metrics to calculate for multiple references?


Illegal instruction (core dumped) at datasets import

Something isn't working

It seems to be an issue with how conda-forge is building the binaries. It works on some machines, but not a machine with AMD Opteron 8384 processors.

## Describe the bug
I install datasets using conda and when I import datasets I get: "Illegal instruction (core dumped)"

## Steps to reproduce the bug

```
conda create --prefix path/to/env
conda activate path/to/env
conda install -c huggingface -c conda-forge datasets
# exits with output "Illegal instruction (core dumped)"
python -m datasets
```

## Environment info

When I run "datasets-cli env", I also get "Illegal instruction (core dumped)"

If I run the following commands:

```
conda create --prefix path/to/another/new/env
conda activate path/to/another/new/env
conda install -c huggingface transformers
transformers-cli env
```

Then I get:

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-67-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Let me know what additional information you need in order to debug this issue. Thanks in advance!


Sacrebleu unexpected behaviour/requirement for data format

Something isn't working

Hi @BramVanroy!

Good question. This project relies on PyArrow (tables) to store data too big to fit in RAM. In the case of metrics, this means that the number of predictions and references has to match to form a table.

That's why your example throws an error even though it matches the schema:
```python
refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]  # len(refs) = 2

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'] # len(hyps) = 3
```

Instead, it should be:
```python
refs = [
    ['The dog bit the man.', 'The dog had bit the man.'],
    ['It was not unexpected.', 'No one was surprised.'],
    ['The man bit him first.', 'The man had bitten the dog.'],
]  # len(refs) = 3

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'] # len(hyps) = 3
```

However, `sacreblue` works with the format that's described in your example, hence this part:
https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L94-L99

Hope you get an idea!

Thanks, that makes sense. It is a bit unfortunate because it may be confusing to users since the input format is suddenly different than what they may expect from the underlying library/metric. But it is understandable due to how `datasets` works!

## Describe the bug
When comparing with the original `sacrebleu` implementation, the `datasets` implementation does some strange things that I do not quite understand. This issue was triggered when I was trying to implement TER and found the datasets implementation of BLEU [here](https://github.com/huggingface/datasets/pull/3153).

In the below snippet, the original sacrebleu snippet works just fine whereas the datasets implementation throws an error.

## Steps to reproduce the bug
```python
import sacrebleu
import datasets

refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

expected_bleu = 48.530827

ds_bleu = datasets.load_metric("sacrebleu")

bleu_score_sb = sacrebleu.corpus_bleu(hyps, refs).score
print(bleu_score_sb, expected_bleu)
# works: 48.5308...
bleu_score_ds = ds_bleu.compute(predictions=hyps, references=refs)["score"]
print(bleu_score_ds, expected_bleu)
# ValueError: Predictions and/or references don't match the expected format.
```
This seems to be related to how datasets forces the features format here:

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L94-L99

and then manipulates the references during the compute stage here

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L119-L122

I do not quite understand why that is required since sacrebleu handles argument parsing quite well [by itself](https://github.com/mjpost/sacrebleu/blob/2787185dd0f8d224c72ee5a831d163c2ac711a47/sacrebleu/metrics/base.py#L229).
## Actual results
Traceback (most recent call last):
  File "C:\Users\bramv\AppData\Roaming\JetBrains\PyCharm2020.3\scratches\scratch_23.py", line 23, in <module>
    bleu_score_ds = ds_bleu.compute(predictions=hyps, references=refs)["score"]
  File "C:\dev\python\datasets\src\datasets\metric.py", line 392, in compute
    self.add_batch(predictions=predictions, references=references)
  File "C:\dev\python\datasets\src\datasets\metric.py", line 439, in add_batch
    raise ValueError(
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Value(dtype='string', id='sequence'), 'references': Sequence(feature=Value(dtype='string', id='sequence'), length=-1, id='references')},
Input predictions: ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'],
Input references: [['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'], ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.']]

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2
- PyArrow version: 4.0.1


Add TER (as implemented in sacrebleu)

The problem appears to stem from the omission of the lines that you mentioned. If you add them back and try examples from [this](https://huggingface.co/docs/datasets/using_metrics.html) tutorial (sacrebleu metric example) the code you implemented works fine.

I think the purpose of these lines is follows:

1. Sacrebleu metrics confusingly expect a nested list of strings when you have just one reference for each hypothesis (i.e. `[["example1", "example2", "example3]]`), while for cases with more than one reference a _nested list of lists of strings_ (i.e. `[["ref1a", "ref1b"], ["ref2a", "ref2b"], ["ref3a", "ref3b"]]`) is expected instead. So `transformed_references` line outputs the required single reference format for sacrebleu's ter implementation which you can't pass directly to `compute`.
2. I'm assuming that an additional check is also related to that confusing format with one/many references, because it's really difficult to tell what exactly you're doing wrong if you're not aware of that issue.

Implements TER (Translation Edit Rate) as per its implementation in sacrebleu. Sacrebleu for BLEU scores is already implemented in `datasets` so I thought this would be a nice addition.

I started from the sacrebleu implementation, as the two metrics have a lot in common.

Verified with sacrebleu's [testing suite](https://github.com/mjpost/sacrebleu/blob/078c440168c6adc89ba75fe6d63f0d922d42bcfe/test/test_ter.py) that this indeed works as intended.

```python
import datasets

test_cases = [
    (['aaaa bbbb cccc dddd'], ['aaaa bbbb cccc dddd'], 0),  # perfect match
    (['dddd eeee ffff'], ['aaaa bbbb cccc'], 1),  # no overlap
    ([''], ['a'], 1),  # corner case, empty hypothesis
    (['d e f g h a b c'], ['a b c d e f g h'], 1 / 8),  # a single shift fixes MT
    (
        [
            'wählen Sie " Bild neu berechnen , " um beim Ändern der Bildgröße Pixel hinzuzufügen oder zu entfernen , damit das Bild ungefähr dieselbe Größe aufweist wie die andere Größe .',
            'wenn Sie alle Aufgaben im aktuellen Dokument aktualisieren möchten , wählen Sie im Menü des Aufgabenbedienfelds die Option " Alle Aufgaben aktualisieren . "',
            'klicken Sie auf der Registerkarte " Optionen " auf die Schaltfläche " Benutzerdefiniert " und geben Sie Werte für " Fehlerkorrektur-Level " und " Y / X-Verhältnis " ein .',
            'Sie können beispielsweise ein Dokument erstellen , das ein Auto über die Bühne enthält .',
            'wählen Sie im Dialogfeld " Neu aus Vorlage " eine Vorlage aus und klicken Sie auf " Neu . "',
        ],
        [
            'wählen Sie " Bild neu berechnen , " um beim Ändern der Bildgröße Pixel hinzuzufügen oder zu entfernen , damit die Darstellung des Bildes in einer anderen Größe beibehalten wird .',
            'wenn Sie alle Aufgaben im aktuellen Dokument aktualisieren möchten , wählen Sie im Menü des Aufgabenbedienfelds die Option " Alle Aufgaben aktualisieren . "',
            'klicken Sie auf der Registerkarte " Optionen " auf die Schaltfläche " Benutzerdefiniert " und geben Sie für " Fehlerkorrektur-Level " und " Y / X-Verhältnis " niedrigere Werte ein .',
            'Sie können beispielsweise ein Dokument erstellen , das ein Auto enthalt , das sich über die Bühne bewegt .',
            'wählen Sie im Dialogfeld " Neu aus Vorlage " eine Vorlage aus und klicken Sie auf " Neu . "',
        ],
        0.136  # realistic example from WMT dev data (2019)
    ),
]

ter = datasets.load_metric(r"path\to\datasets\metrics\ter")

predictions = ["hello there general kenobi", "foo bar foobar"]
references = [["hello there general kenobi", "hello there !"], ["foo bar foobar", "foo bar foobar"]]
print(ter.compute(predictions=predictions, references=references))

for hyp, ref, score in test_cases:
    # Note the reference transformation which is different from scarebleu's input format
    results = ter.compute(predictions=hyp, references=[[r] for r in ref])
    assert 100*score == results["score"], f"expected {100*score}, got {results['score']}"
```


Fix some typos in the documentation


Re-add faiss to windows testing suite

In recent versions, `faiss-cpu` seems to be available for Windows as well. See the [PyPi page](https://pypi.org/project/faiss-cpu/#files) to confirm. We can therefore included it for Windows in the setup file.

At first tests didn't pass due to problems with permissions as caused by `NamedTemporaryFile` on Windows. This built-in library is notoriously poor in playing nice on Windows. The required change isn't pretty, but it works. First set `delete=False` to not automatically try to delete the file on `exit`. Then, manually delete the file with `unlink`. It's weird, I know, but it works.

```python
with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
    # do stuff
os.unlink(tmp_file.name)
```

closes #3150


Faiss _is_ available on Windows

In the setup file, I find the following:

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/setup.py#L171

However, FAISS does install perfectly fine on Windows on my system. You can also confirm this on the [PyPi page](https://pypi.org/project/faiss-cpu/#files), where Windows wheels are available. Maybe this was true for older versions? For current versions, this can be removed I think.

(This isn't really a bug but didn't know how else to tag.)

If you agree I can do a quick PR and remove that line.


Add CMU Hinglish DoG Dataset for MT

Hi @lhoestq, thanks a lot for the help. I have moved the part as suggested.
Although still while running the dummy data script, I face this issue

```
Traceback (most recent call last):
  File "/home/ishan/anaconda3/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/commands/dummy_data.py", line 318, in run
    self._autogenerate_dummy_data(
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/commands/dummy_data.py", line 363, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator)
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/builder.py", line 1103, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/features/features.py", line 981, in encode_example
    return encode_nested_example(self, example)
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/features/features.py", line 775, in encode_nested_example
    return {
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/features/features.py", line 775, in <dictcomp>
    return {
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 99, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 99, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
KeyError: 'status'
```
This KeyError is at times different from 'status' also.
when I run
```
datasets-cli dummy_data datasets/cmu_hinglish_dog --auto_generate --json_field='history'
```
I have tried removing unnecessary feature type definition, but that didn't help. Please let me know if I am missing something, thanks!

The CI fail is unrelated to this PR and fixed on master. Merging !

Address part of #2841

Added the CMU Hinglish DoG Dataset as in GLUECoS. Added it as a seperate dataset as unlike other tasks of GLUE CoS this can't be evaluated for a BERT like model.
Consists of parallel dataset between Hinglish (Hindi-English) and English, can be used for Machine Translation between the two.

The data processing part is inspired from the GLUECoS repo [here](https://github.com/microsoft/GLUECoS/blob/7fdc51653e37a32aee17505c47b7d1da364fa77e/Data/Preprocess_Scripts/preprocess_mt_en_hi.py)
The dummy data part is not working properly, it shows
``` UnboundLocalError: local variable 'generator_splits' referenced before assignment ```
when I run without ``--auto_generate``.

Please let me know how I can fix that.
Thanks


Streaming with num_workers != 0

Something isn't working

I can confirm that I was able to reproduce the bug. This seems odd given that #3423 reports duplicate data retrieval when `num_workers` and `streaming` are used together, which is obviously different from what is reported here.

Any update? A possible solution is to have multiple arrow files as shards, and handle them like what webdatasets does.
![image](https://user-images.githubusercontent.com/11533479/148176637-72746b2c-c122-47aa-bbfe-224b13ee9a71.png)

Pytorch's new dataset RFC is supporting sharding now, which may helps avoid duplicate data under streaming mode. (https://github.com/pytorch/pytorch/blob/master/torch/utils/data/datapipes/iter/grouping.py#L13)

Hi ! Thanks for the insights :) Note that in streaming mode there're usually no arrow files. The data are streamed from TAR, ZIP, text, etc. files directly from the web. Though for sharded datasets we can definitely adopt a similar strategy !

## Describe the bug
When using dataset streaming with pytorch DataLoader, the setting num_workers to anything other than 0 causes the code to freeze forever before yielding the first batch.

The code owner is likely @lhoestq

## Steps to reproduce the bug

For your convenience, we've prepped a colab notebook that reproduces the bug
https://colab.research.google.com/drive/1Mgl0oTZSNIE3UeGl_oX9wPCOIxRg19h1?usp=sharing
```python
!pip install datasets==1.14.0

should_freeze_forever = True
# ^-- set this to True in order to freeze forever, set to False in order to work normally

import torch
from datasets import load_dataset

data = load_dataset("oscar", "unshuffled_deduplicated_bn", split="train", streaming=True)
data = data.map(lambda x: {"text": x["text"], "orig": f"oscar[{x['id']}]"}, batched=True)
data = data.shuffle(100, seed=1337)

data = data.with_format("torch")
loader = torch.utils.data.DataLoader(data, batch_size=2, num_workers=2 if should_freeze_forever else 0)

# v-- the code should freeze forever at this line
for i, row in enumerate(loader):
  print(row)
  if i > 10: break
print("DONE!")
```

## Expected results
The code should not freeze forever with num_workers=2

## Actual results
The code freezes forever with num_workers=2

## Environment info
- `datasets` version: 1.14.0 (also found in previous versions)
- Platform: google colab (also locally)
- Python version: 3.7, (also 3.8)
- PyArrow version: 3.0.0


Fix CLI test to ignore verfications when saving infos

Fix #3146.


CLI test command throws NonMatchingSplitsSizesError when saving infos

Something isn't working

When trying to generate a datset JSON metadata, a `NonMatchingSplitsSizesError` is thrown:
```
$ datasets-cli test datasets/arabic_billion_words --save_infos --all_configs
Testing builder 'Alittihad' (1/10)
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: Unknown size, post-processed: Unknown size, total: 332.13 MiB) to .cache\arabic_billion_words\Alittihad\1.1.0\8175ff1c9714c6d5d15b1141b6042e5edf048276bb81a9c14e35e149a7a62ae4...
Traceback (most recent call last):
  File "path\huggingface\datasets\.venv\Scripts\datasets-cli-script.py", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "path\huggingface\datasets\src\datasets\commands\datasets_cli.py", line 33, in main
    service.run()
  File "path\huggingface\datasets\src\datasets\commands\test.py", line 144, in run
    builder.download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 709, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "path\huggingface\datasets\src\datasets\utils\info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='arabic_billion_words'), 'recorded': SplitInfo(name='train', num_bytes=1601790302, num_examples=349342, dataset_name='arabic_billion_words')}]
```

This is due because a previous run generated a wrong `dataset_info.json`.

This error can be avoided by passing `--ignore_verifications`, but I think this should be assumed when passing `--save_infos`.


[when Image type will exist] provide a way to get the data as binary + filename

enhancement

New feature or request

dataset-viewer

Related to the dataset viewer on huggingface.co

@severo, maybe somehow related to this PR ?
- #3129

@severo I'll keep that in mind.

You can track progress on the Image feature in #3163 (still in the early stage).

Hi ! As discussed with @severo offline it looks like the dataset viewer already supports reading PIL images, so maybe the dataset viewer doesn't need to disable decoding after all

**Is your feature request related to a problem? Please describe.**

When a dataset cell contains a value of type Image (be it from a remote URL, an Array2D/3D, or any other way to represent images), I want to be able to write the image to the disk, with the correct filename, and optionally to know its mimetype, in order to serve it on the web.

Note: this issue would apply exactly the same for the `Audio` type.

**Describe the solution you'd like**

If a "cell" has the type `Image`, provide a way to get the binary content of the file, and the filename, eg as:

```python
  filename: str
  data: bytes
```

**Describe alternatives you've considered**

A way to write the cell to the disk (passing a local directory), and then return the pathname, filename, and mimetype.


Infer the features if missing

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

Some datasets, in particular community datasets, have no info file, thus no features.

**Describe the solution you'd like**

If a dataset has no features, the first loaded data (5-10 rows) could be used to infer the type.

Related: `datasets` would provide a way to load the data, and get the rows AND the features as the result.

**Describe alternatives you've considered**

The HF hub could also provide some UI to help the dataset maintainers to explicit the types of their rows, or automatically infer them as an initial proposal.


Provide a way to check if the features (in info) match with the data of a split

New feature or request

**Is your feature request related to a problem? Please describe.**

I understand that currently the data loaded has not always the type described in the info features

**Describe the solution you'd like**

Provide a way to check if the rows have the type described by info features

**Describe alternatives you've considered**

Always check it, and raise an error when loading the data if their type doesn't match the features.


Provide a way to write a streamed dataset to the disk

New feature or request

Yes, I agree this feature is much needed. We could do something similar to what TF does (https://www.tensorflow.org/api_docs/python/tf/data/Dataset#cache).

Ideally, if the entire streamed dataset is consumed/cached, the generated cache should be reusable for the Arrow dataset.

**Is your feature request related to a problem? Please describe.**

The streaming mode allows to get the 100 first rows of a dataset very quickly. But it does not cache the answer, so a posterior call to get the same 100 rows will send a request to the server again and again.

**Describe the solution you'd like**

Provide a way to write the streamed rows of a dataset on the disk, and to load from it later.

**Describe alternatives you've considered**

Provide a third mode: `lazy`, which would use the local cache for the data that have already been fetched previously, and use streaming to get the rest of the requested data.


Fix caching bugs

This PR fixes some caching bugs (most likely introduced in the latest refactor):
* remove ")" added by accident in the dataset dir name
* correctly pass the namespace kwargs in `CachedDatasetModuleFactory`
* improve the warning message if `HF_DATASETS_OFFLINE is `True`


Add DER metric

New feature or request

Add DER metric for speaker diarization task.

This is used by SUPERB beenchmark, for example.


Fix file/directory deletion on Windows

bug

Something isn't working

Currently, on Windows, some attempts to delete a dataset file/directory will fail with the `PerimissionError`.

Examples:
- download a dataset, then force redownload it in the same session while keeping a reference to the downloaded dataset
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset = load_dataset("sst", split="train", download_mode="force_redownload")
```
- try to clean up the cache files while keeping a reference to those files (via the mapped dataset):
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset_mapped = dset.map(lambda _: {"dummy_col": 1})
dset.cleanup_cache_files()
```
We should fix those.


More fine-grained taxonomy of error types

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

Exceptions like `FileNotFoundError` can be raised by different parts of the code, and it's hard to detect which one did

**Describe the solution you'd like**

Give a specific exception type for every group of similar errors

**Describe alternatives you've considered**

Rely on the error message, using regex


Fix numpy deprecation warning for ragged tensors

This'll be a really helpful fix, thank you!

Numpy shows a deprecation warning when we call `np.array` on a list of ragged tensors without specifying the `dtype`. If their shapes match, the tensors can be collated together, otherwise the resulting array should have `dtype=np.object`.

Fix #3084
cc @Rocketknight1


Fix script of Arabic Billion Words dataset to return all data

The script has a bug and only parses and generates a portion of the entire dataset.

This PR fixes the loading script so that is properly parses the entire dataset.

Current implementation generates the same number of examples as reported in the [original paper](https://arxiv.org/abs/1611.04033) for all configurations except for one:
- For "Youm7" we generate more examples (1172136) than the ones reported by the paper (1025027)

|                | Number of examples | Number of examples according to the source |
|:---------------|-------------------:|-----:|
| Alittihad      |             349342 |349342 |
| Almasryalyoum  |             291723 |291723 |
| Almustaqbal    |             446873 |446873 |
| Alqabas        |             817274 |817274 |
| Echoroukonline |             139732 |139732 |
| Ryiadh         |             858188 | 858188 |
| Sabanews       |              92149 |92149 |
| SaudiYoum      |             888068 |888068 |
| Techreen       |             314597 |314597 |
| Youm7          |            1172136 |1025027 |

Fix #3126.


Make inspect.get_dataset_config_names always return a non-empty list of configs

New feature or request

Hi @severo, I guess this issue requests not only to be able to access the configuration name (by using `inspect.get_dataset_config_names`), but the configuration itself as well (I mean you use the name to get the configuration afterwards, maybe using `builder_cls.builder_configs`), is this right?

Yes, maybe the issue could be reformulated. As a user, I want to avoid having to manage special cases:
- I want to be able to get the names of a dataset's configs, and use them in the rest of the API (get the data, get the split names, etc).
- I don't want to have to manage datasets with named configs (`glue`) differently from datasets without named configs (`acronym_identification`, `Check/region_1`)

**Is your feature request related to a problem? Please describe.**

Currently, some datasets have a configuration, while others don't. It would be simpler for the user to always have configuration names to refer to

**Describe the solution you'd like**

In that sense inspect.get_dataset_config_names should always return at least one configuration name, be it `default` or `Check___region_1` (for community datasets like `Check/region_1`).

https://github.com/huggingface/datasets/blob/c5747a5e1dde2670b7f2ca6e79e2ffd99dff85af/src/datasets/inspect.py#L161


bug

Something isn't working

Hi,

Did you try to run the code multiple times (GitHub URLs can be down sometimes for various reasons)? I can access `https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py`, so this code is working without an error on my side.

Additionally, can you please run the `datasets-cli env` command because it seems to me that you are using the `datasets` version different from `1.12.1`?

Same issue when running `metric = datasets.load_metric("accuracy")`.
Error info is:
```
metric = datasets.load_metric("accuracy")
Traceback (most recent call last):

  File "<ipython-input-2-d25db38b26c5>", line 1, in <module>
    metric = datasets.load_metric("accuracy")

  File "D:\anaconda3\lib\site-packages\datasets\load.py", line 610, in load_metric
    module_path, _ = prepare_module(

  File "D:\anaconda3\lib\site-packages\datasets\load.py", line 330, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)

  File "D:\anaconda3\lib\site-packages\datasets\utils\file_utils.py", line 288, in cached_path
    output_path = get_from_cache(

  File "D:\anaconda3\lib\site-packages\datasets\utils\file_utils.py", line 605, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/accuracy/accuracy.py
```

 My `datasets-cli env` result is as follows:
- `datasets` version: 1.11.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 6.0.0

@yananchen1989 did you find a way to solve this?

It seems to be able to solve this issue by adding the equivalent `accuracy.py` locally.
change `metric = datasets.load_metric("accuracy")` to  `metric = datasets.load_metric(path = "./accuracy.py")`.
Copy `accuracy.py` from browser at [accuracy.py](https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/accuracy/accuracy.py)

datasets version: 1.12.1

`metric = datasets.load_metric('rouge')`

The error:

> ConnectionError                           Traceback (most recent call last)
> <ipython-input-3-dd10a0c5212f> in <module>
> ----> 1 metric = datasets.load_metric('rouge')
>
> /usr/local/lib/python3.6/dist-packages/datasets/load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
>     613         download_config=download_config,
>     614         download_mode=download_mode,
> --> 615         dataset=False,
>     616     )
>     617     metric_cls = import_main_class(module_path, dataset=False)
>
> /usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, **download_kwargs)
>     328                 file_path = hf_github_url(path=path, name=name, dataset=dataset, version=script_version)
>     329                 try:
> --> 330                     local_path = cached_path(file_path, download_config=download_config)
>     331                 except FileNotFoundError:
>     332                     if script_version is not None:
>
> /usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
>     296             use_etag=download_config.use_etag,
>     297             max_retries=download_config.max_retries,
> --> 298             use_auth_token=download_config.use_auth_token,
>     299         )
>     300     elif os.path.exists(url_or_filename):
>
> /usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
>     603             raise FileNotFoundError("Couldn't find file at {}".format(url))
>     604         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
> --> 605         raise ConnectionError("Couldn't reach {}".format(url))
>     606
>     607     # Try a second time
>
> ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py

Is there any remedy to solve the connection issue ?


Support Audio feature in streaming mode

Fix #3132.


Support Audio feature in streaming mode

enhancement

New feature or request

Currently, Audio feature is only supported for non-streaming datasets.

Due to the large size of many speech datasets, we should also support Audio feature in streaming mode.


Add ADE20k

dataset request

Requesting to add a new dataset

vision

Vision datasets

I think we can close this issue since PR [#3607](https://github.com/huggingface/datasets/pull/3607) solves this.

## Adding a Dataset
- **Name:** ADE20k (actually it's called the MIT Scene Parsing Benchmark, it's actually a subset of ADE20k but a lot of authors still call it ADE20k)
- **Description:** A semantic segmentation dataset, consisting of 150 classes.
- **Paper:** http://people.csail.mit.edu/bzhou/publication/scene-parse-camera-ready.pdf
- **Data:** http://sceneparsing.csail.mit.edu/
- **Motivation:** I am currently adding Transformer-based semantic segmentation models that achieve SOTA on this dataset. It would be great to directly access this dataset using HuggingFace Datasets, in order to make example scripts in HuggingFace Transformers.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Create SECURITY.md

Hi @zidingz, thanks for your contribution.

However I am closing it because it is a duplicate of a previous PR:
 -  #2958

To let the repository confirm feedback@huggingface.co as its security contact.


Support Audio feature for TAR archives in sequential access

Also do you think we can adapt `cast_column` to keep the same value for this new parameter when the user only wants to change the sampling rate ?

Thanks for your comments, @lhoestq, I will address them afterwards.

But, I think it is more important/urgent first address the current blocking non-passing test: https://github.com/huggingface/datasets/runs/4143579241?check_suite_focus=true
- I am thinking of a way of solving it, but if you have any hint, it will be more than welcome! 😅

Basically:
```
{'audio': '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_dataset_with_audio_featur1/data/test_audio_44100.wav'}
```
becomes
```
{'audio': {'bytes': None, 'path': '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_dataset_with_audio_featur1/data/test_audio_44100.wav'}}
```
after a `map`, which is what was stored in the Arrow file. However we expect it remains invariant after this `map`.

@lhoestq, @mariosasko I finally proposed another implementation different from my last one:
- Before: store Audio always a struct<path: string, bytes: binary>, where bytes can be None
- Now, depending on the examples, either store Audio as a struct (as before), or as a string.

Please note that the main motivation for this change was the issue mentioned above: https://github.com/huggingface/datasets/pull/3129#issuecomment-964347056

Until here we had the assumption that a Features object always has an associated, deterministic, pyarrow schema. This is useful to ensure that we are able to concatenate two datasets that have the same features for example.

By breaking this assumption for the Audio type, how can we ensure that we can concatenate two audio datasets if one has Audio as a struct and the other a string ?

Oh I noticed that the Audio feature type has a private attribute `_storage_dtype`, so the assumption still holds, since they are now different feature types depending on the this attribute :)
(i mean different from the python equal operator point of view)

I think this PR is ready, @lhoestq, @mariosasko.

Nit: We should also mention the new storage structure in the `Features` docstring [here](https://github.com/huggingface/datasets/blob/b29fb550c31de337b952035a7584147e0f18c0cf/src/datasets/features/features.py#L966) for users to know what type of value to return in their dataset scripts (we also have a link to that docstring in the `ADD_NEW_DATASET` template).

Add Audio feature support for TAR archived files in sequential access.

Fix #3128.


enhancement

New feature or request

Currently, Audio feature accesses each audio file by their file path.

However, streamed TAR archive files do not allow random access to their archived files.

Therefore, we should enhance the Audio feature to support TAR archived files in sequential access.


datasets-cli: convertion of a tfds dataset to a huggingface one.

Hi,

the MNIST dataset is already available on the Hub. You can use it as follows:
```python
import datasets
dataset_dict = datasets.load_dataset("mnist")
```

As for the conversion of TFDS datasets to HF datasets, we will be working on it in the coming months, so stay tuned.

### Discussed in https://github.com/huggingface/datasets/discussions/3079

Originally posted by **vitalyshalumov** October 14, 2021
I'm trying to convert a tfds dataset to a huggingface one.

I've tried:

1.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/mnist/3.0.1/

2.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/

and other permutations.
The script appears to be running and finishing without an error but when looking in the  huggingface/datasets/ folder nothing is created.


"arabic_billion_words" dataset does not create the full dataset

bug

Something isn't working

Thanks for reporting, @vitalyshalumov.

Apparently the script to parse the data has a bug, and does not generate the entire dataset.

I'm fixing it.

## Describe the bug
When running:
raw_dataset = load_dataset('arabic_billion_words','Alittihad')
the correct dataset file is pulled from the url.
But, the generated dataset includes just a small portion of the data included in the file.
This is true for all other portions of the "arabic_billion_words" dataset ('Almasryalyoum',.....)

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
raw_dataset = load_dataset('arabic_billion_words','Alittihad')

#The screen message
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 20.62 MiB, post-processed: Unknown size, total: 352.74 MiB)

## Expected results
over 100K sentences

## Actual results
only 11K sentences

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1


Add SLR83 to OpenSLR

The PR resolves #3119, adding SLR83 (UK and Ireland dialects) to the previously created OpenSLR dataset.


More efficient nested features encoding

@lhoestq @albertvillanova @mariosasko
Can you please check this out?

Nested encoding of features wastes a lot of time on operations which are effectively doing nothing when lists are used.
For example, if in the input we have a list of integers, `encoded_nested_example` will iterate over it and apply `encoded_nested_example` on every element even though it just return the int as is.

A similar issue is handled at an earlier stage when casting pytorch/tensorflow/pandas objects to python lists/numpy arrays:
https://github.com/huggingface/datasets/blob/c98c23c4260edadab00f997d1a5d66b7f2e93ce9/src/datasets/features/features.py#L149-L156
https://github.com/huggingface/datasets/blob/c98c23c4260edadab00f997d1a5d66b7f2e93ce9/src/datasets/features/features.py#L212-L228

In this pull request I suggest to use the same approach in `encoded_nested_example`.
In my setup there was a major speedup with this change: loading the data was at least x4 faster.


Segmentation fault when loading datasets from file

Something isn't working

Hi ! I created an issue on Arrow's JIRA after making a minimum reproducible example

https://issues.apache.org/jira/browse/ARROW-14439

```python
import io

import pyarrow.json as paj

batch = b'{"a": [], "b": 1}\n{"b": 1}'
block_size = 12

paj.read_json(
    io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
)
```

I don't see a way to workaround this properly now without hurting the performance of the JSON loader significantly though

The issue has been fixed in pyarrow 6.0.0, please update pyarrow :)

The issue was due to missing fields in the JSON data of type list. Now it's working fine and missing list fields are replaced with empty lists

## Describe the bug
Custom dataset loading sometimes segfaults and kills the process if chunks contain a variety of features/

## Steps to reproduce the bug

Download an example file:
```
wget https://gist.githubusercontent.com/TevenLeScao/11e2184394b3fa47d693de2550942c6b/raw/4232704d08fbfcaf93e5b51def9e5051507651ad/tiny_kelm.jsonl
```
Then in Python:
```
import datasets
tiny_kelm = datasets.load_dataset("json", data_files="tiny_kelm.jsonl", chunksize=100000)
```

## Expected results
a `tiny_kelm` functional dataset

## Actual results
☠ `Segmentation fault (core dumped)` ☠

## Environment info
- `datasets` version: 1.14.0
- Platform: Linux-5.11.0-38-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0


OSError with a custom dataset loading script

Something isn't working

Hi,

there is a difference in how the `data_dir` is zipped between the `classla/janes_tag` and the `classla/reldi_hr` dataset. After unzipping, for the former, the data files (`*.conllup`) are in the root directory (root -> data files), and for the latter, they are inside the `data` directory (root -> `data` -> data files).

This can be fixed by removing the `os.path.join` call in https://huggingface.co/datasets/classla/janes_tag/blob/main/janes_tag.py#L86

Let me know if this works for you.

Hi Mario,

I had already tried that before, but it didn't work. I have now recreated the `classla/janes_tag`  zip file so that it also contains the `data` directory, but I am still getting the same error.

Hi,

I just tried to download the `classla/janes_tag` dataset, and this time the zip file is extracted correctly. However, the script is now throwing the IndexError, probably due to a bug in the `_generate_examples`.

Let me know if you are still getting the same error.

Hi,

could you try to download the dataset with a different `cache_dir` like so:
```python
import datasets
dataset = datasets.load_dataset('classla/janes_tag', split='validation', cache_dir="path/to/different/cache/dir")
```
If this works, then most likely the cached extracted data is causing issues. This data is stored at `~/.cache/huggingface/datasets/downloads/extracted` and needs to be deleted, and then it should work (you can easily locate the directory with the path given in the `OSError` message). Additionally, I'd suggest you to update `datasets` to the newest version with:
```
pip install -U datasets
```

Thank you, deleting the `~/.cache/huggingface/datasets/downloads/extracted` directory helped. However, I am still having problems.

There was indeed a bug in the script that was throwing an `IndexError`, which I have now corrected (added the condition to skip the lines starting with '# text') and it is working locally, but still throws an error when I try to load the dataset from HuggingFace. I literally copied and pasted the `_generate_examples` function and ran it on the `dev_all.conllup` file, which I even re-downloaded from the repository to be certain that the files are exactly the same. I also deleted everything again just in case, but it didn't help. The code works locally, but throws an `IndexError` when loading from `datasets.`

Hi,

Did some investigation.

To fix the dataset script on the Hub, append the following labels to the `names` list of the `upos_tags` field:
```'INTJ NOUN', 'AUX PRON', 'PART ADV', 'PRON ADP', 'INTJ INTJ', 'VERB NOUN', 'NOUN AUX'```.

This step is required to avoid an error due to missing labels in the following step which is:
```python
load_dataset("classla/janes_tag", split="validation", download_mode="force_redownload")
```
This will generate and cache the dataset, so specifying `download_mode` will not be required anymore unless you update the script/data on the Hub.

## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0


Use huggingface_hub.HfApi to list datasets/metrics

Delete `datasets.inspect.HfApi` and use `huggingface_hub.HfApi` instead.

WIP until https://github.com/huggingface/huggingface_hub/pull/429 is merged, then wait for the new release of  `huggingface_hub`, update the `huggingface_hub` version in `setup.py` and merge this PR.

cc: @lhoestq


Correctly update metadata to preserve features when concatenating datasets with axis=1

This PR correctly updates metadata to preserve higher-level feature types (e.g. `ClassLabel`) in `datasets.concatenate_datasets` when `axis=1`. Previously, we would delete the feature metadata in `datasets.concatenate_datasets` if `axis=1` and restore the feature types from the arrow table schema in `Dataset.__init__`. However, this approach only works for simple feature types (e.g. `Value`).

Fixes #3111


Add OpenSLR 83 - Crowdsourced high-quality UK and Ireland English Dialect speech

dataset request

Requesting to add a new dataset

Ugh. The index files for SLR83 are CSV, not TSV. I need to add logic to process these index files.

## Adding a Dataset
- **Name:** *openslr**
- **Description:** *Data set which contains male and female recordings of English from various dialects of the UK and Ireland.*
- **Paper:** *https://www.openslr.org/resources/83/about.html*
- **Data:** *Eleven separate data files can be found via https://www.openslr.org/resources/83/*
- **Motivation:** *Increase english ASR data with UK and Irish dialects*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

The *openslr* dataset already exists, this will add additional subset, *SLR83*.


Fix CI error at each release commit

Fix test_load_dataset_canonical at release commit.

Fix #3117.


CI error at each release commit

bug

Something isn't working

After 1.12.0, there is a recurrent CI error at each release commit: https://app.circleci.com/pipelines/github/huggingface/datasets/8289/workflows/665d954d-e409-4602-8202-e678594d2946/jobs/51110

```
____________________ LoadTest.test_load_dataset_canonical _____________________
[gw0] win32 -- Python 3.6.8 C:\tools\miniconda3\python.exe

self = <tests.test_load.LoadTest testMethod=test_load_dataset_canonical>

    def test_load_dataset_canonical(self):
        scripts_version = os.getenv("HF_SCRIPTS_VERSION", SCRIPTS_VERSION)
        with self.assertRaises(FileNotFoundError) as context:
            datasets.load_dataset("_dummy")
        self.assertIn(
            f"https://raw.githubusercontent.com/huggingface/datasets/{scripts_version}/datasets/_dummy/_dummy.py",
>           str(context.exception),
        )
E       AssertionError: 'https://raw.githubusercontent.com/huggingface/datasets/1.14.0/datasets/_dummy/_dummy.py' not found in "Couldn't find a dataset script at C:\\Users\\circleci\\datasets\\_dummy\\_dummy.py or any data file in the same directory. Couldn't find '_dummy' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/_dummy/_dummy.py"

tests\test_load.py:358: AssertionError
```


Update doc links to point to new docs

documentation

Improvements or additions to documentation

This PR:
* updates the README links and the ADD_NEW_DATASET template to point to the new docs (the new docs don't have a section with the list of all the possible features, so I added that info to the `Features` docstring, which is then referenced in the ADD_NEW_DATASET template)
* fixes some broken links in the `.rst` files (fixed with the `make linkcheck` tool)


Fill in dataset card for NCBI disease dataset


load_from_disk in DatasetsDict/Dataset not working with PyArrowHDFS wrapper implementing fsspec.spec.AbstractFileSystem

bug

Something isn't working

Hi ! Can you try again with pyarrow 6.0.0 ? I think it includes some changes regarding filesystems compatibility with fsspec.

Hi @lhoestq! I ended up using `fsspec.implementations.arrow.HadoopFileSystem` which doesn't have the problem I described with pyarrow 5.0.0.

I'll try again with `PyArrowHDFS` once I update arrow to 6.0.0.

Thanks!

## Describe the bug
Passing a PyArrowHDFS implementation of fsspec.spec.AbstractFileSystem (in the `fs` param required by `load_from_disk` methods in `DatasetDict` (in datasets_dict.py) and `Dataset` (in arrow_dataset.py) results in an error when calling the download method in the `fs` parameter.

## Steps to reproduce the bug

The documentation for the `fs` parameter states:

```
fs (:class:`~filesystems.S3FileSystem` or ``fsspec.spec.AbstractFileSystem``, optional, default ``None``):
                Instance of the remote filesystem used to download the files from.
```

`PyArrowHDFS` from [fsspec](https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/hdfs.html) implements `fsspec.spec.AbstractFileSystem`. However, when using it as shown below, I get an error.

```python
from fsspec.implementations.hdfs import PyArrowHDFS
...
transformed_corpus_path = "/user/my_user/clickbait/transformed_ds/"
fs = PyArrowHDFS(host, port, user, kerb_ticket=kerb_ticket)
dss = DatasetDict.load_from_disk(transformed_corpus_path, fs, True)
```

## Expected results

Previous to load from disk, I have managed to successfully store in HDFS the data and meta-information of a DatasetDict by doing:
```python
transformed_corpus_path = "/user/my_user/clickbait/transformed_ds/"
fs = PyArrowHDFS(host, port, user, kerb_ticket=kerb_ticket)
my_datasets.save_to_disk(transformed_corpus_path, fs=fs)
```

As I have 3 datasets in the DatasetDict named `my_datasets`, the previous Python code creates the following contents in HDFS:

```sh
$ hadoop fs -ls "/user/my_user/clickbait/transformed_ds/"
Found 4 items
-rw-------   3 my_user users         43 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/dataset_dict.json
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/test
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/train
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/validation
```

I would expect to recover on `dss` the Arrow-backed datasets I previously saved in HDFS calling the `save_to_disk` method on the `DatasetDict` object when invoking `DatasetDict.load_from_disk(...)` as described above.

## Actual results

However, when trying to recover the saved datasets, I get this error:

```
...
  File "/home/fperez/dev/neuromancer/neuromancer/corpus.py", line 186, in load_transformed_corpus_from_disk
    dss = DatasetDict.load_from_disk(transformed_corpus_path, fs, True)
  File "/home/fperez/anaconda3/envs/neuromancer/lib/python3.9/site-packages/datasets/dataset_dict.py", line 748, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
  File "/home/fperez/anaconda3/envs/neuromancer/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1048, in load_from_disk
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
  File "pyarrow/_hdfsio.pyx", line 438, in pyarrow._hdfsio.HadoopFileSystem.download
TypeError: download() got an unexpected keyword argument 'recursive'
```

Examining the [signature of the download method in pyarrow 5.0.0](https://github.com/apache/arrow/blob/54d2bd89c99df72fa091b025452f85dd5d88e3cf/python/pyarrow/_hdfsio.pyx#L438) we can see that there's no download parameter:

```python
    def download(self, path, stream, buffer_size=None):
        with self.open(path, 'rb') as f:
            f.download(stream, buffer_size=buffer_size)
```

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0


Loading Data from HDF files

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
More often than not I come along big HDF datasets, and currently there is no straight forward way to feed them to a dataset.

**Describe the solution you'd like**
I would love to see a `from_h5` method that gets an interface implemented by the user on how items are extracted from dataset (in case of multiple datasets containing elements like arrays and metadata and etc).

**Describe alternatives you've considered**
Currently I manually load hdf files using `h5py` and implement PyTorch dataset interface. For small h5 files I load them into a pandas dataframe and use `from_pandas` function in the `datasets` package to load them, but for big datasets this is not feasible.

**Additional context**
HDF files are widespread throughout different domains and are one of the go to's for many researchers/scientists/engineers who work with numerical data. Given `datasets`' usecases have outgrown NLP use cases, it will make a lot of sense focusing on things like supporting HDF files.


bug

Something isn't working

I am very unsure on why you tagged me here. I am not a maintainer of the Datasets library and have no idea how to help you.

Ok got it, tensor full of NaNs, cf.

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
315 # This check fails with FloatArrays with nans, which is not what we want, so account for that:

Actually this is is a live bug, documented yet still live so reopening

## Describe the bug
Despite having batches way under 2Gb when running `datasets.map()`, after processing correctly the data of the first batch without fuss and irrespective of writer_batch_size (say 2,4,8,16,32,64 and 128 in my case), it returns the following error :

> OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

Note that I always run `batch_size=writer_batch_size` :

## Steps to reproduce the bug
```python
datasets.map(lambda example : {"column_name" : function(arguments)}, batched=False, remove_columns = datasets.column_names, batch_size=batch_size, writer_batch_size=batch_size, disable_nullable=True, num_proc=None, desc="blablabla")
```
## Introspecting CUDA memory during bug

Placed within `function(arguments)` the following statement to introspect memory usage, merely a little over 1/4 of 2Gb
`print(torch.cuda.memory_summary(device=device, abbreviated=False))`

> |===========================================================================|
 |                  PyTorch CUDA memory summary, device ID 0                 |
 |---------------------------------------------------------------------------|
 |            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
 |===========================================================================|
 |        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
 |---------------------------------------------------------------------------|
 | Allocated memory      |  541418 KB |  545725 KB |  555695 KB |   14276 KB |
 |       from large pool |  540672 KB |  544431 KB |  544431 KB |    3759 KB |
 |       from small pool |     746 KB |    1714 KB |   11264 KB |   10517 KB |
 |---------------------------------------------------------------------------|
 | Active memory         |  541418 KB |  545725 KB |  555695 KB |   14276 KB |
 |       from large pool |  540672 KB |  544431 KB |  544431 KB |    3759 KB |
 |       from small pool |     746 KB |    1714 KB |   11264 KB |   10517 KB |
 |---------------------------------------------------------------------------|
 | GPU reserved memory   |  598016 KB |  598016 KB |  598016 KB |       0 B  |
 |       from large pool |  595968 KB |  595968 KB |  595968 KB |       0 B  |
 |       from small pool |    2048 KB |    2048 KB |    2048 KB |       0 B  |
 |---------------------------------------------------------------------------|
 | Non-releasable memory |   36117 KB |   52292 KB |  274275 KB |  238158 KB |
|       from large pool |   34816 KB |   51537 KB |  261713 KB |  226897 KB |
 |       from small pool |    1301 KB |    2045 KB |   12562 KB |   11261 KB |
 |---------------------------------------------------------------------------|
 | Allocations           |     198    |     224    |     478    |     280    |
 |       from large pool |      74    |      75    |      75    |       1    |
 |       from small pool |     124    |     150    |     403    |     279    |
 |---------------------------------------------------------------------------|
 | Active allocs         |     198    |     224    |     478    |     280    |
 |       from large pool |      74    |      75    |      75    |       1    |
 |       from small pool |     124    |     150    |     403    |     279    |
 |---------------------------------------------------------------------------|
 | GPU reserved segments |      21    |      21    |      21    |       0    |
 |       from large pool |      20    |      20    |      20    |       0    |
 |       from small pool |       1    |       1    |       1    |       0    |
 |---------------------------------------------------------------------------|
 | Non-releasable allocs |      18    |      23    |     166    |     148    |
 |       from large pool |      17    |      18    |      19    |       2    |
 |       from small pool |       1    |       6    |     147    |     146    |
 |===========================================================================|

## Expected results
Efficiently process the datasets and write it down to disk.

## Actual results
--------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2390                             else:
-> 2391                                 writer.write(example)
   2392                 else:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write(self, example, key, writer_batch_size)
    367
--> 368             self.write_examples_on_file()
    369

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
    316                 if not isinstance(pa_array[0], pa.lib.FloatScalar):
--> 317                     raise OverflowError(
    318                         "There was an overflow in the {}. Try to reduce writer_batch_size to have batches smaller than 2GB".format(

OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

During handling of the above exception, another exception occurred:

OverflowError                             Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_16268/2456940807.py in <module>
      3 #tracker = OfflineEmissionsTracker(country_iso_code="FRA", project_name='xxx'+time_stamp,output_dir='./codecarbon')
      4 #tracker.start()
----> 5 process_datasets(source_datasets_paths, dataset_dir, LM_tokenizer, LMhead_model, datasets_selection=['wikipedia'], from_scratch=True,
      6                  clean_sentences=False, negative_sampling=False, translate=False, tokenize=False, generate_embeddings=True, concatenate_embeddings=False,
      7                  max_sample=10000, padding='do_not_pad', truncation=True, cpu_batch_size=1000, gpu_batch_size=2, cpu_writer_batch_size=1000, gpu_writer_batch_size=2, disable_nullable=True, num_proc=None) #

~\xxx\xxx.py in process_datasets(source_datasets_paths, dataset_dir, LM_tokenizer, LMhead_model, datasets_selection, from_scratch, clean_sentences, translate, negative_sampling, tokenize, generate_embeddings, concatenate_embeddings, max_sample, padding, truncation, cpu_batch_size, gpu_batch_size, cpu_writer_batch_size, gpu_writer_batch_size, disable_nullable, num_proc)
    481             for column in tqdm(dataset.column_names, desc=f'Processing column', leave=False):
    482                 if "xxx_" in column:
--> 483                     dataset = dataset.map(lambda example :
    484                                                 {"embeddings_"+str(column).replace("translated_",""):function(input_ids=example[column],
    485                                                                                token_type_ids=example[column.replace("input_ids","token_type_ids")],

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2034
   2035         if num_proc is None or num_proc == 1:
-> 2036             return self._map_single(
   2037                 function=function,
   2038                 with_indices=with_indices,

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
    501             self: "Dataset" = kwargs.pop("self")
    502         # apply actual function
--> 503         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    504         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    505         for dataset in datasets:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
    468         }
    469         # apply actual function
--> 470         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    471         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    472         # re-apply format to the output

~\anaconda3\envs\xxx\lib\site-packages\datasets\fingerprint.py in wrapper(*args, **kwargs)
    404             # Call actual function
    405
--> 406             out = func(self, *args, **kwargs)
    407
    408             # Update fingerprint of in-place transforms + update in-place history of transforms

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2425                 if update_data:
   2426                     if writer is not None:
-> 2427                         writer.finalize()
   2428                     if tmp_file is not None:
   2429                         tmp_file.close()

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in finalize(self, close_stream)
    440             # Re-intializing to empty list for next batch
    441             self.hkey_record = []
--> 442         self.write_examples_on_file()
    443         if self.pa_writer is None:
    444             if self._schema is not None:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
    315                 # This check fails with FloatArrays with nans, which is not what we want, so account for that:
    316                 if not isinstance(pa_array[0], pa.lib.FloatScalar):
--> 317                     raise OverflowError(
    318                         "There was an overflow in the {}. Try to reduce writer_batch_size to have batches smaller than 2GB".format(
    319                             type(pa_array)

OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0

##Next steps
Testing on Linux.
@albertvillanova


concatenate_datasets removes ClassLabel typing.

Something isn't working

Something like this would fix it I think: https://github.com/huggingface/datasets/compare/master...Dref360:HF-3111/concatenate_types?expand=1

## Describe the bug

When concatenating two datasets, we lose typing of ClassLabel columns.

I can work on this if this is a legitimate bug,

## Steps to reproduce the bug
```python
import datasets
from datasets import Dataset, ClassLabel, Value, concatenate_datasets

DS_LEN = 100
my_dataset = Dataset.from_dict(
    {
        "sentence": [f"{chr(i % 10)}" for i in range(DS_LEN)],
        "label": [i % 2 for i in range(DS_LEN)]
    }
)
my_predictions = Dataset.from_dict(
    {
        "pred": [(i + 1) % 2 for i in range(DS_LEN)]
    }
)

my_dataset = my_dataset.cast(datasets.Features({"sentence": Value("string"), "label": ClassLabel(2, names=["POS", "NEG"])}))
print("Original")
print(my_dataset)
print(my_dataset.features)

concat_ds = concatenate_datasets([my_dataset, my_predictions], axis=1)
print("Concatenated")
print(concat_ds)
print(concat_ds.features)

```

## Expected results
The features of `concat_ds` should contain ClassLabel.

## Actual results

On master, I get:
```
{'sentence': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None), 'pred': Value(dtype='int64', id=None)}
```

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1


Stream TAR-based dataset using iter_archive

I'm creating a new branch `stream-tar-audio` just for the audio datasets since they need https://github.com/huggingface/datasets/pull/3129 to be merged first

The CI fails are only related to missing sections or tags in the dataset cards - which is unrelated to this PR

I converted all the dataset based on TAR archive to use iter_archive instead, so that they can be streamable.

It means that around 80 datasets become streamable :)


Update BibTeX entry

Update BibTeX entry.


Add Google BLEU (aka GLEU) metric

This PR adds the NLTK implementation of Google BLEU metric. This is also a part of an effort to resolve an unfortunate naming collision between GLEU for machine translation and GLEU for grammatical error correction.
I used [this page](https://huggingface.co/docs/datasets/add_metric.html)  for reference. Please, point me to the right direction if I missed anything.


Add paper BibTeX citation

Add paper BibTeX citation to README file.


Fix URLs in blog_authorship_corpus dataset

After contacting the authors of the paper "Effects of Age and Gender on Blogging", they confirmed:
- the old URLs are no longer valid
- there are alternative host URLs

Fix #3091.


download_mode=`force_redownload` does not work on removed datasets

Something isn't working

## Describe the bug

If a cached dataset is removed from the library, I don't see how to delete it programmatically. I thought that using `force_redownload` would try to refresh the cache, then raise an exception, but it reuses the cache instead.

## Steps to reproduce the bug

_requires to already have `wit` in the cache_: see https://github.com/huggingface/datasets/pull/2981

```python
import datasets as ds
dataset = ds.load_dataset("wit", split="train", download_mode='force_redownload')
```

## Expected results

It should raise an exception, since the dataset does not exist anymore.

## Actual results

It uses the cached result

```
Using the latest cached version of the module from /home/slesage/.cache/huggingface/modules/datasets_modules/datasets/wit/107afbffd48e058b19101bddc47fbee25fa68eb6d50a733e262875f1285a5171 (last modified on Wed Sep 29 08:21:10 2021) since it couldn't be found locally at wit, or remotely on the Hugging Face Hub.
```

## Environment info

- `datasets` version: 1.13.4.dev0
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


Missing Zenodo 1.13.3 release

Something isn't working

Zenodo has fixed on their side the 1.13.3 release: https://zenodo.org/record/5589150

After `datasets` 1.13.3 release, this does not appear in Zenodo releases: https://zenodo.org/record/5570305

TODO:
- [x] Contact Zenodo support
- [x] Check it is fixed


Fix project description in PyPI

Fix project description appearing in PyPI, so that it contains the content of the README.md file (like transformers).

Currently, `datasets` project description appearing in PyPI shows the release instructions addressed to core maintainers: https://pypi.org/project/datasets/1.13.3/

Fix #3102.


Unsuitable project description in PyPI

Currently, `datasets` project description appearing in PyPI shows the release instructions addressed to core maintainers: https://pypi.org/project/datasets/1.13.3/


Update SUPERB to use Audio features

Thank you! Sorry I forgot this one @albertvillanova

This is the same dataset refresh as the other Audio ones: https://github.com/huggingface/datasets/pull/3081

cc @patrickvonplaten


Replace FSTimeoutError with parent TimeoutError

PR #3050 introduced a dependency on `fsspec.FSTiemoutError`. Note that this error only exists from `fsspec` version `2021.06.0` (June 2021).

To fix #3097, there are 2 alternatives:
- Either pinning `fsspec` to versions newer or equal to `2021.06.0`
- Or replacing `fsspec.FSTimeoutError` wth its parent `asyncio.TimeoutError`, which exists from Python 3.8.0 (Sep 2018).

This PR implements the second approach.

Fix #3097.


Something isn't working

Hi @JTWang2000, thanks for reporting.

However, I cannot reproduce your reported bug:
```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("sst", "default")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 8544
    })
    validation: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 1101
    })
    test: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 2210
    })
})
```

Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
```
pip install -U huggingface_hub
```

Im facing the same issue. I did run the upgrade command but that doesnt seem to resolve the issue

Hi @aneeshjain, could you please specify which `huggingface_hub` version you are using?

Besides that, please run `datasets-cli env` and copy-and-paste its output below.

The problem seems to be with the latest version of `datasets`. After running `pip install -U datasets huggingface_hub`, I get the following:

```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/opt/conda/lib/python3.6/site-packages/datasets/data_files.py", line 122, in <module>
    allowed_extensions: Optional[list] = None,
AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
````
Note that pip reports the latest `datasets` version as
```bash
 pip show datasets
Name: datasets
Version: 1.14.0
```
However, if I downgrade datasets with `pip install datasets==1.11.0`, things now work
```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
dvers=1.11.0
````

> Hi @JTWang2000, thanks for reporting.
>
> However, I cannot reproduce your reported bug:
>
> ```python
> >>> from datasets import load_dataset
>
> >>> dataset = load_dataset("sst", "default")
> >>> dataset
> DatasetDict({
>     train: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 8544
>     })
>     validation: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 1101
>     })
>     test: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 2210
>     })
> })
> ```
>
> Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
>
> ```
> pip install -U huggingface_hub
> ```

My problem solved after updating huggingface hub command. Thanks a lot and sorry for the late reply.

@tjruwase, please note that versions of `datsets` and `huggingface_hub` must be compatible one with each other:
- In `datasets` version `1.11.0`, the requirement on `huggingface_hub` was: `huggingface_hub<0.1.0`
  https://github.com/huggingface/datasets/blob/2cc00f372a96133e701275eec4d6b26d15257289/setup.py#L90
  - Therefore, your installed `huggingface_hub` version `0.0.8` was compatible
- In `datasets` version `1.12.0`, the requirement on `huggingface_hub` was: `huggingface_hub>=0.0.14,<0.1.0`
  https://github.com/huggingface/datasets/blob/6c766f9115d686182d76b1b937cb27e099c45d68/setup.py#L104
  - Therefore, your installed `huggingface_hub` version `0.0.8` was no longer compatible
- Currently, in `datasets` version `1.15.1`, the requirement on `huggingface_hub` is: `huggingface_hub>=0.1.0,<1.0.0`
  https://github.com/huggingface/datasets/blob/018100679d21cf27136f0eccb1c50e3a9c968ce2/setup.py#L102

@JTWang2000, thanks for your answer. I close this issue then.

## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0