Error in `Json(datasets.ArrowBasedBuilder)` class

Something isn't working

I have additionally identified the source of the error, being that [this condition](https://github.com/huggingface/datasets/blob/fc46bba66ba4f432cc10501c16a677112e13984c/src/datasets/packaged_modules/json/json.py#L124-L126) in the file
`python3.8/site-packages/datasets/packaged_modules/json/json.py` is not being entered correctly:
```python
                                    if (
                                        isinstance(e, pa.ArrowInvalid)
                                        and "straddling" not in str(e)
                                        or block_size > len(batch)
                                    ):
```

From what I can tell, in my case the block_size simply needs to be increased, but the error message does not contain "straddling" so the condition does trigger correctly and we fail to reach [the line to increase block_size](https://github.com/huggingface/datasets/blob/fc46bba66ba4f432cc10501c16a677112e13984c/src/datasets/packaged_modules/json/json.py#L135).

Changing the condition above to simply
```python
                                    if (
                                        block_size > len(batch)
                                    ):
```

Fixes the error for me. I'm happy to create a PR containing this fix if the developers deem the other conditions unnecessary.

Hi ! I think the issue comes from the fact that your JSON file is not a valid JSON Lines file.
Each example should be on one single line.

Can you try fixing the format to have one line per example and try again ?

:open_mouth: you're right, that did it! I just put everything on a single line (my file only has a single example) and that fixed the error. Thank you so much!

## Describe the bug
When a json file contains a `text` field that is larger than the block_size, the JSON dataset builder fails.

## Steps to reproduce the bug
Create a folder that contains the following:
```
.
├── testdata
│   └── mydata.json
└── test.py
```

Please download [this file](https://github.com/huggingface/datasets/files/7491797/mydata.txt) as `mydata.json`. (The error does not occur in JSON files with shorter text, but it is reproducible when the text is long as in the file I provide)
:exclamation: :exclamation: GitHub doesn't allow me to upload JSON so this file is a TXT, and you should rename it to `.json`!

`test.py` simply contains:
```python
from datasets import load_dataset
my_dataset = load_dataset("testdata")
```

To reproduce the error, simply run
```
python test.py
```

## Expected results
The data should load correctly without error.

## Actual results
The dataset builder fails with:
```
Using custom data configuration testdata-d490389b8ab4fd82
Downloading and preparing dataset json/testdata to /home/junshern.chan/.cache/huggingface/datasets/json/testdata-d490389b8ab4fd82/0.0.0/3333a8af0db9764dfcff43a42ff26228f0f2e267f0d8a0a294452d188beadb34...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2264.74it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 447.01it/s]
Failed to read file '/home/junshern.chan/hf-json-bug/testdata/mydata.json' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Missing a name for object member. in row 0
Traceback (most recent call last):
  File "test.py", line 28, in <module>
    my_dataset = load_dataset("testdata")
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 1156, in _prepare_split
    for key, table in utils.tqdm(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/tqdm/std.py", line 1168, in __iter__
    for obj in iterable:
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/packaged_modules/json/json.py", line 146, in _generate_tables
    raise ValueError(
ValueError: Not able to read records in the JSON file at /home/junshern.chan/hf-json-bug/testdata/mydata.json. You should probably indicate the field of the JSON file containing your records. This JSON file contain the following fields: ['text']. Select the correct one and provide it as `field='XXX'` to the dataset loading method.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.0


Fix paper BibTeX citation with proceedings reference.


Update tatoeba to v2021-07-22

How about this? @lhoestq @abhishekkrthakur

Hi ! I think it would be nice if people could still be able to load the old version.
Maybe this can be a parameter ? For example to load the old version they could do
```python
load_dataset("tatoeba", lang1="en", lang2="mr", date="v2020-11-09")
```

If it sounds good to you, we can add this parameter to the TatoebaConfig:
```python
class TatoebaConfig(datasets.BuilderConfig):
    def __init__(self, *args, lang1=None, lang2=None, date="v2021-07-22", **kwargs):
    self.date = date
```
and then pass the date to the URL
```python
_BASE_URL = "https://object.pouta.csc.fi/OPUS-Tatoeba/{}/moses/{}-{}.txt.zip"
```
```python
        def _base_url(lang1, lang2, date):
            return _BASE_URL.format(date, lang1, lang2)
```

What do you think ?

`_DATE = "v" + "-".join(s.zfill(2) for s in _VERSION.split("."))` seems rather tricky but works well. How about this? @lhoestq

The CI is only failing because of the missing sections in the dataset card, and because of an issue with the CER metric that is unrelated to this PR

Tatoeba's latest version is v2021-07-22


User-pickling with dynamic sub-classing

@lhoestq Feel free to have a look. The implementation is slightly different from what you suggested. I have opted to overwrite `save` instead of meddling with `save_global`. `save_global` is called very late down in dill/pickle so it is hard to control for what is happening there. I might be wrong. Pickling is more complex than I thought!

The linked issue (`map` with spaCy) also works now!

```python
import pickle
import spacy
from spacy import Language
from datasets import load_dataset
from datasets.utils.py_utils import dumps, pklregister

@pklregister(Language, allow_subclasses=True)
def hash_spacy_language(pickler, nlp: Language):
    pickler.save(nlp.to_bytes())

def main():
    fin = r"large/file.txt"
    nlp = spacy.load("en_core_web_sm")

    def tokenize(l):
        return {"tok": [t.text for t in nlp(l["text"])]}

    ds =  load_dataset("text", data_files=fin)
    ds = ds["train"].map(tokenize)

    # Sanity check: load NLP from pickle created with our own `dumps`
    config = nlp.config
    lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
    nlp2 = lang_cls.from_config(config)
    nlp2.from_bytes(pickle.loads(dumps(nlp)))

    assert isinstance(nlp2, type(nlp))
    assert dumps(nlp) == dumps(nlp2)

if __name__ == '__main__':
    main()
```

If this all looks good to you, I'll start writing on some documentation and examples.

One more thing. This is a reduction function for SpaCy Language that should work with the new API:
```python
@pklregister(Language, allow_subclasses=True)
def hash_spacy_language(pickler, obj):
    def create_language(config, bytes_data):
        lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
        nlp = lang_cls.from_config(config)
        return nlp.from_bytes(bytes_data)

    args = (obj.config, obj.to_bytes())
    pickler.save_reduce(create_language, args, obj=obj)
```
so IMO we are missing a test with `pickler.save_reduce`.

> One more thing. This is a reduction function for SpaCy Language that should work with the new API:
>
> ```python
> @pklregister(Language, allow_subclasses=True)
> def hash_spacy_language(pickler, obj):
>     def create_language(config, bytes_data):
>         lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
>         nlp = lang_cls.from_config(config)
>         return nlp.from_bytes(bytes_data)
>
>     args = (obj.config, obj.to_bytes())
>     pickler.save_reduce(create_language, args, obj=obj)
> ```
>
> so IMO we are missing a test with `pickler.save_reduce`.

Sure that seems a good idea, but I do not quite understand what `save_reduce` does. Could you give some more info about what reduce functions do and how they differ from regular `save` and `save_global`? I've read about it but the docs nor the built-in `pickle` code seem really helpful.

I'm no pickle expect, but here is my understanding. I believe the pickler uses the reduce function when you do `loads` to reconstructs the original object from the parameters/arguments that were saved with `dumps`.

For example your sanity check could be simplified from
```python
    config = nlp.config
    lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
    nlp2 = lang_cls.from_config(config)
    nlp2 = nlp2.from_bytes(pickle.loads(dumps(nlp)))
```
to
EDIT: <s>pickle.loads(pickle.dumps(nlp))</s>
```python
    nlp2 = loads(dumps(nlp))  # using our custom pickler
```

Though note that while it can be convenient for tests, we actually don't care about the reconstruction of the object since we're only using the pickler for `dumps` to compute hashes.

> I'm no pickle expect, but here is my understanding. I believe the pickler uses the reduce function when you do `loads` to reconstructs the original object from the parameters/arguments that were saved with `dumps`.
>
> For example your sanity check could be simplified from
>
> ```python
>     config = nlp.config
>     lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
>     nlp2 = lang_cls.from_config(config)
>     nlp2 = nlp2.from_bytes(pickle.loads(dumps(nlp)))
> ```
>
> to
>
> ```python
>     nlp2 = pickle.loads(pickle.dumps(nlp))
> ```
>
> Though note that while it can be convenient for tests, we actually don't care about the reconstruction of the object since we're only using the pickler for `dumps` to compute hashes.

Yes, the sanity check can be simplified like that _if_ we use `pickle.dumps` - but that would not test our own `dumps` functionality and would do a naive dump instead of using `to_bytes`. It won't work if we use our own `dumps`, exactly because of the reason that we want custom pickling and being able to call `to_bytes`. To reconstruct the object from the pickled bytes from `to_bytes` we need `from_bytes`. The result of pickle/dill loads will therefore always be a `bytes` object and not a `Language` object.

But `save_reduce` is called when saving, right? Not when loading, AFAICT. I am just not sure what exactly it is saving. It is _potentially_ called [at the end of `save`](https://github.com/python/cpython/blob/24af9a40a8f85af813ea89998aa4e931fcc78cd9/Lib/pickle.py#L603) but only if we haven't returned by then. I just can't figure out what that base case is.

I don't think we expect users to write the reduce function that isn't going to be used anyway. So maybe let's stick with `save` ?

@BramVanroy
As I understand `save_reduce` is very similar to `copyreg.pickle`, so I'd suggest you to check the following links:
* https://docs.python.org/3/library/copyreg.html#copyreg.pickle
* https://docs.python.org/3/library/pickle.html#object.__reduce__

@lhoestq
> I don't think we expect users to write the reduce function that isn't going to be used anyway. So maybe let's stick with save ?

I agree.

`save_reduce` is very similar to `copyreg.pickle` and `object.__reduce__`, which are part of public API (and `save` isn't), so I expect more advanced users to know how to write their own reduction functions. But, as you say, `pklregister` should also work with `save` (even though I think `save` is a bit lower-level, and harder to understand than `save_reduce`).

All our examples in `py_utils` that use `pklregister`  also use `save_reduce` in the last step, so my reduction for SpaCy is meant to be added there, and not to be written by users (because SpaCy is very popular, so the official support by us makes sense :)).

And in the tests, let's ignore the reconstruction part of pickle/dill, because it's not important for us, and focus on the generated dumps. What do you think?

@mariosasko What exactly do you mean with "isn't part of the public API"? It is [a public method](https://github.com/python/cpython/blob/24af9a40a8f85af813ea89998aa4e931fcc78cd9/Lib/pickle.py#L535) in base pickle, just like `dump` is but maybe you mean something else.

@BramVanroy Oh sorry, it's public (not prefixed with `"_"`)  but it's not documented in the docs. `save_reduce` is also not in the docs, but its signature/functionality is similar to `copyreg.pickle` and I see it more often being used in the projects on GH, so it's seems "more public" to me.

Unfortunately I feel that pickle in general is under-documented. 😄

For the documentation, I can add a brief example, maybe under "How-to Guides"? The only thing that isn't immediately obvious to me is how I can add that doc page to the TOC?

Yes great idea ! To add that doc page to the TOC, you just have to add it to the index.rst file in the "How-to guides" TOC section

@mariosasko @lhoestq Feel free to make any edits or suggestions in the text!

Hi @mariosasko. I wish you'd told me sooner, as I spent quite some time writing on this.

I'm also not sure whether it is too advanced to have in the documentation. The spaCy use-case seems potentially frequent. Or do you wish to add that case to the defaults, and whenever new issues come up that seem like frequent/obvious cases, add those internally as well?

Documenting the internal `pklregister` is overkill IMO (and it can be kept in docstrings), but we can document something higher level like `register_hash_func` once it's implemented.

So we keep the nice documentation you've written (thank you!), except we can rename it to "Advanced caching" and show an API that is similar to
```python
>>> @register_hash_func(Language, allow_subclasses=True)
>>> def hash_spacy_language(nlp: Language):
>>>     return (nlp.to_bytes(),)
```
This way we keep the documentation centered around the public API rather than the internals that may evolve/be too complicated to fit only one section.

> Or do you wish to add that case to the defaults, and whenever new issues come up that seem like frequent/obvious cases, add those internally as well?

Let's add it to the defaults since it's a frequent use-case. And also allow users to control the hashing using the API mentioned above if they face other non-trivially-hashable objects

Sure, I can have a go at implementing spaCy as a built-in. Should it be included in the tests? (Therefore adding spaCy to the tests requirements.)

Next, from your example, it seems that the return value of `register_hash_func` will be used in pickler.save automatically (calling pklregister a bit deeper). Any reason why it returns a tuple? I can work on this as well, if needed.

> Sure, I can have a go at implementing spaCy as a built-in. Should it be included in the tests? (Therefore adding spaCy to the tests requirements.)

That would be perfect !

> Next, from your example, it seems that the return value of register_hash_func will be used in pickler.save automatically (calling pklregister a bit deeper).

Yes I think so. For example register_hash_func can call pklregister with the user's function, but wrapped to use pickler.save.

> Any reason why it returns a tuple? I can work on this as well, if needed.

It can either return an arbitrary object or a tuple. I like it a bit better if it's a tuple, so users understand more easily how to make the function take into account more than one item for the hash. It's also consistent with the streamlit caching functions, that also require a tuple. No strong opinion on this though

Let me know if I can help with anything

@lhoestq I do not have the time anymore to work on this. Can someone else pick this up?

Hi ! Sure someone else can continue this PR (either someone from HF, or other contributors can fork the PR).
I think I can work on this next week or the week after, but if anyone wants to work on this earlier feel free to comment here :)

This is a continuation of the now closed PR in https://github.com/huggingface/datasets/pull/3206. The discussion there has shaped a new approach to do this.

In this PR, behavior of `pklregister` and `Pickler` is extended. Earlier, users were already able to register custom pickle functions. That is useful if they have objects that are not easily picklable with default methods. When one registers a custom function to a type, an object of that type will be pickled with the given function by `Pickler` which looks up the type in its `dispatch` table. The downside of this method, and of `pickle` in general, is that it is limited to direct type-matching and does not allow sub-classes. In many, default, cases that is not an issue. But when you are using external libraries where classes (e.g. parsers, models) are sub-classed this is not ideal.

```python
from datasets.fingerprint import Hasher
from datasets.utils.py_utils import pklregister

class BaseParser:
    pass

class EnglishParser(BaseParser):
    pass

@pklregister(BaseParser)
def custom_pkl_func(pickler, obj):
    print(f"Called the custom pickle function for type {type(obj)}!")
    # do something with the obj and ultimately save with the pickler

base = BaseParser()
en = EnglishParser()

# Hasher.hash uses the Pickler behind the scenes
# `custom_pkl_func` called for base
Hasher.hash(base)
# `custom_pkl_func` not called for en :-(
Hasher.hash(en)
```

In the example above we'd want to sub-class `EnglishParser` to be handled in the same way as its super-class `BaseParser`. This PR solves that by allowing for a keyword-argument `allow_subclasses` in `pklregister` (default: `False`).

```python
@pklregister(BaseParser, allow_subclasses=True)
```

When this option is enabled, we not only save the function in `Pickler.dispatch` but also save it in a custom table `Pickler.subclass_dispatch` **which allows us to dynamically add sub-classes of that class to the real dispatch table**. Then, if we want to pickle an  object `obj` with `Pickler.dump()` (which ultimately will call `Pickler.save()`) we _first_  check whether any of the object's super-classes exist in `Pickler.sublcass_dispatch` and get the related custom pickle function. If we find one, we add the type of `obj` alongside the function to `Pickler.dispatch`. All of this happens at the start of the call to `Pickler.save()`. _Only then_ dill.Pickler's `save` will be called, which in turn will call `pickle._Pickler.save` which handles everything. Here, the `Pickler.dispatch` table will be used to look up custom pickler functions - and it now also includes the function for `obj`, which was copied from its super-class, which we added at the very start of our custom `Pickler.save()`.

For edge cases and, especially, for testing, a contextmanager class `TempPickleRegistry` is included that resets the pickle registry on exit to its previous state.

```python
with TempPickleRegistry():
    @pklregister(MyObjClass)
    def pickle_registry_test_false(pickler, obj):
        pickler.save(obj.fancy_method())

    some_obj = MyObjClass()
    dumps(some_obj)
    # `MyObjClass` is in Pickler.dispatch

# ... `MyObjClass` is _not_ in Pickler.dispatch anymore
```

closes https://github.com/huggingface/datasets/issues/3178

To Do
====
- [x] Write tests
- [ ] Write documentation/examples?


Update BibTeX entry

Update BibTeX entry.


Add docs for audio processing

documentation

Improvements or additions to documentation

Nice ! love it this way. I guess you can set this PR to "ready for review" ?

This PR adds documentation for the `Audio` feature. It describes:

- The difference between loading `path` and `audio`, as well as use-cases/best practices for each of them.
- Resampling audio files with `cast_column`, and then calling `ds[0]["audio"]` to automatically decode and resample to the desired sampling rate.
- Resampling with `map`.

Preview [here](https://52969-250213286-gh.circle-artifacts.com/0/docs/_build/html/audio_process.html), let me know if I'm missing anything!


Resolve data_files by split name

Really cool!
When splitting by folder, what do we use for validation set ("valid", "validation" or both)?

> When splitting by folder, what do we use for validation set ("valid", "validation" or both)?

Both are fine :) As soon as it has "valid" in it

Merging for now, if you have comments about the documentation we can address them in subsequent PRs :)

Thanks for the comments @stevhliu :) I just opened https://github.com/huggingface/datasets/pull/3233 to take them into account

As discussed in https://github.com/huggingface/datasets/issues/3027 we should automatically infer what file is supposed to go to what split automatically, based on filenames.

I added the support for different kinds of patterns, for both dataset repositories and local directories:

```
    Input structure:

        my_dataset_repository/
        ├── README.md
        └── dataset.csv

    Output patterns:

        {"train": ["*"]}
```
```
    Input structure:

        my_dataset_repository/
        ├── README.md
        ├── train.csv
        └── test.csv

        my_dataset_repository/
        ├── README.md
        └── data/
            ├── train.csv
            └── test.csv

        my_dataset_repository/
        ├── README.md
        ├── train_0.csv
        ├── train_1.csv
        ├── train_2.csv
        ├── train_3.csv
        ├── test_0.csv
        └── test_1.csv

    Output patterns:

        {"train": ["*train*"], "test": ["*test*"]}
```
```
    Input structure:

        my_dataset_repository/
        ├── README.md
        └── data/
            ├── train/
            │   ├── shard_0.csv
            │   ├── shard_1.csv
            │   ├── shard_2.csv
            │   └── shard_3.csv
            └── test/
                ├── shard_0.csv
                └── shard_1.csv

    Output patterns:

        {"train": ["*train*/*", "*train*/**/*"], "test": ["*test*/*", "*test*/**/*"]}
```

and also this pattern that allows to have custom split names, and that is the structure used by #3098 for `push_to_hub` (cc @LysandreJik ):

```
    Input structure:

        my_dataset_repository/
        ├── README.md
        └── data/
            ├── train-00000-of-00003.csv
            ├── train-00001-of-00003.csv
            ├── train-00002-of-00003.csv
            ├── test-00000-of-00001.csv
            ├── random-00000-of-00003.csv
            ├── random-00001-of-00003.csv
            └── random-00002-of-00003.csv

    Output patterns:

        {
            "train": ["data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"],
            "test": ["data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"],
            "random": ["data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"],
        }
```

You can check the documentation about structuring your repository [here](https://52640-250213286-gh.circle-artifacts.com/0/docs/_build/html/repository_structure.html). cc @stevhliu

Fix https://github.com/huggingface/datasets/issues/3027
Fix https://github.com/huggingface/datasets/issues/3212

In the future we can also add support for dataset configurations.


Add documentation about dataset viewer feature

enhancement

New feature or request

Add to the docs more details about the dataset viewer feature in the Hub.

CC: @julien-c


Eventual Invalid Token Error at setup of private datasets

bug

Something isn't working

## Describe the bug
From time to time, there appear Invalid Token errors with private datasets:

- https://app.circleci.com/pipelines/github/huggingface/datasets/8520/workflows/d44629f2-4749-40f8-a657-50931d0b3434/jobs/52534
  ```
  ____________ ERROR at setup of test_load_streaming_private_dataset _____________
  ValueError: Invalid token passed!

  ____ ERROR at setup of test_load_streaming_private_dataset_with_zipped_data ____
  ValueError: Invalid token passed!

  =========================== short test summary info ============================
  ERROR tests/test_load.py::test_load_streaming_private_dataset - ValueError: I...
  ERROR tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data
  ```

- https://app.circleci.com/pipelines/github/huggingface/datasets/8557/workflows/a8383181-ba6d-4487-9d0a-f750b6dcb936/jobs/52763
  ```
  ____ ERROR at setup of test_load_streaming_private_dataset_with_zipped_data ____
  [gw1] linux -- Python 3.6.15 /home/circleci/.pyenv/versions/3.6.15/bin/python3.6

  hf_api = <huggingface_hub.hf_api.HfApi object at 0x7f4899bab908>
  hf_token = 'vgNbyuaLNEBuGbgCEtSBCOcPjZnngJufHkTaZvHwkXKGkHpjBPwmLQuJVXRxBuaRzNlGjlMpYRPbthfHPFWXaaEDTLiqTTecYENxukRYVAAdpeApIUPxcgsowadkTkPj'
  zip_csv_path = PosixPath('/tmp/pytest-of-circleci/pytest-0/popen-gw1/data16/dataset.csv.zip')

      @pytest.fixture(scope="session")
      def hf_private_dataset_repo_zipped_txt_data_(hf_api: HfApi, hf_token, zip_csv_path):
          repo_name = "repo_zipped_txt_data-{}".format(int(time.time() * 10e3))
          hf_api.create_repo(token=hf_token, name=repo_name, repo_type="dataset", private=True)
          repo_id = f"{USER}/{repo_name}"
          hf_api.upload_file(
              token=hf_token,
              path_or_fileobj=str(zip_csv_path),
              path_in_repo="data.zip",
              repo_id=repo_id,
  >           repo_type="dataset",
          )

  tests/hub_fixtures.py:68:

  ...

  ValueError: Invalid token passed!
  =========================== short test summary info ============================
  ERROR tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data
  ```


Fix code quality in riddle_sense dataset

Fix trailing whitespace.

Fix #3217.


Fix code quality bug in riddle_sense dataset

Something isn't working

To give more context: https://github.com/psf/black/issues/318. `black` doesn't treat this as a bug, but `flake8` does.

## Describe the bug
```
datasets/riddle_sense/riddle_sense.py:36:21: W291 trailing whitespace
```


Pin version exclusion for tensorflow incompatible with keras

Once `tensorflow` version 2.6.2 is released:
- https://github.com/tensorflow/tensorflow/commit/c1867f3bfdd1042f694df7a9870be51ba80543cb
- https://pypi.org/project/tensorflow/2.6.2/

with the patch:
- tensorflow/tensorflow#52927

we can remove the temporary fix we introduced in:
- #3208

Fix #3209.


Small updates to to_tf_dataset documentation

@stevhliu Accepted both suggestions, thanks for the review!

I added a little more description about `to_tf_dataset` compared to just setting the format


Add ACAV100M Dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** *ACAV100M*
- **Description:** *contains 100 million videos with high audio-visual correspondence, ideal for self-supervised video representation learning.*
- **Paper:** *https://arxiv.org/abs/2101.10803*
- **Data:** *https://github.com/sangho-vision/acav100m*
- **Motivation:** *The largest dataset (to date) for audio-visual learning.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Fix tuple_ie download url

Fix #3204


Sort files before loading

enhancement

New feature or request

This will be fixed by https://github.com/huggingface/datasets/pull/3221

When loading a dataset that consists of several files (e.g. `my_data/data_001.json`, `my_data/data_002.json` etc.) they are not loaded in order when using `load_dataset("my_data")`.

This could lead to counter-intuitive results if, for example, the data files are sorted by date or similar since they would appear in different order in the `Dataset`.

The straightforward solution is to sort the list of files alphabetically before loading them.

cc @lhoestq


Fix disable_nullable default value to False

Currently the `disable_nullable` parameter is not consistent across all dataset transforms. For example it is `False` in `map` but `True` in `flatten_indices`.

This creates unexpected behaviors like this
```python
from datasets import Dataset, concatenate_datasets

d1 = Dataset.from_dict({"a": [0, 1, 2, 3]})
d2 = d1.filter(lambda x: x["a"] < 2).flatten_indices()
d1.data.schema == d2.data.schema  # False
```
This can cause issues when concatenating datasets for example.

For consistency I set `disable_nullable` to `False` in `flatten_indices` and I fixed some docstrings

cc @SBrandeis


dataset bug

A bug in a dataset script provided in the library

Hi ! Do you have some kind of proxy in your browser that gives you access to internet ?

Maybe you're having this error because you don't have access to this URL from python ?

Hi，do you fixed this error?
I still have this issue when use "use_auth_token=True"

You don't need authentication to access those github hosted files
Please check that you can access this URL from your browser and also from your terminal

when I use python examples/pytorch/translation/run_translation.py     --model_name_or_path examples/pytorch/translation/opus-mt-en-ro     --do_train     --do_eval     --source_lang en     --target_lang ro     --dataset_name wmt16     --dataset_config_name ro-en     --output_dir /tmp/tst-translation     --per_device_train_batch_size=4     --per_device_eval_batch_size=4     --overwrite_output_dir     --predict_with_generate to finetune translation model on huggingface, I get the issue"ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.15.1/datasets/wmt16/wmt16.py".But I can open the https://raw.githubusercontent.com/huggingface/datasets/1.15.1/datasets/wmt16/wmt16.py by using website. What should I do to solve the issue?


Unpin keras once TF fixes its release

Related to:
- #3208


Pin keras version until TF fixes its release

Fix #3207.


CI error: Another metric with the same name already exists in Keras 2.7.0

bug

Something isn't working

## Describe the bug
Release of TensorFlow 2.7.0 contains an incompatibility with Keras. See:
- keras-team/keras#15579

This breaks our CI test suite: https://app.circleci.com/pipelines/github/huggingface/datasets/8493/workflows/055c7ae2-43bc-49b4-9f11-8fc71f35a25c/jobs/52363


[WIP] Allow user-defined hash functions via a registry

Hi @BramVanroy, thanks for your PR.

There was a bug in TensorFlow/Keras. We have made a temporary fix in master branch. Please, merge master into your PR branch, so that the CI tests pass.

```
git checkout registry
git fetch upstream master
git merge upstream/master
```

@albertvillanova Done. Although new tests will need to be added. I am looking for some feedback on my initial proposal in this PR. Reviews and ideas welcome!

Hi ! Thanks for diving into this :)

With this approach you get the right hash when doing `Hasher.hash(nlp)` but if you try to hash an object that has `nlp` as one of its attributes for example you will get different hashes every time.

This is because `Hasher.hash` is not recursive itself. Indeed what happens when you try to hash an object is that:
1. it is dumped with our custom `dill` pickler (which is recursive)
2. the bytes of the dump are hashed

To fix this we must integrate the custom hashing as a custom pickler dumping instead.

Note that we're only using the `pickler.dumps` method and not `pickler.loads` since we only use it to get hashes, so it doesn't matter if `loads` doesn't reconstruct the object exactly. What's important it only to capture all the necessary information that defines how the object transforms the data (here `nlp.to_bytes()` determines how the spacy pipeline transforms the text).

Our pickler already has a registry and you can register new dump functions with:
```python
import dill
import spacy
from datasets.utils.py_utils import pklregister

@pklregister(spacy.Language)
def _save_spacy_language(pickler, nlp):
    pickler.save_reduce(...)  # I think we can use nlp.to_bytes() here
    dill._dill.log.info(...)
```

You can find some examples of custom dump functions in `py_utils.py`

Ah, darn it. Completely missed that register. Time wasted, unfortunately.

To better understand what you mean, I figured I'd try the basis of your snippet and I've noticed quite an annoying side-effect of how the pickle dispatch table seems to work. It explicitly uses an object's [`type()`](https://github.com/python/cpython/blob/87032cfa3dc975d7442fd57dea2c6a56d31c911a/Lib/pickle.py#L557-L558), which makes sense for pickling some (primitive) types it is not ideal for more complex ones, I think. `Hasher.hash` has the same issue as far as I can tell.

https://github.com/huggingface/datasets/blob/d21ce54f2c2782f854f975eb1dc2be6f923b4314/src/datasets/fingerprint.py#L187-L191

This is very restrictive, and won't work for subclasses. In the case of spaCy, for instance, we register `Language`, but `nlp` is an instance of `English`, which is a _subclass_ of `Language`. These are different types, and so they will not match in the dispatch table. Maybe this is more general approach to cover such cases? Something like this is a start but too broad, but ideally a hierarchy is constructed and traversed of all classes in the table and the lowest class is selected to ensure that the most specific class function is dispatched.

```python
    def hash(cls, value: Any) -> str:
        # Try to match the exact type
        if type(value) in cls.dispatch:
            return cls.dispatch[type(value)](cls, value)

        # Try to match instance (superclass)
        for type_cls, func in cls.dispatch.items():
            if isinstance(value, type_cls):
                return cls.dispatch[type_cls](cls, value)

        return cls.hash_default(value)
```

This does not solve the problem for pickling, though. That is quite unfortunate IMO because that implies that users always have to specify the most specific class, which is not always obvious. (For instance, `spacy.load`'s signature returns `Language`, but as said before a subclass might be returned.)

Second, I am trying to understand `save_reduce` but I can find very little documentation about it, only the source code which is quite cryptic. Can you explain it a bit? The required arguments are not very clear to me and there is no docstring.

```python
    def save_reduce(self, func, args, state=None, listitems=None, dictitems=None, obj=None):
```

Here is an example illustrating the problem with sub-classes.

```python
import spacy

from spacy import Language
from spacy.lang.en import English

from datasets.utils.py_utils import Pickler, pklregister

# Only useful in the registry (matching with `nlp`)
# if you swap it out for very specific `English`
@pklregister(English)
def hash_spacy_language(pickler, nlp):
    pass

def main():
    print(Pickler.dispatch)
    nlp = spacy.load("en_core_web_sm")
    print(f"NLP type {type(nlp)} in dispatch table? ", type(nlp) in Pickler.dispatch)

if __name__ == '__main__':
    main()
```

Indeed that's not ideal.
Maybe we could integrate all the subclasses directly in `datasets`. That's simple to do but the catch is that if users have new subclasses of `Language` it won't work.

Otherwise we can see how to make the API simpler for users by allowing subclasses
```python
# if you swap it out for very specific `English`
@pklregister(Language, allow_subclasses=True)
def hash_spacy_language(pickler, nlp):
    pass
```

Here is an idea how to make this work, let me know what you think:

When `Pickler.dumps` is called, it uses `Pickler.save_global` which is a method that is going to be called recursively on all the objects. We can customize this part, and make it work as we want when it encounters a subclass of `Language`.

For example when it encounters a subclass of `Language`, we can dynamically register the hashing function for the subclass (`English` for example) in `Pickler.save_global`, right before calling the actual `dill.Pickler.save_global(self, obj, name=name)`:
```python
pklregister(type(obj))(hash_function_registered_for_parent_class)
dill.Pickler.save_global(self, obj, name=name)
```

In practice that means we can have an additional dispatch dictionary (similar to `Pickler.dispatch`) to store the hashing functions when `allow_subclasses=True`, and use this dictionary in `Pickler.save_global` to check if we need to use a hashing function registered with `allow_subclasses=True` and get `hash_function_registered_for_parent_class`.

If I understood you correctly, I do not think that that is enough because you are only doing this for a type and its direct parent class. You could do this for all superclasses (so traverse all ancestors and find the registered function for the first that is encountered). I can work on that, if you agree. The one thing that I am not sure about is how you want to create the secondary dispatch table. An empty dict as class variable in Pickler? (It doesn't have to be a true dispatcher, I think.)

I do not think that dynamic registration is the ideal situation (it feels a bit hacky). An alternative would be to subclass Pickle and Dill to make sure that instead of just type() checking in the dispatch table also superclasses are considered. But that is probably overkill.

> You could do this for all superclasses (so traverse all ancestors and find the registered function for the first that is encountered)

That makes sense indeed !

> The one thing that I am not sure about is how you want to create the secondary dispatch table. An empty dict as class variable in Pickler? (It doesn't have to be a true dispatcher, I think.)

Sure, let's try to not use too complicated stuff

> I do not think that dynamic registration is the ideal situation (it feels a bit hacky). An alternative would be to subclass Pickle and Dill to make sure that instead of just type() checking in the dispatch table also superclasses are considered. But that is probably overkill.

Indeed that would feel less hacky, but maybe it's too complex just for this. I feel like this part of the library is already hard to understand when you're not familiar with pickle. IMO having only a few changes that are simpler to understand is better than having a rewrite of `dill`'s core code.

Thanks a lot for your insights, it looks like we're going to have something that works well and that unlocks some nice flexibility for users :) Feel free to ping me anytime if I can help on this

Sure, thanks for brainstorming! I'll try to work on it this weekend. Will also revert the current changes in this PR and rename it.

It seems like this is going in the right direction :).

@BramVanroy Just one small suggestion for future contributions: instead of using `WIP` in the PR title, you can create a draft PR if you're still working on it.

Maybe I should just create a new (draft) PR then, seeing that I'll have to rename and revert the changes anyway? I'll link to this PR so that the discussion is at least referenced.

I can convert this PR to a draft PR. Let me know what would you prefer.

I think reverting my previous commits would make for a dirty (or confusing) commit history, so I'll just create a new one. Thanks.

Inspired by the discussion on hashing in https://github.com/huggingface/datasets/issues/3178#issuecomment-959016329, @lhoestq suggested that it would be neat to allow users more control over the hashing process. Specifically, it would be great if users can specify specific hashing functions depending on the **class** of the object.

As an example, we found in the linked topic that loaded spaCy models (`Language` objects) have different hashes when `dump`'d, but their byte representation with `Language.to_bytes()` _is_ deterministic. It would therefore be great if we could specify that for `Language` objects, the hasher should hash the objects `to_bytes()` return value instead of the object itself.

This PR adds a new, but tiny, dependency to manage the registry, namely [`catalogue`](https://github.com/explosion/catalogue).

Two files have been changed (apart from the added dependency in `setup.py`) and one file has been added.

**utils.registry** (added)

This file defines our custom Registry and builds a registry called "hashers". A Registry is basically dictionary from names (str) to functions. A function can be added to the registry by a decorator, e.g.

```python
@hashers.register(spacy.Language)
def hash_spacy_language(nlp):
    return Hasher.hash(nlp.to_bytes())
```

You'll notice that `spacy.Language` is not a string, even though the registry holds a str->func mapping. To accomplish this with classes in a dynamic way, catalogue.Registry needed to be subclassed and modified as `DatasetsRegistry`. All methods that use a name as an input are now modified so that classes are deterministically converted in strings in such a way that we can later retrieve the actual class from the string (below).

**utils.py_utils** (modified)

Added two functions to deal with classes and their qualified names, that is, their full descriptive name including the module. On the one hand it allows us to retrieve a string from a given class, e.g. given `Module` class, return `torch.nn.Module` str. Conversly, a function is added to convert such a full qualified name into a class. For instance, given the string `torch.nn.Module`, return the `Module` class. These straightforward methods allow us to interchangeably use classes and strings without any needed user interaction - they can just register a class, and behind the scenes `DatasetsRegistry` converts these to deterministic strings.

**fingerprint** (modified)

Updated Hasher.hash so that if the object to hash is an instance of a class in the registry, the registered function is used to hash the object instead of the default behavior. To do so we iterate over the registry `hashers` and convert its keys (strings) into classes, and then we can use `isinstance`.

```python
# Check if the current object is an instance that is
# applicable to the user-defined hashers. If so, hash
# with the user-defined function
for full_module_name, func in hashers.get_all().items():
    registered_cls = get_cls_from_qualname(full_module_name)
    if isinstance(value, registered_cls):
        return func(value)
```

**Putting it all together**

To test this, you can try the following example with spaCy. First install spaCy from source and checkout a specific commit.

```shell
git clone https://github.com/explosion/spaCy.git
cd spaCy/
git checkout cab9209c3dfcd1b75dfe5657f10e52c4d847a3cf
cd ..

git clone https://github.com/BramVanroy/datasets.git
cd datasets
git checkout registry
pip install -e .
pip install ../spaCy
spacy download en_core_web_sm
```

Now you can run the following script. By default it will use the custom hasher function for the Language object. You can enable the default behavior by commenting out `@hashers.register...`.

```python
import spacy

from datasets.fingerprint import Hasher
from datasets.utils.registry import hashers

# Register a function so that when the Hasher encounters a spacy.Language object
# it uses this custom function to hash instead of the default
@hashers.register(spacy.Language)
def hash_spacy_language(nlp):
    return Hasher.hash(nlp.to_bytes())

def main():
    print(hashers.get_all())
    nlp = spacy.load("en_core_web_sm")
    dump1 = Hasher.hash(nlp)
    nlp = spacy.load("en_core_web_sm")
    dump2 = Hasher.hash(nlp)
    print(dump1)
    # succeeds when using the registered custom function
    # fails if using the default
    assert dump1 == dump2

if __name__ == '__main__':
    main()
```

To do
====
- The above is just a proof-of-concept. I am open to changes/suggestions
- Tests still need to be written
- We should consider whether we can make `DatasetsRegistry` very restrictive and ONLY allowing classes. That would make testing easier - otherwise we also need to test for other sorts of objects.
- Maybe the `hashers` definition is better suited in `fingerprint`?
- Documentation/examples need to be updated
- Not sure why the logger is not working in `hash()`
- `get_cls_from_qualname` might need a fail-safe: is it possible for a full_qualname to not have a module, and if so how do we deal with that?


Add Multidoc2dial Dataset

Hi @sivasankalpp, thanks for your PR.

There was a bug in TensorFlow/Keras. We have made a temporary fix in our master branch. Please, merge master into your PR branch, so that the CI tests pass.

```
git checkout multidoc2dial
git fetch upstream master
git merge upstream/master
```

Hi @albertvillanova, I have merged master into my PR branch. All tests are passing.
Please take a look when you get a chance, thanks!

Thanks for your feedback @lhoestq. We addressed your comments in the latest commit. Let us know if everything looks okay :)

This PR adds the MultiDoc2Dial dataset introduced in this [paper](https://arxiv.org/pdf/2109.12595v1.pdf )


FileNotFoundError for TupleIE dataste

Something isn't working

@mariosasko @lhoestq Could you give me an update on how to load the dataset after the fix?
Thanks.

Hi @arda-vianai,

first, you can try:
```python
import datasets
dataset = datasets.load_dataset('tuple_ie', 'all', revision="master")
```
If this doesn't work, your version of `datasets` is missing some features that are required to run the dataset script, so install the master version with the following command:
```
pip install git+https://github.com/huggingface/datasets.git
```
and then:
```python
import datasets
dataset = datasets.load_dataset('tuple_ie', 'all')
```
should work (even without `revision`).

@mariosasko
Thanks, it is working now. I actually did that before but I didn't restart the kernel. I restarted it and it works now. My bad!!!
Many thanks and great job!
-arda

Hi,
`dataset = datasets.load_dataset('tuple_ie', 'all')`

returns a FileNotFound error. Is the data not available?

Many thanks.


Updated: DaNE - updated URL for download

Actually it looks like the old URL is still working, and it's also the one that is mentioned in https://github.com/alexandrainst/danlp/blob/master/docs/docs/datasets.md

What makes you think we should use the new URL ?

@lhoestq Sorry! I might have jumped to conclusions a bit too fast here...

I was working in Google Colab and got an error that it was unable to use the URL. I then forked the project, updated the URL, ran it locally and it worked. I therefore assumed that my URL update fixed the issue, however, I see now that it might rather be a Google Colab issue...

Still - this seems to be the official URL for downloading the dataset, and I think that it will be most beneficial to use. :-)

It looks like they're using these new urls for their new datasets. Maybe let's change to the new URL in case the old one stops working at one point. Thanks

It seems that DaNLP has updated their download URLs and it therefore also needs to be updated in here...


Add mIoU metric

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Recently, some semantic segmentation models were added to HuggingFace Transformers, including [SegFormer](https://huggingface.co/transformers/model_doc/segformer.html) and [BEiT](https://huggingface.co/transformers/model_doc/beit.html).

Semantic segmentation (which is the task of labeling every pixel of an image with a corresponding class) is typically evaluated using the Mean Intersection and Union (mIoU). Together with the upcoming Image Feature, adding this metric could be very handy when creating example scripts to fine-tune any Transformer-based model on a semantic segmentation dataset.

An implementation can be found [here](https://github.com/open-mmlab/mmsegmentation/blob/504965184c3e6bc9ec43af54237129ef21981a5f/mmseg/core/evaluation/metrics.py#L132) for instance.


Add GSM8K dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** GSM8K (short for Grade School Math 8k)
- **Description:** GSM8K is a dataset of 8.5K high quality linguistically diverse grade school math word problems created by human problem writers.
- **Paper:** https://openai.com/blog/grade-school-math/
- **Data:** https://github.com/openai/grade-school-math
- **Motivation:** The dataset is useful to investigate the reasoning abilities of large Transformer models, such as GPT-3.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Catch token invalid error in CI

The staging back end sometimes returns invalid token errors when trying to delete a repo.
I modified the fixture in the test that uses staging to ignore this error


Bump huggingface_hub

huggingface_hub just released its first minor version, so we need to update the dependency

It was supposed to be part of 1.15.0 but I'm adding it for 1.15.1


Add Multi-Lingual LibriSpeech

Add https://www.openslr.org/94/


Fix optimized encoding for arrays

Hi !

#3124 introduced a regression that made the benchmarks CI fail because of a bad array comparison when checking the first encoded element. This PR fixes this by making sure that encoding is applied on all sequence types except lists.

cc @eladsegal fyi (no big deal)


QOL improvements: auto-flatten_indices and desc in map calls

This PR:
* automatically calls `flatten_indices` where needed: in `unique` and `save_to_disk` to avoid saving the indices file
* adds descriptions to the map calls

Fix #3040


More robust `None` handling

I also created a PR regarding `disable_nullable` that must be always `False` by default, in order to always allow None values
https://github.com/huggingface/datasets/pull/3211

@lhoestq I addressed your comments, added tests, did some refactoring to make the implementation cleaner and added support for `None` values in `map` transforms when the feature type is `ArrayXD` (previously, I only implemented `None` decoding).

My only concern is that during decoding `ArrayXD` arrays with `None` values will be auto-casted to `float64` to allow `np.nan` insertion and this might be unexpected if `dtype` is not `float`, so one option would be to allow `None` values only if the storage type is `float32` or `float64`. Let me know WDYT would be the most consistent behavior here.

Cool ! :D
> My only concern is that during decoding ArrayXD arrays with None values will be auto-casted to float64 to allow np.nan insertion and this might be unexpected if dtype is not float, so one option would be to allow None values only if the storage type is float32 or float64. Let me know WDYT would be the most consistent behavior here.

Yes that makes sense to only fill with nan if the type is compatible

After some more experimenting, I think we can keep auto-cast to float because PyArrow also does it:
```python
import pyarrow as pa
arr = pa.array([1, 2, 3, 4, None], type=pa.int32()).to_numpy(zero_copy_only=False) # None present - int32 -> float64
assert arr.dtype == np.float64
```
Additional changes:
* fixes a bug in the `_is_zero_copy_only` implementation for the ArraXD types. Previously, `_is_zero_copy_only` would always return False for these types. Still have to see if it's possible to optimize copying of the non-extension types (`Sequence`, ...), but I plan to work on that in a separate PR.
* https://github.com/huggingface/datasets/pull/2891 introduced a bug where the dtype of `ArrayXD` wouldn't be preserved due to `to_pylist` call in NumPy Formatter (`np.array(np.array(..).tolist())` doesn't necessarily preserve dtype of the initial array), so I'm also fixing that.

The CI fail for windows is unrelated to this PR, merging

PyArrow has explicit support for `null` values, so it makes sense to support Nones on our side as well.

[Colab Notebook with examples](https://colab.research.google.com/drive/1zcK8BnZYnRe3Ao2271u1T19ag9zLEiy3?usp=sharing)

Changes:
* allow None for the features types with special encoding (`ClassLabel, TranslationVariableLanguages, Value, _ArrayXD`)
* handle None in `class_encode_column` (also there is an option to stringify Nones and treat them as a class)
* support None sorting in `sort` (use pandas for that)
* handle None in align_labels_with_mapping
* support for None in ArrayXD (converts `None` to `np.nan` to align the behavior with PyArrow)
* support for None in the Audio/Image feature
* allow promotion when concatenating tables (`pa.concat_tables(table_list, promote=True)`) and `null` row/~~column~~ broadcasting similar to pandas

Additional notes:
* use `null` instead of `none` for function arguments for consistency with existing `disable_nullable`
* fixes a bug with the `update_metadata_with_features` call in `Dataset.rename_columns`
* had to update some tests, let me know if that's ok

TODO:
- [x] check how the Audio features behaves with Nones
- [x] Better None handling in `concatenate_datasets`/`add_item`
- [x] Fix formatting with Nones
- [x] Add Colab with examples
- [x] Tests

TODOs for subsequent PRs:
-  Mention None handling in the docs
- Add `drop_null`/`fill_null` to `Dataset`/`DatasetDict`

Fix #3181 #3253


Update link to Datasets Tagging app in Spaces

Fix #3193.


Update link to datasets-tagging app

Once datasets-tagging has been transferred to Spaces:
- huggingface/datasets-tagging#22

We should update the link in Datasets.


Multiprocessing filter/map (tests) not working on Windows

bug

Something isn't working

While running the tests, I found that the multiprocessing examples fail on Windows, or rather they do not complete: they cause a deadlock. I haven't dug deep into it, but they do not seem to work as-is. I currently have no time to tests this in detail but at least the tests seem not to run correctly (deadlocking).

## Steps to reproduce the bug
```shell
pytest tests/test_arrow_dataset.py -k "test_filter_multiprocessing"
pytest tests/test_arrow_dataset.py -k "test_map_multiprocessing"
```

## Expected results
The functionality to work on all platforms.

## Actual results
Deadlock.

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2, also tested with 3.7.9
- PyArrow version: 4.0.1


dataset-viewer

Related to the dataset viewer on huggingface.co

```python
>>> import datasets
>>> dataset = datasets.load_dataset('compguesswhat', name='compguesswhat-original',split='train', streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/compguesswhat/4d08b9e0a8d1cf036c9626c93be4a759fdd9fcce050ea503ea14b075e830c799/compguesswhat.py", line 251, in _generate_examples
    with gzip.open(filepath) as in_file:
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/gzip.py", line 173, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'zip://compguesswhat-original/0.2.0/compguesswhat.train.jsonl.gz::https://www.dropbox.com/s/l0nc13udml6vs0w/compguesswhat-original.zip?dl=1'
```

It's an issue with the streaming mode. Note that normal mode is used by the dataset viewer when streaming is failing, but only for the smallest datasets. This dataset is above the limit, hence the error.

Same case as https://github.com/huggingface/datasets/issues/3186#issuecomment-1096549774.

## Dataset viewer issue for '*compguesswhat*'

**Link:** https://huggingface.co/datasets/compguesswhat

File not found

Am I the one who added this dataset ? No


combination of shuffle and filter results in a bug

bug

Something isn't working

I cannot reproduce this on master and pyarrow==4.0.1.

Hi ! There was a regression in `datasets` 1.12 that introduced this bug. It has been fixed in #3019 in 1.13

Can you try to update `datasets` and try again ?

## Describe the bug
Hi,
I would like to shuffle a dataset, then filter it based on each existing label. however, the combination of `filter`, `shuffle` seems to results in a bug. In the minimal example below, as you see in the filtered results, the filtered labels are not unique, meaning filter has not worked. Any suggestions as a temporary fix is appreciated @lhoestq.

Thanks.
 Best regards
Rabeeh

## Steps to reproduce the bug
```python
import numpy as np
import datasets

datasets = datasets.load_dataset('super_glue', 'rte', script_version="master")
shuffled_data = datasets["train"].shuffle(seed=42)
for label in range(2):
    print("label ", label)
    data = shuffled_data.filter(lambda example: int(example['label']) == label)
    print("length ", len(data), np.unique(data['label']))
```

## Expected results
Filtering per label, should only return the data with that specific label.

## Actual results
As you can see, filtered data per label, has still two labels of [0, 1]
```
label  0
length  1249 [0 1]
label  1
length  1241 [0 1]
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: linux
- Python version: 3.7.11
- PyArrow version: 5.0.0


conll2003 incorrect label explanation

Something isn't working

Hi @BramVanroy,

since these fields are of type `ClassLabel` (you can check this with `dset.features`), you can inspect the possible values with:
```python
dset.features[field_name].feature.names  # .feature because it's a sequence of labels
```

and to find the mapping between names and integers, use:
```python
dset.features[field_name].feature.int2str(value_or_values_list)  # map integer value to string value
# or
dset.features[field_name].feature.str2int(value_or_values_list)  # map string value to integer value
```

In the [conll2003](https://huggingface.co/datasets/conll2003#data-fields) README, the labels are described as follows

> - `id`: a `string` feature.
> - `tokens`: a `list` of `string` features.
> - `pos_tags`: a `list` of classification labels, with possible values including `"` (0), `''` (1), `#` (2), `$` (3), `(` (4).
> - `chunk_tags`: a `list` of classification labels, with possible values including `O` (0), `B-ADJP` (1), `I-ADJP` (2), `B-ADVP` (3), `I-ADVP` (4).
> - `ner_tags`: a `list` of classification labels, with possible values including `O` (0), `B-PER` (1), `I-PER` (2), `B-ORG` (3), `I-ORG` (4) `B-LOC` (5), `I-LOC` (6) `B-MISC` (7), `I-MISC` (8).

First of all, it would be great if we can get a list of ALL possible pos_tags.

Second, the chunk tags labels cannot be correct. The description says the values go from 0 to 4 whereas the data shows values from at least 11 to 21 and 0.

EDIT: not really a bug, sorry for mistagging.


conll2002 issues

Hi ! Thanks for reporting :)

This is related to https://github.com/huggingface/datasets/issues/2742, I'm working on it. It should fix the viewer for around 80 datasets.

Ah, hadn't seen that sorry.

The scrambled "point of contact" is a separate issue though, I think.

@lhoestq The "point of contact" is still an issue.

It will be fixed in https://github.com/huggingface/datasets/pull/3274, thanks

**Link:** https://huggingface.co/datasets/conll2002

The dataset viewer throws a server error when trying to preview the dataset.

```
Message:       Extraction protocol 'train' for file at 'https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/esp.train' is not implemented yet
```

In addition, the "point of contact" has encoding issues and does not work when clicked.

Am I the one who added this dataset ? No, @lhoestq  did


Add ChrF(++) (as implemented in sacrebleu)

Similar to my [PR for TER](https://github.com/huggingface/datasets/pull/3153), it feels only right to also include ChrF and friends. These are present in Sacrebleu and are therefore very similar to implement as TER and sacrebleu. I tested the implementation with sacrebleu's tests to verify. You can try this below for yourself

```python
import datasets

EPSILON = 1e-4
chrf = datasets.load_metric(r"path\to\datasets\metrics\chrf")

test_cases = [
    (["abcdefg"], ["hijklmnop"], 0.0),
    (["a"], ["b"], 0.0),
    ([""], ["b"], 0.0),
    ([""], ["ref"], 0.0),
    ([""], ["reference"], 0.0),
    (["aa"], ["ab"], 8.3333),
    (["a", "b"], ["a", "c"], 8.3333),
    (["a"], ["a"], 16.6667),
    (["a b c"], ["a b c"], 50.0),
    (["a b c"], ["abc"], 50.0),
    ([" risk assessment must be made of those who are qualified and expertise in the sector - these are the scientists ."],
     ["risk assessment has to be undertaken by those who are qualified and expert in that area - that is the scientists ."], 63.361730),
    ([" Die    Beziehung zwischen  Obama und Netanjahu ist nicht gerade  freundlich. "],
     ["Das Verhältnis zwischen Obama und Netanyahu ist nicht gerade freundschaftlich."], 64.1302698),
    (["Niemand hat die Absicht, eine Mauer zu errichten"], ["Niemand hat die Absicht, eine Mauer zu errichten"], 100.0),
]

for hyp, ref, score in test_cases:
    # Note the reference transformation which is different from scarebleu's input format
    results = chrf.compute(predictions=hyp, references=[[r] for r in ref],
                           char_order=6, word_order=0, beta=3, eps_smoothing=True)
    if abs(score - results["score"]) > EPSILON:
        print(f"expected {score}, got {results['score']} for {hyp} - {ref}")

test_cases_effective_order = [
    (["a"], ["a"], 100.0),
    ([""], ["reference"], 0.0),
    (["a b c"], ["a b c"], 100.0),
    (["a b c"], ["abc"], 100.0),
    ([""], ["c"], 0.0),
    (["a", "b"], ["a", "c"], 50.0),
    (["aa"], ["ab"], 25.0),
]

for hyp, ref, score in test_cases_effective_order:
    # Note the reference transformation which is different from scarebleu's input format
    results = chrf.compute(predictions=hyp, references=[[r] for r in ref],
                           char_order=6, word_order=0, beta=3, eps_smoothing=False)
    if abs(score - results["score"]) > EPSILON:
        print(f"expected {score}, got {results['score']} for {hyp} - {ref}")

test_cases_keep_whitespace = [
    (
        ["Die Beziehung zwischen Obama und Netanjahu ist nicht gerade freundlich."],
        ["Das Verhältnis zwischen Obama und Netanyahu ist nicht gerade freundschaftlich."],
        67.3481606,
    ),
    (
        ["risk assessment must be made of those who are qualified and expertise in the sector - these are the scientists ."],
        ["risk assessment has to be undertaken by those who are qualified and expert in that area - that is the scientists ."],
        65.2414427,
    ),
]

for hyp, ref, score in test_cases_keep_whitespace:
    # Note the reference transformation which is different from scarebleu's input format
    results = chrf.compute(predictions=hyp, references=[[r] for r in ref],
                           char_order=6, word_order=0, beta=3,
                           whitespace=True)
    if abs(score - results["score"]) > EPSILON:
        print(f"expected {score}, got {results['score']} for {hyp} - {ref}")

predictions = ["The relationship between Obama and Netanyahu is not exactly friendly."]
references = [["The ties between Obama and Netanyahu are not particularly friendly."]]
print(chrf.compute(predictions=predictions, references=references))
```


Dataset viewer for nli_tr

It's an issue with the streaming mode:

```python
>>> import datasets
>>> dataset = datasets.load_dataset('nli_tr', name='snli_tr',split='test', streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/nli_tr/c2ddd0c0a70caddac6a81c2dae5ca7939f00060d517d08f1983927818dba6521/nli_tr.py", line 155, in _generate_examples
    with codecs.open(filepath, encoding="utf-8") as f:
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: 'zip://snli_tr_1.0_test.jsonl::https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip'
```

Note that normal mode is used by the dataset viewer when streaming is failing, but only for the smallest datasets. `nli_tr` is above the limit, hence the error.

## Dataset viewer issue for '*nli_tr*'

**Link:** https://huggingface.co/datasets/nli_tr

Hello,

Thank you for the new dataset preview feature that will help the users to view the datasets online.

We just noticed that the dataset viewer widget in the `nli_tr` dataset shows the error below. The error must be due to a temporary problem that may have blocked access to the dataset through the dataset viewer. But the dataset is currently accessible through the link in the error message.   May we kindly ask if it would be possible to rerun the job so that it can access the dataset for the dataset viewer function?

Thank you.
Emrah

------------------------------------------
Server Error
Status code:   404
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: 'zip://snli_tr_1.0_train.jsonl::https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip
------------------------------------------

Am I the one who added this dataset ? Yes


7z dataset preview not implemented?

It's a bug in the dataset viewer: the dataset cannot be downloaded in streaming mode, but since the dataset is relatively small, the dataset viewer should have fallback to normal mode. Working on a fix.

Fixed. https://huggingface.co/datasets/samsum/viewer/samsum/train

<img width="1563" alt="Capture d’écran 2022-04-12 à 13 47 45" src="https://user-images.githubusercontent.com/1676121/162953339-cd8922d7-9037-408b-b896-eac1af0bb54f.png">

Thanks for reporting!

## Dataset viewer issue for dataset 'samsum'

**Link:** https://huggingface.co/datasets/samsum

Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol '7z' for file at 'https://arxiv.org/src/1911.12237v2/anc/corpus.7z' is not implemented yet


RONEC v2

@lhoestq Thanks for the review. I totally understand what you are saying. Normally, I would definitely agree with you, but in this particular case, the quality of v1 is poor, and the dataset itself is small (at the time we created v1 it was the only RO NER dataset, and its size was limited by the available resources).

This is why we worked to build a larger one, with much better inter-annotator agreement. Fact is, models trained on v1 will be of very low quality and I would not recommend to anybody to use/do that. That's why I'd strongly suggest we replace v1 with v2, and kindof make v1 vanish :)

What do you think? If you insist on having v1 accessible, I'll add the required code. Thanks!

Ok I see ! I think it's fine then, no need to re-add V1

Hi, as we've recently finished with the new RONEC (Romanian Named Entity Corpus), we'd like to update the dataset here as well. It's actually essential as links to V1 are no longer valid.

In reality we'd like to replace completely v1, as v2 is a full re-annotation of v1 with additional data (up to 2x size vs v1).

I've run the make style and all the dummy and real data test, and they passed.

I hope it's okay to merge the new RONEC v2 in the datasets.

Thanks!


Add missing docstring to DownloadConfig

Document the `use_etag` and `num_proc` attributes in `DownloadConig`.


Don't memoize strings when hashing since two identical strings may have different python ids

This change slows down the hash computation a little bit but from my tests it doesn't look too impactful. So I think it's fine to merge this.

When hashing an object that has several times the same string, the hashing could return a different hash if the identical strings share the same python `id()` or not.

Here is an example code that shows how the issue can affect the caching:
```python
import json
import pyarrow as pa
from datasets.features import Features
from datasets.fingerprint import Hasher

schema = pa.schema([pa.field("some_string", pa.string()), pa.field("another_string", pa.string())])
features_from_schema = Features.from_arrow_schema(schema)
Hasher.hash(features_from_schema) # dffa9dca9a73fd8c

features_dict = json.loads('{"some_string": {"dtype": "string", "id": null, "_type": "Value"}, "another_string": {"dtype": "string", "id": null, "_type": "Value"}}')
features_from_json = Features.from_dict(features_dict)
Hasher.hash(features_from_json) # 3812e76b15e6420e

features_from_schema == features_from_json # True
```

This is because in `features_dict`, some strings like "dtype" are repeated but don't share the same id, contrary to the ones in `features_from_schema`.

I fixed that by disabling memoization for strings.

This could be optimized in the future by implementing a smarter memoization with a special handling for strings.


`None` converted to `"None"` when loading a dataset

Something isn't working

Hi @eladsegal, thanks for reporting.

@mariosasko I saw you are already working on this, but maybe my comment will be useful to you.

All values are casted to their corresponding feature type (including `None` values). For example if the feature type is `Value("bool")`, `None` is casted to `False`.

It is true that strings were an exception, but this was recently fixed by @lhoestq (see #3158).

Thanks for reporting.

This is actually a breaking change that I think can cause issues when users preprocess their data. String columns used to be nullable. Maybe we can correct https://github.com/huggingface/datasets/pull/3158 to keep the None values and avoid this breaking change ?

EDIT: the other types (bool, int, etc) can also become nullable IMO

So what would be the best way to handle a feature that can have a null value in some of the instances? So far I used `None`.
Using the empty string won't be a good option, as it can be an actual value in the data and is not the same as not having a value at all.

Hi @eladsegal,

Use `None`. As @albertvillanova correctly pointed out, this change in conversion was introduced (by mistake) in #3158. To avoid it, install the earlier revision with:
```
pip install git+https://github.com/huggingface/datasets.git@8107844ec0e7add005db0585c772ee20adc01a5e
```

I'm making all the feature types nullable as we speak, and the fix will be merged probably early next week.

Hi @mariosasko, is there an estimation as to when this issue will be fixed?

https://github.com/huggingface/datasets/pull/3195 fixed it, we'll do a new release soon :)

For now feel free to install `datasets` from the master branch

Thanks, but unfortunately looks like it isn't fixed yet 😢
[notebook for 1.14.0](https://colab.research.google.com/drive/1SV3sFXPJMWSQgbm4pr9Y1Q8OJ4JYKcDo?usp=sharing)
[notebook for master](https://colab.research.google.com/drive/145wDpuO74MmsuI0SVLcI1IswG6aHpyhi?usp=sharing)

Oh, sorry. I deleted the fix by accident when I was resolving a merge conflict. Let me fix this real quick.

## Describe the bug
When loading a dataset `None` values of the type `NoneType` are converted to `'None'` of the type `str`.

## Steps to reproduce the bug
```python
from datasets import load_dataset
qasper = load_dataset("qasper", split="train", download_mode="reuse_cache_if_exists")
print(qasper[60]["full_text"]["section_name"])
```

When installing version 1.1.40, the output is
`[None, 'Introduction', 'Benchmark Datasets', ...]`

When installing from the master branch, the output is
`['None', 'Introduction', 'Benchmark Datasets', ...]`

Notice how the first element was changed from `NoneType` to `str`.

## Expected results
`None` should stay as is.

## Actual results
`None` is converted to a string.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: master
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1


fix label mapping

heck, test failings. moving to draft. will come back to this later today hopefully

Thanks for fixing this :)
I just updated the dataset_infos.json and added the missing `pretty_name` tag to the dataset card

thank you @lhoestq! running around as always it felt through as a lower priority...

Fixing label mapping for hlgd.
0 correponds to same event and 1 corresponds to different event

nt


Cannot load dataset when the config name is "special"

Something isn't working

The issue is that the datasets are malformed. Not a bug with the datasets library

## Describe the bug

After https://github.com/huggingface/datasets/pull/3159, we can get the config name of "Check/region_1", which is "Check___region_1".

But now we cannot load the dataset (not sure it's related to the above PR though). It's the case for all the similar datasets, listed in https://github.com/huggingface/datasets-preview-backend/issues/78

## Steps to reproduce the bug
```python
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names("Check/region_1")
['Check___region_1']
>>> load_dataset("Check/region_1")
Using custom data configuration Check___region_1-d2b3bc48f11c9be2
Downloading and preparing dataset json/Check___region_1 to /home/slesage/.cache/huggingface/datasets/json/Check___region_1-d2b3bc48f11c9be2/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 4443.12it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1277.19it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1159, in _prepare_split
    writer.write_table(table)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/arrow_writer.py", line 442, in write_table
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/arrow_writer.py", line 442, in <listcomp>
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "pyarrow/table.pxi", line 1249, in pyarrow.lib.Table.__getitem__
  File "pyarrow/table.pxi", line 1825, in pyarrow.lib.Table.column
  File "pyarrow/table.pxi", line 1800, in pyarrow.lib.Table._ensure_integer_index
KeyError: 'Field "builder_name" does not exist in table schema'
```

Loading in streaming mode also returns something strange:

```python
>>> list(load_dataset("Check/region_1", streaming=True, split="train"))
Using custom data configuration Check___region_1-d2b3bc48f11c9be2
[{'builder_name': None, 'citation': '', 'config_name': None, 'dataset_size': None, 'description': '', 'download_checksums': None, 'download_size': None, 'features': {'speech': {'feature': {'dtype': 'float64', 'id': None, '_type': 'Value'}, 'length': -1, 'id': None, '_type': 'Sequence'}, 'sampling_rate': {'dtype': 'int64', 'id': None, '_type': 'Value'}, 'label': {'dtype': 'string', 'id': None, '_type': 'Value'}}, 'homepage': '', 'license': '', 'post_processed': None, 'post_processing_size': None, 'size_in_bytes': None, 'splits': None, 'supervised_keys': None, 'task_templates': None, 'version': None}, {'_data_files': [{'filename': 'dataset.arrow'}], '_fingerprint': 'f1702bb5533c549c', '_format_columns': ['speech', 'sampling_rate', 'label'], '_format_kwargs': {}, '_format_type': None, '_indexes': {}, '_indices_data_files': None, '_output_all_columns': False, '_split': None}]
```

## Expected results

The dataset should be loaded

## Actual results

An error occurs

## Environment info

- `datasets` version: 1.14.1.dev0
- Platform: Linux-5.11.0-1020-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1