NyssaW/github-issues huggingface dataset:
delete the hardcoded license list in `datasets`

> Feel free to delete the license list in `datasets` [...]
>
> Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238401662_

> [...], in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now?

_Originally posted by @julien-c in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238390659_

Comments:
fix: avoid casting tuples after Dataset.map

This PR updates features.py to avoid casting tuples to lists when reading the results of Dataset.map as suggested by @lhoestq [here](https://github.com/huggingface/datasets/issues/4676#issuecomment-1187371367) in https://github.com/huggingface/datasets/issues/4676.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix missing tags in dataset cards

Fix missing tags in dataset cards:
- aeslc
- empathetic_dialogues
- event2Mind
- gap
- iwslt2017
- newsgroup
- qa4mre
- scicite

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931
- #4979

Comments:
_The documentation is not available anymore as the PR was closed or merged._
"no-token" is passed to `huggingface_hub` when token is `None`

## Describe the bug

In the 2 lines listed below, a token is passed to `huggingface_hub` to get information from a dataset. If no token is provided, a "no-token" string is passed. What is the purpose of it ? If no real, I would prefer if the `None` value could be sent directly to be handle by `huggingface_hub`. I feel that here it is working because we assume the token will never be validated.

https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L753
https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L1121

## Expected results
Pass `token=None` to `huggingface_hub`.

## Actual results
`token="no-token"` is passed.

## Environment info
`huggingface_hub v0.10.0dev`

Comments:
Hi @Wauplin, thanks for raising this potential issue.

The choice of passing `"no-token"` instead of `None` was made in this PR:
- #4536 

According to the PR description, the reason why it is passed is to avoid that `HfApi.dataset_info` uses the local token when no token should be used.

Hi @albertvillanova , thanks for finding the original issue :+1: 

As of next release of `huggingface_hub`, the `token` argument will be deprecated in favor of the `use_auth_token` argument in `dataset_info` method. This change as been done by @SBrandeis in https://github.com/huggingface/huggingface_hub/pull/928. `use_auth_token` is a bit different and allow the case "don't sent the cached token by default".

If you want to strictly avoid sending the cached token from `datasets`, you can use:
```py
# token=token if token else "no-token", <- will fail because token is not valid

use_auth_token=token if token else False,  # using the new `use_auth_token` parameter
```

And as a note, I am currently updating the "don't send the cached token by default"-rule to "don't send the cached token on public repos by default but use it in private ones" in https://github.com/huggingface/huggingface_hub/pull/1064. This will not change the fact that `use_auth_token=False` doesn't send the token at all.


What is current strategy in term of updating `huggingface_hub` version in `datasets` ? I don't want to break stuff in the next release so let's find a proper solution :) 

As soon as `token` is deprecated and hfh has a new release, we'll update `datasets` to use the new argument instead. Does it sound good to you ?

Perfect :ok_hand: 

Hi @Wauplin, thanks for the warning about the deprecation of `token` in favor of `use_auth_token`.

Indeed, in datasets we use internally `use_auth_token`, which in this case was transformed to `token` to call `HfApi.dataset_info`:
https://github.com/huggingface/datasets/blob/1a9385d7cc8a3241b44015145ef56a230fdadc51/src/datasets/load.py#L747

Therefore, for the new hfh release, the fix will be trivial: we will pass directly `use_auth_token`.

As discussed during our meeting yesterday, due to the fact that at datasets we support multiple hfh versions, I think we should handle passing `token` or `use_auth_token` depending on the hfh version.
Running add_column() seems to corrupt existing sequence-type column info

I have a dataset that contains a column ("foo") that is a sequence type of length 4. So when I run .to_pandas() on it, the resulting dataframe correctly contains 4 columns - foo_0, foo_1, foo_2, foo_3. So the 1st row of the dataframe might look like:

ds = load_dataset(...)
df = ds.to_pandas()

df:
foo_0 | foo_1 | foo_2 | foo_3
0.0 | 1.0 | 2.0 | 3.0

If I run .add_column("new_col", data) on the dataset, and then .to_pandas() on the resulting new dataset, the resulting dataframe contains only 2 columns - foo, new_col. The values in column foo are lists of length 4, the 4 elements that should have been split into separate columns. Dataframe 1st row would be:

ds = load_dataset(...)
new_ds = ds.add_column("new_col", data)
df = new_ds.to_pandas()

df:
foo | new_col
[0.0, 1.0, 2.0, 3.0] | new_val

I've explored the 2 datasets in a debugger and haven't noticed any changes to any attributes related to the foo column, but I can't determine why the dataframes are so different.

Comments:
Nevermind, I was incorrect.
Add `IterableDataset.from_generator` to the API

We've just added `Dataset.from_generator` to the API. It would also be cool to add `IterableDataset.from_generator` to support creating an iterable dataset from a generator.

cc @lhoestq 

Comments:
#take

Thanks @hamid-vakilzadeh ! Let us know if you have some questions or if we can help

Thank you! I certainly will reach out if I need any help.
[doc] Fix broken snippet that had too many quotes

Hello!

### Pull request overview
* Fix broken snippet in https://huggingface.co/docs/datasets/main/en/process that has too many quotes

### Details
The snippet in question can be found here: https://huggingface.co/docs/datasets/main/en/process#map
This screenshot shows the issue, there is a quote too many, causing the snippet to be colored incorrectly:
![image](https://user-images.githubusercontent.com/37621491/190640627-f7587362-0e44-4464-a5d1-a0b98df6986f.png)

The change speaks for itself.

Thank you for the detailed documentation, by the way. 

- Tom Aarsen


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Spent the day familiarising myself with the huggingface line of products, and happened to run into some small issues here and there. Magically, I've found exactly one small issue in `transformers`, one in `accelerate` and now one in `datasets`, hah!

As for this PR, the issue seems solved according to the [new PR documentation](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4986/en/process#map):
![image](https://user-images.githubusercontent.com/37621491/190646405-6afa06fa-9eac-48f6-ab30-2677944fb7b6.png)
Prefer split patterns from directories over split patterns from filenames

related to https://github.com/huggingface/datasets/issues/4895


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can we merge this one since the issue this PR fixes was reported for the second time? I also think we don't need a test for this simple change.

@mariosasko sure! could you please approve it? 

Hi there @polinaeterna @mariosasko! I have installed 5.2.3.dev0, which should have this fix. Unfortunately, I am still getting the error:
`ValueError: Unknown split "validation". Should be one of ['train'].` When I call `load_dataset("csv", data_files=files, split=split)`

Any help would be greatly appreciated!
docs: ✏️ add links to the Datasets API

I added some links to the Datasets API in the docs. See https://github.com/huggingface/datasets-server/pull/566 for a companion PR in the datasets-server. The idea is to improve the discovery of the API through the docs.

I'm a bit shy about pasting a lot of links to the API in the docs, so it's minimal for now. I'm interested in ideas to integrate the API better in these docs without being too much. cc @lhoestq @julien-c @albertvillanova @stevhliu.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

OK, thanks @lhoestq. I'll close this PR, and come back to it with @stevhliu once we work on https://github.com/huggingface/datasets-server/issues/568
Make `pyarrow` optional

**Is your feature request related to a problem? Please describe.**
Is `pyarrow` really needed for every dataset?

**Describe the solution you'd like**
It is made optional.

**Describe alternatives you've considered**
Likely, no.


Comments:
The whole datasets library is pretty much a wrapper to pyarrow (just take a look at some of the source for a Dataset) https://github.com/huggingface/datasets/blob/51aef08ad7053c0bfe8f9a961207b26df15850d3/src/datasets/arrow_dataset.py#L639 

I think removing the pyarrow dependency would involve a complete rewrite / a different library with minimal functionality (datasets-lite ?)

Thanks for the proposal, @KOLANICH. And also thanks for your answer, @dconathan.

Indeed, we are using `pyarrow` as the backend for our datasets, in order to cache them and also allow memory-mapping (using datasets larger than your RAM memory).

One way to avoid using `pyarrow` could be loading the datasets in streaming mode, by passing `streaming=True` to `load_dataset`. This way you basically get a generator for the dataset; nothing is downloaded, nor cached. 

Thanks for the info. Could `datasets` then be made optional for `transformers` instead? I used `transformers` only to deal with pretrained models to deploy them (convert to ONNX, and then I use TVM), so I don't really need `pyarrow` and `datasets` by now.
Fix missing tags in dataset cards

Fix missing tags in dataset cards:
- amazon_us_reviews
- art
- discofuse
- indic_glue
- ubuntu_dialogs_corpus

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Providing dataset size

**Is your feature request related to a problem? Please describe.**
Especially for big datasets like [LAION](https://huggingface.co/datasets/laion/laion2B-en/), it's hard to know exactly the downloaded size (because there are many files and you don't have their exact size when downloaded).

**Describe the solution you'd like**
Auto-populating the downloaded dataset size on the dataset page would be really useful, including that of each split (when there are some).

**Describe alternatives you've considered**
People should be adding this to dataset cards, but I don't think that is systematically the case :slightly_smiling_face: 

**Additional context**
Mentioned to @lhoestq 


Comments:
Hi @sashavor, thanks for your suggestion.

Until now we have the CLI command 
```
datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
```
that generates the `dataset_infos.json` with the size of the downloaded dataset, among other information.

We are currently in the middle of removing those JSON files and putting their information directly in the header of the `README.md` (as YAML tags). Normally, the CLI command should continue working but saving its output to the dataset card instead. See:
- #4926

Additionally, the download size can be inferred by doing HEAD requests to the files to be downloaded. And for files hosted on the hub you can even get the file sizes using the Hub API

Amazing @albertvillanova ! I think just having that information visible in the dataset info (without having to do any requests/additional coding) would be really useful :hugs:
Hope to adapt Python3.9 as soon as possible

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.


Comments:
Hi! `datasets` should work in Python 3.9. What kind of issue have you encountered?

There is this related issue already: https://github.com/huggingface/datasets/issues/4113
And I guess we need a CI job for 3.9 ^^

Perhaps we should report this issue in the `filelock` repo?
Add `fn_kwargs` param to `IterableDataset.map`

Add the `fn_kwargs` parameter to `IterableDataset.map`.

("Resolves" https://discuss.huggingface.co/t/how-to-use-large-image-text-datasets-in-hugging-face-hub-without-downloading-for-free/22780/3)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for adding this fix! 

Would it be possible to get `fn_kwargs` added to `IterableDatasetDict.map` as well? It looks like a very similar problem, and hopefully shouldn't be a huge change. 


Hi @brianhill11! https://github.com/huggingface/datasets/pull/5810 adds this (opened a couple of days ago). It should be merged soon.

That's fantastic news, thanks @mariosasko ! I'll give it a shot once the changes are merged in.
[GH->HF] Part 2: Remove all dataset scripts from github

Now that all the datasets live on the Hub we can remove the /datasets directory that contains all the dataset scripts of this repository

- [x] Needs https://github.com/huggingface/datasets/pull/4973 to be merged first
- [x] and PR to be enabled on the Hub for non-namespaced datasets

Comments:
_The documentation is not available anymore as the PR was closed or merged._

So this means metrics will be deleted from this repo in favor of the "evaluate" library? Maybe you guys could just redirect metrics to that library.

We are deprecating the metrics in `datasets` indeed and suggest users to switch to `evaluate` (via a warning message)

We'll keep the current metrics as they are for now, but they'll be completely removed at one point

I guess this is ready to merge ?

It should break nothing except one rare case:

If someone is using an old version of `datasets` to try to load a recent dataset. Indeed in that case it fetches the `main` branch on github to see if it exists. But since we're removing all the datasets, forward fetching won't work anymore.

e.g. if someone uses "imagenet-1k" with a version of `datasets` that didn't have it at that time. I checked on kibana and one single user would be affected with 4k downloads/months. It should still work for them though thanks to the `datasets` cache

But if they delete their cache, the workaround is... 🥁 update `datasets` 😅

Let's merge this on monday if we can, to make sure contributors who wanted to merge their dataset PRs here could do it

Alright, merging !
[GH->HF] Load datasets from the Hub

Currently datasets with no namespace (e.g. squad, glue) are loaded from github.

In this PR I changed this logic to use the Hugging Face Hub instead.

This is the first step in removing all the dataset scripts in this repository

related to discussions in https://github.com/huggingface/datasets/pull/4059 (I should have continued from this PR actually)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Duplicate of:
- #4059
Fix map batched with torch output

Reported in https://discuss.huggingface.co/t/typeerror-when-applying-map-after-set-format-type-torch/23067/2

Currently it fails  if one uses batched `map` and the map function returns a torch tensor.

I fixed it for torch, tf, jax and pandas series.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Preserve non-`input_colums` in `Dataset.map` if `input_columns` are specified

Currently, if the `input_columns` list in `Dataset.map` is specified, the columns not in that list are dropped after the `map` transform.

This makes the behavior inconsistent with `IterableDataset.map`.
 
(It seems this issue was introduced by mistake in https://github.com/huggingface/datasets/pull/2246) 

Fix https://github.com/huggingface/datasets/issues/4858

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix data URL and metadata of vivos dataset

After contacting the authors of the VIVOS dataset to report that their data server is down, we have received a reply from Hieu-Thi Luong that their data is now hosted on Zenodo: https://doi.org/10.5281/zenodo.7068130

This PR updates their data URL and some metadata (homepage, citation and license).

Fix #4936.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Dataset without script does not support regular JSON data file

### Link

https://huggingface.co/datasets/julien-c/label-studio-my-dogs

### Description

<img width="1115" alt="image" src="https://user-images.githubusercontent.com/326577/189422048-7e9c390f-bea7-4521-a232-43f049ccbd1f.png">


### Owner

Yes

Comments:
Hi @julien-c,

Out of the box, we only support JSON lines (NDJSON) data files, but your data file is a regular JSON file. The reason is we use `pyarrow.json.read_json` and this only supports line-delimited JSON.
Update setup.py

exclude broken version of fsspec. See the [related issue](https://github.com/huggingface/datasets/issues/4961)

Comments:
Before addressing this PR, we should be sure about the issue. See my comment in:
- https://github.com/huggingface/datasets/issues/4961#issuecomment-1243376247

Once we know 2022.8.2 works, I'm closing this PR, as the corresponding issue.
Fix data URLs of compguesswhat dataset

After we informed the `compguesswhat` dataset authors about an error with their data URLs, they have updated them:
- https://github.com/CompGuessWhat/compguesswhat.github.io/issues/1

This PR updates their data URLs in our loading script.

Related to:
- #3191

Comments:
_The documentation is not available anymore as the PR was closed or merged._
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.4.0/datasets/jsonl/jsonl.py

Hi,
When I use load_dataset from local jsonl files, below error happens, and I type the link into the browser prompting me `404: Not Found`. I download the other `.py` files using the same method and it works. It seems that the server is missing the appropriate file, or it is a problem with the code version.

```
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b08342004c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")))

```



Comments:
I have solved this problem... The extension of the file should be `.json` not `.jsonl`
Add `Dataset.from_generator`

Add `Dataset.from_generator` to the API to allow creating datasets from data larger than RAM. The implementation relies on a packaged module not exposed in `load_dataset` to tie this method with `datasets`' caching mechanism.

Closes https://github.com/huggingface/datasets/issues/4417

Comments:
I restarted the builder PR job just in case

_The documentation is not available anymore as the PR was closed or merged._

CI is now green. https://github.com/huggingface/doc-builder/pull/296 explains why it failed.
Raise a more precise error when the URL is unreachable in streaming mode

See for example:

- https://github.com/huggingface/datasets/issues/3191
- https://github.com/huggingface/datasets/issues/3186

It would help provide clearer information on the Hub and help the dataset maintainer solve the issue by themselves quicker. Currently:

- https://huggingface.co/datasets/compguesswhat

  <img width="1029" alt="Capture d’écran 2022-09-08 à 15 51 37" src="https://user-images.githubusercontent.com/1676121/189139946-6deffb91-f21b-4281-8825-a98026c69740.png">

- https://huggingface.co/datasets/nli_tr

  <img width="1032" alt="Capture d’écran 2022-09-08 à 15 51 44" src="https://user-images.githubusercontent.com/1676121/189139963-d26490ed-ad23-48ea-9cfc-1ab9c4d08d0c.png">


cc @albertvillanova 

Comments:
CI test of TensorFlow is failing

## Describe the bug
The following CI test fails: https://github.com/huggingface/datasets/runs/8246722693?check_suite_focus=true
```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - AssertionError:
```

Details:
```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw0] linux -- Python 3.7.13 /opt/hostedtoolcache/Python/3.7.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        def gen_random_output():
            model = layers.Dense(2)
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
        with temp_seed(42, set_tensorflow=True):
            out1 = gen_random_output()
        with temp_seed(42, set_tensorflow=True):
            out2 = gen_random_output()
        out3 = gen_random_output()
    
>       np.testing.assert_equal(out1, out2)
E       AssertionError: 
E       Arrays are not equal
E       
E       Mismatched elements: 2 / 2 (100%)
E       Max absolute difference: 0.84619296
E       Max relative difference: 16.083529
E        x: array([[-0.793581,  0.333286]], dtype=float32)
E        y: array([[0.052612, 0.539708]], dtype=float32)

tests/test_py_utils.py:149: AssertionError
```




Comments:
Add test-datasets CI job

To avoid having too many conflicts in the datasets and metrics dependencies I split the CI into test and test-catalog

test does the test of the core of the `datasets` lib, while test-catalog tests the datasets scripts and metrics scripts

This also makes `pip install -e .[dev]` much smaller for developers

WDYT @albertvillanova ?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Closing this one since the dataset scripts will be removed in https://github.com/huggingface/datasets/pull/4974
Update Enwik8 broken link and information

The current enwik8 dataset link give a 502 bad gateway error which can be view on https://huggingface.co/datasets/enwik8 (click the dropdown to see the dataset preview, it will show the error). This corrects the links, and json metadata as well as adds a little bit more information about enwik8.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update enwik8 fixing the broken link

The current enwik8 dataset link give a 502 bad gateway error which can be view on https://huggingface.co/datasets/enwik8 (click the dropdown to see the dataset preview, it will show the error). This corrects the links, and json metadata as well as adds a little bit more information about enwik8.

Comments:
Closing pull request to following contributing guidelines of making a new branch and will make a new pull request
larger dataset, larger GPU memory in the training phase? Is that correct?

    from datasets import set_caching_enabled
    set_caching_enabled(False)
    for ds_name in ["squad","newsqa","nqopen","narrativeqa"]:
            train_ds = load_from_disk("../../../dall/downstream/processedproqa/{}-train.hf".format(ds_name))

        break
    train_ds = concatenate_datasets([train_ds,train_ds,train_ds,train_ds]) #operation 1


   trainer = QuestionAnsweringTrainer( #huggingface trainer
        model=model,
        args=training_args,
        train_dataset=train_ds,
        eval_dataset= None,
        eval_examples=None,
        answer_column_name=answer_column,
        dataset_name="squad",
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
    )

with operation 1, the GPU memory increases from 16G to 23G

Comments:
does the trainer save it in GPU? sooo curious... how to fix it

It's my bad. didn't limit the input length
Add splits to MBPP dataset

This PR addresses https://github.com/huggingface/datasets/issues/4795

Comments:
```
(env) cwarny@Cedrics-Air datasets % RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_mbpp
================================================================================================ test session starts =================================================================================================
platform darwin -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0
rootdir: /Users/cwarny/datasets, configfile: setup.cfg
collected 1 item                                                                                                                                                                                                     

tests/test_dataset_common.py .                                                                                                                                                                                 [100%]

================================================================================================= 1 passed in 1.12s ==================================================================================================
(env) cwarny@Cedrics-Air datasets % RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_mbpp               
================================================================================================ test session starts =================================================================================================
platform darwin -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0
rootdir: /Users/cwarny/datasets, configfile: setup.cfg
collected 1 item                                                                                                                                                                                                     

tests/test_dataset_common.py .                                                                                                                                                                                 [100%]

================================================================================================= 1 passed in 0.35s ==================================================================================================

```

_The documentation is not available anymore as the PR was closed or merged._

Hi @cwarny ! Thanks for adding the correct splits :)

You can fix the CI error by running `make style` - this should reformat the dataset script

done
Trec Dataset has incorrect labels

## Describe the bug
Both coarse and fine labels seem to be out of line.


## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = "trec"
raw_datasets = load_dataset(dataset)
df = pd.DataFrame(raw_datasets["test"])
df.head()
```

## Expected results
text (string) | coarse_label (class label) | fine_label (class label)
-- | -- | --
How far is it from Denver to Aspen ? | 5  	(NUM) | 40  	(NUM:dist)
What county is Modesto , California in ? | 4  	(LOC) | 32  	(LOC:city)
Who was Galileo ? | 3  	(HUM) | 31  	(HUM:desc)
What is an atom ? | 2  	(DESC) | 24  	(DESC:def)
When did Hawaii become a state ? | 5  	(NUM) | 39  	(NUM:date)

## Actual results
  index | label-coarse  |label-fine                  |                    text
-- |-- | -- | --
0    |         4     |     40     | How far is it from Denver to Aspen ?
1      |       5        |  21  | What county is Modesto , California in ?
2        |     3        |  12        |                 Who was Galileo ?
3         |    0       |    7         |                What is an atom ?
4         |    4       |    8       |   When did Hawaii become a state ?

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-1086-azure-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3



Comments:
Thanks for reporting, @wmpauli. 

Indeed we recently fixed this issue:
- #4801 

The fix will be accessible after our next library release. In the meantime, you can have it by passing `revision="main"` to `load_dataset`.
Remove deprecated identical_ok

`huggingface-hub` says that the `identical_ok` argument of `HfApi.upload_file` is now deprecated, and will be removed soon. It even has no effect at the moment when it's passed:

```python
Args:
...
    identical_ok (`bool`, *optional*, defaults to `True`):
        Deprecated: will be removed in 0.11.0.
        Changing this value has no effect.
...
```

There was only one occurence of `identical_ok=False` but it's maybe not worth adding a check ti verify if the files were the same.

cc @mariosasko 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
vivos (Vietnamese speech corpus) dataset not accessible

## Describe the bug
VIVOS data is not accessible anymore, neither of these links work (at least from France):
* https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (data)
* https://ailab.hcmus.edu.vn/vivos (dataset page) 

Therefore `load_dataset` doesn't work.

## Steps to reproduce the bug
```python
ds = load_dataset("vivos")
```

## Expected results
dataset loaded

## Actual results
```
ConnectionError: Couldn't reach https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='ailab.hcmus.edu.vn', port=443): Max retries exceeded with url: /assets/vivos.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d8a27d190>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))")))
```

Will try to contact the authors, as we wanted to use Vivos as an example in documentation on how to create scripts for audio datasets (https://github.com/huggingface/datasets/pull/4872), because it's small and straightforward and uses tar archives. 

Comments:
If you need an example of a small audio datasets, I just created few hours ago a speech dataset with only 300MB of compressed audio files https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia. It works also with streaming (@albertvillanova helped me adding this functionality) :-)

@cahya-wirawan omg this is awesome!! thank you! 

We have contacted the authors to ask them.
Dataset Viewer issue for ubuntu_dialogs_corpus

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments:
The dataset maintainers (https://huggingface.co/datasets/ubuntu_dialogs_corpus) decided to forbid the dataset from being downloaded automatically (https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download), and the dataset viewer respects this.
We will try to improve the error display though. Thanks for reporting.
Dataset Viewer issue for bigscience-biomedical/biosses

### Link

https://huggingface.co/datasets/bigscience-biomedical/biosses

### Description

I've just been working on adding the dataset loader script to this dataset and working with the relative imports. I'm not sure how to interpret the error below (show where the dataset preview used to be) . 
```
Status code:   400
Exception:     ModuleNotFoundError
Message:       No module named 'datasets_modules.datasets.bigscience-biomedical--biosses.ddbd5893bf6c2f4db06f407665eaeac619520ba41f69d94ead28f7cc5b674056.bigbiohub'
```

### Owner

Yes

Comments:
Possibly not related to the dataset viewer in itself. cc @huggingface/datasets.

In particular, I think that the import of bigbiohub is not working here: https://huggingface.co/datasets/bigscience-biomedical/biosses/blob/main/biosses.py#L29 (requires a relative path?)

```python
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names('bigscience-biomedical/biosses')
Downloading builder script: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.00k/8.00k [00:00<00:00, 7.47MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 289, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1247, in dataset_module_factory
    raise e1 from None
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1220, in dataset_module_factory
    return HubDatasetModuleFactoryWithScript(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 931, in get_module
    local_imports = _download_additional_modules(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 215, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use bigscience-biomedical/biosses, you need to install the following dependency: bigbiohub.
Please install it using 'pip install bigbiohub' for instance'
```

Opened a PR here to (hopefully) fix the dataset script: https://huggingface.co/datasets/bigscience-biomedical/biosses/discussions/1/files

thanks for taking a look @severo . agree this isn't related to dataset viewer (sorry just clicked on the auto issue creator). also thanks @lhoestq , I see the format to use for relative imports. was a bit confused b/c it seems to be working here 

https://huggingface.co/datasets/bigscience-biomedical/scitail/blob/main/scitail.py#L31

I'll try this PR a see what happens. 

closing as I think the issue is relative imports and attempting to read json files directly in the repo (thanks again @lhoestq )
Fix missing tags in dataset cards

Fix missing tags in dataset cards:
- coqa
- hyperpartisan_news_detection
- opinosis
- scientific_papers
- scifact
- search_qa
- wiki_qa
- wiki_split
- wikisql

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add cc-by-nc-2.0 to list of licenses

This PR adds the `cc-by-nc-2.0` to the list of licenses because it is required by `scifact` dataset: https://github.com/allenai/scifact/blob/master/LICENSE.md

Comments:
_The documentation is not available anymore as the PR was closed or merged._

this list needs to be kept in sync with the ones in moon-landing and hub-docs :)

@julien-c don't you think it might be better to a have a single file (source of truth) in one of the repos and then use it in every other repo, instead of having 3 copies of the same file that must be kept in sync?

Also note that the licenses we are adding were all already present in our previous `licenses.json` file: are we regenerating it, step by step? Why don't we use a file with ALL the licenses we previously had in the list?

Licenses added:
- #4887
- #4930 

Previous `licenses.json` file:
- https://github.com/huggingface/datasets/blob/b7612754928e0fd43b9e3c3becb906ec280ff5d4/src/datasets/utils/resources/licenses.json
- removed in this commit: https://github.com/huggingface/datasets/pull/4613/commits/9f7725412dac1089b3e057f9e3fcf39cc222bc26

Let me know what you think and I can take care of this.

> Let me know what you think and I can take care of this.

What I think is that we shouldn't add licenses that are just used in a couple of datasets, and just use `license_details` for this.

> don't you think it might be better to a have a single file (source of truth) in one of the repos and then use it in every other repo, instead of having 3 copies of the same file that must be kept in sync?

Yes, in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now? 


Feel free to delete the license list in `datasets` @albertvillanova ;)

Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)
Concatenate_datasets loads everything into RAM

## Describe the bug
When loading the datasets seperately and saving them on disk, I want to concatenate them. But `concatenate_datasets` is filling up my RAM and the process gets killed. Is there a way to prevent this from happening or is this intended behaviour? Thanks in advance

## Steps to reproduce the bug
```python
gcs = gcsfs.GCSFileSystem(project='project')
datasets = [load_from_disk(f'path/to/slice/of/data/{i}', fs=gcs, keep_in_memory=False) for i in range(10)]

dataset = concatenate_datasets(datasets)
```

## Expected results
A concatenated dataset which is stored on my disk.

## Actual results
Concatenated dataset gets loaded into RAM and overflows it which gets the process killed.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 8.0.1
- Pandas version: 1.4.3

Comments:
decode mp3 with librosa if torchaudio is > 0.12 as a temporary workaround 

`torchaudio>0.12` fails with decoding mp3 files if `ffmpeg<4`. currently we ask users to downgrade torchaudio, but sometimes it's not possible as torchaudio version is binded to torch version. as a temporary workaround we can decode mp3 with librosa (though it 60 times slower, at least it works)

another option would be to ask users to install the required version of `ffmpeg`, but is non-trivial on colab: it's not in apt packages in ubuntu 18 and `conda` is not preinstalled (with `conda` it would be easily installable)

- [x] decode with torchaudio anyway if the version of ffmpeg is correct? it's 60 times faster
- [x] tests 
- [x] DO NOT FORGET to get back all the tests

see https://github.com/huggingface/datasets/issues/4776 and https://github.com/huggingface/datasets/issues/3663#issuecomment-1225797165 (there is a Colab notebook to reproduce the error)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks ! Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

@lhoestq 

> Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

I'm not sure here, because from the one hand, if `torchaudio` works - it works 60 times faster then `librosa`.
But from the other hand, we will get inconsistent behavior (=different results of decoding) for users of `torchaudio>=0.12`. 
I'd better go for using `librosa` only to avoid inconsistency then. wdyt?

It seems a bit too constraining to not allow users who have a working torchaudio 0.12 setup to not use it. 

If the issue is about avoiding silent errors if the decoding changes, maybe we can log which back-end is used ? It can even be a warning with performance suggestions ("you're using librosa but torchaudio 0.xx is recommended").

Note that users can still have a requirements.txt or whatever in their projects if they really want full reproducibility (and it's the bare minimum imo)

There are multiple possible back-ends so it's maybe not reasonable to only allow one back-end, especially since each back-end has installation constrains and there's no "best" back-end.

Woohoo all green ! Feel free to merge if it's all good for you :)
Fix missing tags in dataset cards

Fix missing tags in dataset cards:
- eraser_multi_rc
- hotpot_qa
- metooma
- movie_rationales
- qanta
- quora
- quoref
- race
- ted_hrlr
- ted_talks_iwslt

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Unable to load local tsv files through load_dataset method

## Describe the bug
Unable to load local tsv files through load_dataset method.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
data_files = {
    'train': 'train.tsv',
    'test': 'test.tsv'
}
raw_datasets = load_dataset('tsv', data_files=data_files)

## Expected results
I am pretty sure the data files exist in the current directory. The above code should load them as Datasets, but threw exceptions.

## Actual results
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-9-24207899c1af>](https://localhost:8080/#) in <module>
----> 1 raw_datasets = load_dataset('tsv', data_files='train.tsv')

2 frames
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1244                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1245                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
-> 1246                     ) from None
   1247                 raise e1 from None
   1248     else:

FileNotFoundError: Couldn't find a dataset script at /content/tsv/tsv.py or any data file in the same directory. Couldn't find 'tsv' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/main/datasets/tsv/tsv.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments:
Hi @DataNoob0723,

Under the hood, we use `pandas` to load CSV/TSV files. Therefore, you should use "csv" and pass `sep="\t"`, as explained in our docs: https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/loading_methods#from-files
```python
ds = load_dataset('csv', sep="\t", data_files=data_files)
```
Dataset Viewer issue for pysentimiento/spanish-targeted-sentiment-headlines

### Link

https://huggingface.co/datasets/pysentimiento/spanish-targeted-sentiment-headlines

### Description

After moving the dataset from my user (`finiteautomata`) to the `pysentimiento` organization, the dataset viewer says that it doesn't exist.

### Owner

_No response_

Comments:
Thanks for reporting, it's fixed now (I refreshed it manually). It's a known issue; we hope it will be fixed permanently in a few days.

<img width="1508" alt="Capture d’écran 2022-09-05 à 18 31 22" src="https://user-images.githubusercontent.com/1676121/188489762-0ed86a7e-dfb3-46e8-a125-43b815a2c6f4.png">


Thanks @severo!
Keys mismatch: make error message more informative

**Is your feature request related to a problem? Please describe.**
When loading a dataset from disk with a defect in its `dataset_info.json` describing its features (I don’t know when/why/how this happens but it deserves its own issue), you will get an error message like:
`ValueError: Keys mismatch: between {'bar': Value(dtype='int64', id=None)} and {'foo': Value(dtype='int64', id=None)}`

Which is fine when you have only a few features like in the example but it gets very hard to read when you have a lot of features in your dataset.

**Describe the solution you'd like**
The error message should give the difference between the features (what keys are in A but missing in B and vice-versa). It should also tell which keys are inferred from `dataset.arrow` and which come from `dataset_info.json`.

Willing to help :)


Comments:
Good idea ! I think this can be improved in `Features.reorder_fields_as()` indeed at

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/features/features.py#L1739-L1740

Is it something you would be interested in contributing ?

Is this open to work on? I'd love to take on this as my first issue.

Hi @daspartho I’ve opened a PR #4919 
I don’t think there’s much left to do

ok : )
[Tests] Ensure `datasets` supports renamed repositories

On https://hf.co/datasets you can rename a dataset (or sometimes move it to another user/org). The website handles redirections correctly and AFAIK `datasets` does as well.

However it would be nice to have an integration test to make sure we don't break support for renamed datasets.

To implement this we can use the /api/repos/move endpoint on hub-ci to rename/move a repo (it is documented at https://huggingface.co/docs/hub/api)

Comments:
You could also switch to using `huggingface_hub` more directly, where such a guarantee is already tested =)

cc @Wauplin 

Hi! @lhoestq, I'd love to work on this issue as part of my ongoing contributions to `datasets`.

I'm planning to implement an integration test that ensures `load_dataset()` works even after a dataset repo is renamed or moved. I can work locally and interact with the `/api/repos/move` endpoint, using either `huggingface_hub` or `requests`, and will also run the test suite to validate behavior end-to-end.

Before I begin, I have two quick questions:
1. Is there a recommended test dataset or sandbox repo that I should use for safely renaming/moving during tests?
2. Do I need special permissions or tokens to use `hub-ci` or to move repos between users/orgs?

Once confirmed, I’ll proceed with writing and validating the test locally and open a PR soon after. Thank you!
Update GLUE evaluation metadata

This PR updates the evaluation metadata for GLUE to:

* Include defaults for all configs except `ax` (which only has a `test` split with no known labels)
* Fix the default split from `test` to `validation` since `test` splits in GLUE have no labels (they're private)
* Fix the `task_id` for some existing defaults

cc @sashavor @douwekiela 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix missing tags in dataset cards

Fix missing tags in dataset cards:
- asnq
- clue
- common_gen
- cosmos_qa
- guardian_authorship
- hindi_discourse
- py_ast
- x_stance

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896

Comments:
_The documentation is not available anymore as the PR was closed or merged._
None Type error for swda datasets

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10


Comments:
Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?

Thanks a lot for your response @lhoestq 
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.

Ok, let us know if you encounter the issue again ;)