josteece/github-issues huggingface dataset:
Add option to find unique elements in nested sequences when calling `Dataset.unique`
Opened by mariosasko (open)

It would be nice to have an option to flatten nested sequences to find unique elements stored in them when calling `Dataset.unique`. ~~Currently, `Dataset.unique` only supports finding unique sequences and not unique elements in that situation.~~

Comments:
Hi @mariosasko!

Has this been patched into any of the releases?

Hi! Not yet, would you be interested in contributing a PR? I can give you some pointers if needed. 

@mariosasko did this ever get implemented? Willing to help if you are still up for it.

@dcruiz01 No, but here is an example of how to do this with the existing API:


```python
ds = Dataset.from_dict({"tokens": [["a", "b"], ["c", "a"], ["c", "e"]]})

def flatten_tokens(pa_table):
    return pa.table([pc.list_flatten(pa_table["tokens"])], ["flat_tokens"])

ds = ds.with_format("arrow")
ds = ds.map(flatten_tokens, batched=True)
ds = ds.with_format(None)

unique_tokens = ds.unique("flat_tokens")
```

When I think about it, `.unique` on `Sequence(Value(...))` should return unique sequences/arrays, not unique elements of these sequences...
Agnews dataset viewer is not working
Opened by pietrolesci (closed)

## Dataset viewer issue for '*name of the dataset*'

**Link:** https://huggingface.co/datasets/ag_news

Hi there, the `ag_news` dataset viewer is not working.

Am I the one who added this dataset? No


Comments:
Hi ! Thanks for reporting
We've already fixed the code that generates the preview for this dataset, we'll release the fix soon :)

Hi @lhoestq, thanks for your feedback!

Fixed in the viewer.

https://huggingface.co/datasets/ag_news
Fix temporary dataset_path creation for URIs related to remote fs
Opened by francisco-perez-sorrosal (closed)

This aims to close #3295

Comments:
Hi ! Thanks for the fix :) 

I think this should be `extract_path_from_uri` 's responsibility to strip the extra `/` from a badly formatted path like `hdfs:///absolute/path` (or raise an error). Do you think you can simply do the changes in `extract_path_from_uri` ? This way this fix will be available for all the other parts of the lib that need to extract the inner path from an URI of a remote filesystem

Then we can also keep your test cases but simply apply them to `extract_path_from_uri` instead

Hi @lhoestq! No problem! Thanks for your interest! :)

I think stripping the 3rd `/` in `hdfs:///absolute/path` inside `extract_path_from_uri` is not the solution. When I provide `hdfs:///absolute/path` to `extract_path_from_uri` we want `/absolute/path` to be returned, as it does now (at least in the case of URIs with `hdfs` schemas, for `s3` is different as it should start with a bucket name).

The problem comes in line 1041 in the original code below:

https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1038-L1042

Lets assume the following parameters for line 1041 after `extract_path_from_uri` has removed the `hdfs` schema part and the `://` from `hdfs:///absolute/path`, and `get_temporary_cache_files_directory()` returns `/tmp/a1b2b3c4`, as it is shown below: 

```python
src_dataset_path = '/absolute/path'
tmp_dir = '/tmp/a1b2b3c4'
dataset_path = Path(tmp_dir, src_dataset_path)
```

After passing those paths to the `Path` object, `dataset_path` contains only `/absolute/path`; that is, it has lost the temporary directory path. This is because, when two (or more) absolute paths are passed to the `Path` function, only the last one is taken. However, if the contents of those variables are:

```python
src_dataset_path = 'relative/path'
tmp_dir = '/tmp/a1b2b3c4'
dataset_path = Path(tmp_dir, src_dataset_path)
```

then `dataset_path` contains `/tmp/a1b2b3c4/relative/path` as expected.

Absolute paths are allowed in hdfs URIs, so that's why I added the extra function `build_local_temp_path` in the PR; so in case the second argument is an absolute path, it still will create the correct absolute path by concatenating the temp dir and the path passed by converting it to a relative path (and it also works for windows paths too.) It also allows to add the tests, checking that the main combinations are ok.

I've checked all the places where the result of `extract_path_from_uri` is used, and as far as I've seen this is the only place where it is concatenated with another possible absolute path, so no need to add `build_local_temp_path` anywhere else.
Temporary dataset_path for remote fs URIs not built properly in arrow_dataset.py::load_from_disk
Opened by francisco-perez-sorrosal (closed)

## Describe the bug
When trying to build a temporary dataset path from a remote URI in this block of code:

https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1038-L1042

the result is not the expected when passing an absolute path in an URI like `hdfs:///absolute/path`.

## Steps to reproduce the bug
```python
dataset_path = "hdfs:///absolute/path"
src_dataset_path = extract_path_from_uri(dataset_path)
tmp_dir = get_temporary_cache_files_directory()
dataset_path = Path(tmp_dir, src_dataset_path)
print(dataset_path)
```

## Expected results
With the code above, we would expect a value in `dataset_path` similar to:
`/tmp/tmpnwxyvao5/absolute/path`

## Actual results
However, we get a `dataset_path` value like:
`/absolute/path`

This is because this line here: https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1041
returns the last absolute path when two absolute paths (the one in `tmp_dir` and the one extracted from the URI in `src_dataset_path`) are passed as arguments.

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0


Comments:
Hi ! Good catch and thanks for opening a PR :)

I just responded in your PR
Add Natural Adversarial Objects dataset
Opened by osanseviero (open)

## Adding a Dataset
- **Name:** Natural Adversarial Objects (NAO)
- **Description:** Natural Adversarial Objects (NAO) is a new dataset to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause state-of-the-art detection models to misclassify with high confidence.
- **Paper:** https://arxiv.org/abs/2111.04204v1
- **Data:** https://drive.google.com/drive/folders/15P8sOWoJku6SSEiHLEts86ORfytGezi8
- **Motivation:** interesting object detection dataset useful for miscclassifications

cc @NielsRogge 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Make several audio datasets streamable
Opened by lhoestq (closed)


Needs https://github.com/huggingface/datasets/pull/3129 to be merged first


Make those audio datasets streamable:
- [x] common_voice
- [x] openslr
- [x] vivos
- [x] librispeech_asr 
(still has some issues to read FLAC)
 *actually it's ok*
- [ ] 
multilingual_librispeech (yet to be converted)</S> *TODO in a separate PR*

Comments:
Reading FLAC (for `librispeech_asr`) works OK for me (`soundfile` version: `0.10.3`):
```python
In [2]: ds = load_dataset("datasets/librispeech_asr/librispeech_asr.py", "clean", streaming=True, split="train.100")

In [3]: item = next(iter(ds))

In [4]: item.keys()
Out[4]: dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])

In [5]: item["file"]
Out[5]: '374-180298-0000.flac'

In [6]: item["audio"].keys()
Out[6]: dict_keys(['path', 'array', 'sampling_rate'])

In [7]: item["audio"]["sampling_rate"]
Out[7]: 16000

In [8]: item["audio"]["path"]
Out[8]: '374-180298-0000.flac'

In [9]: item["audio"]["array"].shape
Out[9]: (232480,)
```

Oh cool ! I think this might have come from an issue with my local `soundfile` installation then

I'll do `multilingual_librispeech` in a separate PR since it requires the data to be in another format (in particular separate the train/dev/test splits in different files)

@lhoestq @albertvillanova - think it would have been nice to have added a big message at the top stating that this is a breaking change and ping `transformers` people a bit more here.
Allow datasets with indices table when concatenating along axis=1
Opened by mariosasko (closed)

Calls `flatten_indices` on the datasets with indices table in `concatenate_datasets` to fix issues when concatenating along `axis=1`.


cc @lhoestq: I decided to flatten all the datasets instead of flattening all the datasets except the largest one in the end. The latter approach fails on the following example:
```python
a = Dataset.from_dict({"a": [10, 20, 30, 40]})
b = Dataset.from_dict({"b": [10, 20, 30, 40, 50, 60]})  # largest dataset
a = a.select([1, 2, 3])
b = b.select([1, 2, 3])
concatenate_datasets([a, b], axis=1) # fails at line concat_tables(...) because the real length of b's data is 6 and a's length is 3 after flattening (was 4 before flattening)
```

Also, it requires additional re-ordering of indices to prepare them for working with the indices table of the largest dataset. IMO not worth when we save only one `flatten_indices` call. (feel free to check the code of that approach at https://github.com/huggingface/datasets/commit/6acd10481c70950dcfdbfd2bab0bf0c74ad80bcb if you are interested)

Fixes #3273


Comments:
Add IEMOCAP dataset
Opened by osanseviero (open)

## Adding a Dataset
- **Name:** IEMOCAP
- **Description:** acted, multimodal and multispeaker database
- **Paper:** https://sail.usc.edu/iemocap/Busso_2008_iemocap.pdf
- **Data:** https://sail.usc.edu/iemocap/index.html
- **Motivation:** Useful multimodal dataset

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
The IEMOCAP dataset is private and available only on request.
```
To obtain the IEMOCAP data you just need to fill out an electronic release form below.
```

- [Request form](https://sail.usc.edu/iemocap/release_form.php)
- [License ](https://sail.usc.edu/iemocap/Data_Release_Form_IEMOCAP.pdf)


> We do not share the dataset for commercial purposes due to privacy concerns surrounding the participants of the research. The login details will only be emailed to the given academic email address.

I think it won't be possible to add this dataset to 🤗 datasets.

Hi @dnaveenr ! We can contact the authors to see if they are interested in hosting the dataset on the Hub. In the meantime, feel free to work on a script with manual download.

Hi @mariosasko . Thanks for your response. Sure, I will mail them and find out if they're open to this.

Work on a script with manual download ? This is new to me, any guidelines would be helpful here.


> Thanks for your response. Sure, I will mail them and find out if they're open to this.

It's best to leave this part to us because we have to explain how login would work and (potentially) set up a custom verification for the dataset.

> Work on a script with manual download ? This is new to me, any guidelines would be helpful here.

For instance, this is one of the scripts with manual download: https://huggingface.co/datasets/arxiv_dataset. Compared to the standard dataset, it has the `manual_download_instructions` attribute and uses `dl_manager.manual_dir` (derived from `load_dataset(..., data_dir="path/to/data")`) to access the dataset's data files.

> It's best to leave this part to us because we have to explain how login would work and (potentially) set up a custom verification for the dataset.

Yes. That would be perfect. Thanks.

----
Okay. Thanks for giving a reference. This is helpful. I will go through it.



@mariosasko  has this been solved? I would like to use login and custom verification for training on my private dataset.

@flckv I think the [gating mechanism](https://huggingface.co/docs/hub/datasets-gated) is what you are looking for. 

@mariosasko Thanks, but no. I would like to keep my HuggingFace Dataset private and train a model on it. Is this possible?
Add VoxLingua107 dataset
Opened by osanseviero (open)

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models. The dataset consists of short speech segments automatically extracted from YouTube videos and labeled according the language of the video title and description, with some post-processing steps to filter out false positives.
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** Nice audio classification dataset

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
#self-assign
Add Speech Commands dataset
Opened by osanseviero (closed)

## Adding a Dataset
- **Name:** Speech commands
- **Description:** A Dataset for Limited-Vocabulary Speech Recognition
- **Paper:** https://arxiv.org/abs/1804.03209
- **Data:** https://www.tensorflow.org/datasets/catalog/speech_commands,  Available:
http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
- **Motivation:** Nice dataset for audio classification training

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
#self-assign
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
Opened by MinionAttack (closed)

## Dataset viewer issue for '*oscar-corpus/OSCAR-2109*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/oscar-corpus/OSCAR-2109)*

*The dataset library cannot download any language from the oscar-corpus/OSCAR-2109 dataset. By entering the URL in your browser I can access the file.*

```
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

Am I the one who added this dataset ? No

Using the older version of [OSCAR](https://huggingface.co/datasets/oscar) I don't have any issues downloading languages with the dataset library.

Comments:
Hi ! Thanks for reporting :)
I think this is because the dataset is behind an access page. We can fix the dataset viewer

If you also have this error when you use the `datasets` library in python, you should probably pass `use_auth_token=True` to the `load_dataset()` function to use your account to access the dataset.

Ah ok, I didn't realise about the login page. I'll try `use_auth_token=True` and see if that solves it.

Regards!

Hi, 

Using `use_auth_token=True` and downloading the credentials with `huggingface-cli login` (stored in .huggingface/token) solved the issue.

Should I leave the issue open until you fix the Dataset viewer issue?

Cool ! Yes let's keep this issue open until the viewer is fixed - I'll close it when this is fixed. Thanks

The error I get when trying to load OSCAR 21.09 is this
```
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

The URL I get in the browser is this
```
https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/blob/main/OSCAR-2109.py
```

Maybe URL is the issue? (resolve vs blob)

> The error I get when trying to load OSCAR 21.09 is this
> 
> ```
> ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
> ```
> 
> The URL I get in the browser is this
> 
> ```
> https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/blob/main/OSCAR-2109.py
> ```
> 
> Maybe URL is the issue? (resolve vs blob)

You need to download your login credentials. See `huggingface-cli login` documentation and when loading the dataset use `use_auth_token=True`:
`
load_dataset(corpus, language, split=None, use_auth_token=True, cache_dir=cache_folder)`

Fixed.

<img width="1542" alt="Capture d’écran 2022-04-12 à 13 57 24" src="https://user-images.githubusercontent.com/1676121/162957585-af96d19c-f86c-47fe-80c4-2b071083cee4.png">
[Datasets] Improve Covost 2
Opened by patrickvonplaten (closed)

It's currently quite confusing to understand the manual data download instruction of Covost and not very user-friendly.

Currenty the user has to:

1. Go on Common Voice website
2. Find the correct dataset which is **not** mentioned in the error message
3. Download it
4. Untar it
5. Create a language id folder (why? this folder does not exist in the `.tar` downloaded file)
6. pass the folder containing the created language id folder

This PR improves this to:

1. Go on Common Voice website
2. Find the correct dataset which **is** mentioned in the error message
3. Download it
4. Untar it
5. pass the untared folder

**Note**: This PR is not at all time-critical 

Comments:
I am trying to use `load_dataset` with the French dataset(common voice corpus 1) which is downloaded from a common voice site and the target language is English (using colab)

Steps I have followed:

**1. untar:**
`!tar xvzf fr.tar -C data_dir`

**2. load data:**
`load_dataset('covost2', 'fr_en', data_dir="/content/data_dir")`

0 rows are loading as shown below:
```
Using custom data configuration fr_en-data_dir=%2Fcontent%2Fdata_dir
Reusing dataset covost2 (/root/.cache/huggingface/datasets/covost2/fr_en-data_dir=%2Fcontent%2Fdata_dir/1.0.0/bba950aae1ffa5a14b876b7e09c17b44de2c3cf60e7bd5d459640beffc78e35b)
100%
3/3 [00:00<00:00, 54.98it/s]
DatasetDict({
    train: Dataset({
        features: ['client_id', 'file', 'audio', 'sentence', 'translation', 'id'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['client_id', 'file', 'audio', 'sentence', 'translation', 'id'],
        num_rows: 0
    })
    test: Dataset({
        features: ['client_id', 'file', 'audio', 'sentence', 'translation', 'id'],
        num_rows: 0
    })
})
```

Can you please provide a sample working example code to load the dataset?

Hi ! I think it only works with the subsets of Common Voice Corpus 4, not Common Voice Corpus 1
Proposed update to the documentation for WER
Opened by wooters (closed)

I wanted to submit a minor update to the description of WER for your consideration. 

Because of the possibility of insertions, the numerator in the WER formula can be larger than N, so the value of WER can be greater than 1.0:

```
>>> from datasets import load_metric
>>> metric = load_metric("wer")
>>> metric.compute(predictions=["hello how are you"], references=["hello"])
3.0
```

and similarly from the underlying jiwer module's `wer` function:

```
>>> from jiwer import wer
>>> wer("hello", "hello how are you")
3.0
```

Comments:
f-string formatting
Opened by Mehdi2402 (closed)

**Fix #3257**

Replaced _.format()_ and _%_ by f-strings in the following modules : 
- [x] **tests**
- [x] **metrics**
- [x] **benchmarks**
- [x] **utils**
- [x] **templates**
- [x] **src/Datasets/\*.py**

Modules in **_src/Datasets/_**: 
- [x] **commands**
- [x] **features**
- [x] **formatting**
- [x] **io**
- [x] **tasks**
- [x] **utils**


Module **datasets** will not be edited as asked by @mariosasko 

-A correction of the first PR (#3267)-


Comments:
Hello @lhoestq, ```make style``` is applied as asked. :)
Force data files extraction if download_mode='force_redownload'
Opened by mariosasko (closed)

Avoids weird issues when redownloading a dataset due to cached data not being fully updated.

With this change, issues #3122 and https://github.com/huggingface/datasets/issues/2956 (not a fix, but a workaround) can be fixed as follows:
```python
dset = load_dataset(..., download_mode="force_redownload")
```

Comments:
Respect row ordering when concatenating datasets along axis=1
Opened by mariosasko (closed)

Currently, there is a bug when concatenating datasets along `axis=1` if more than one dataset has the `_indices` attribute defined. In that scenario, all indices mappings except the first one get ignored.

A minimal reproducible example:
```python
>>> from datasets import Dataset, concatenate_datasets
>>> a = Dataset.from_dict({"a": [30, 20, 10]})
>>> b = Dataset.from_dict({"b": [2, 1, 3]})
>>> d = concatenate_datasets([a.sort("a"), b.sort("b")], axis=1)
>>> print(d[:3])              # expected: {'a': [10, 20, 30], 'b': [1, 2, 3]}
{'a': [10, 20, 30], 'b': [3, 1, 2]}
```

I've noticed the bug while working on #3195.   

Comments:
Make iter_archive work with ZIP files
Opened by lhoestq (open)

Currently users can use `dl_manager.iter_archive` in their dataset script to iterate over all the files of a TAR archive.
It would be nice if it could work with ZIP files too !

Comments:
Hello, is this issue open for any contributor ? can I work on it ?



Hi ! Sure this is open for any contributor. If you're interested feel free to self-assign this issue to you by commenting `#self-assign`. Then if you have any question or if I can help, feel free to ping me.

To begin with, feel free to take a look at both implementations of `iter_archive` for local downloads and for data streaming:

In the `DownloadManager` for local dowloads:
https://github.com/huggingface/datasets/blob/dfa334bd8dc6cbc854b170379c7d2cb7e3d3fe4f/src/datasets/utils/download_manager.py#L218-L242

In the `StreamingDownloadManager` to stream the content of the archive directly from the remote file:
https://github.com/huggingface/datasets/blob/dfa334bd8dc6cbc854b170379c7d2cb7e3d3fe4f/src/datasets/utils/streaming_download_manager.py#L502-L526

Notice the call to `xopen` that opens and streams a file given either an URL or a local path :)

Okay thank you for the information. I will work on this :) 

#self-assign