Minimartzz/github-issues huggingface dataset:
Update docs around audio and vision
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

As part of the strategy to center the docs around the different modalities, this PR updates the quickstart to include audio and vision examples. This improves the developer experience by making audio and vision content more discoverable, enabling users working in these modalities to also quickly get started without digging too deeply into the docs.

Other changes include:

- Moved the installation guide to the Get Started section because it should be part of a user's onboarding to the library before exploring tutorials or how-to's.
- Updated the native TF code at creating a `tf.data.Dataset` because it was throwing an error. The `to_tensor()` bit was redundant and removing it fixed the error (please double-check me here!).
- Added some UI components to the quickstart so it's easier for users to navigate directly to the relevant section with context about what to expect.
- Reverted to the code tabs for content that don't have any framework-specific text. I think this saves space compared to the code blocks. We'll still use the code blocks if the `torch` text is different from the `tf` text.

Let me know what you think, especially if we should include some code samples for training a model in the audio/vision sections. I left this out since we already showed it in the NLP section. I want to keep the focus on using Datasets to load and process a dataset, and not so much the training part. Maybe we can add links to the Transformers docs instead?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Let me know what you think, especially if we should include some code samples for training a model in the audio/vision sections. I left this out since we already showed it in the NLP section. I want to keep the focus on using Datasets to load and process a dataset, and not so much the training part. Maybe we can add links to the Transformers docs instead?

We plan to address this with end-to-end examples (for each modality) more focused on preprocessing than the ones in the Transformers docs.
TIMIT won't load after manual download: Errors about files that don't exist
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug

I get the message from HuggingFace that it must be downloaded manually. From the URL provided in the message, I got to UPenn page for manual download. (UPenn apparently want $250? for the dataset??)   ...So, ok, I obtained a copy from a friend and also a smaller version from Kaggle. But in both cases the HF dataloader fails; it is looking for files that don't exist anywhere in the dataset: it is looking for files with lower-case letters like "**test*" (all the filenames in both my copies are uppercase) and certain file extensions that exclude the .DOC which is provided in TIMIT:


## Steps to reproduce the bug
```python
data = load_dataset('timit_asr', 'clean')['train']
```

## Expected results
The dataset should load with no errors. 

## Actual results
This error message:
```
  File "/home/ubuntu/envs/data2vec/lib/python3.9/site-packages/datasets/data_files.py", line 201, in resolve_patterns_locally_or_by_urls
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to resolve any data file that matches '['**test*', '**eval*']' at /home/ubuntu/datasets/timit with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

But this is a strange sort of error: why is it looking for lower-case file names when all the TIMIT dataset filenames are uppercase?  Why does it exclude .DOC files when the only parts of the TIMIT data set with "TEST" in them have ".DOC" extensions?   ...I wonder, how was anyone able to get this to work in the first place?

The files in the dataset look like the following: 
```
³       PHONCODE.DOC
³       PROMPTS.TXT
³       SPKRINFO.TXT
³       SPKRSENT.TXT
³       TESTSET.DOC
```
...so why are these being excluded by the dataset loader? 


## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-1060-aws-x86_64-with-glibc2.27
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Comments:
To have some context, please see:
- #4145

Please, also note that we have recently made some fixes to the script, which are in our GitHub master branch but not yet released:
- #4422
- #4425 
- #4436

Thanks Albert! I'll try pulling `datasets` from the git repo instead of PyPI, and/or just wait for the next release.


I'm closing this issue then. Please, feel free to reopen it again if the problem persists.
Add missing columns to `blended_skill_talk`
State: closed

Adds the missing columns to `blended_skill_talk` to align the loading logic with [ParlAI](https://github.com/facebookresearch/ParlAI/blob/main/parlai/tasks/blended_skill_talk/build.py).

Fix #4426 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Load a local cached dataset that has been modified
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
I have loaded a dataset as follows:
```
d = load_dataset("emotion", split="validation")
```
Afterwards I make some modifications to the dataset via a `map` call:
```
d.map(some_update_func, cache_file_name=modified_dataset)
```
This generates a cached version of the dataset on my local system in the same directory as the original download of the data (/path/to/cache). Running an `ls` returns:
```
modified_dataset
dataset_info.json  
emotion-test.arrow  
emotion-train.arrow  
emotion-validation.arrow
```
as expected. However, when I try to load up the modified cached dataset via a call to 
```
modified = load_dataset("emotion", split="validation", data_files="/path/to/cache/modified_dataset") 
```
it simply redownloads a new version of the dataset and dumps to a new cache rather than loading up the original modified dataset:
```
Using custom data configuration validation-cdbf51685638421b
Downloading and preparing dataset emotion/validation to ...
```

How am I supposed to load the original modified local cache copy of the dataset?

## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Comments:
Hi! `datasets` caches every modification/loading, so you can either rerun the pipeline up to the `map` call or use `Dataset.from_file(modified_dataset)` to load the dataset directly from the cache file.

Awesome, hvala Mario! This works.
Fix dummy dataset generation script for handling nested types of _URLs
State: closed

It seems that when user specify nested _URLs structures in their dataset script. An error will be raised when generating dummy dataset.

I think the types of all elements in `dummy_data_dict.values()` should be checked because they may have different types.

Linked to issue #4428 

PS: I am not sure whether my code fix this issue in a proper way.

Comments:
Add personaldialog datasets
State: closed

It seems that all tests are passed

Comments:
These test errors are related to issue #4428 


_The documentation is not available anymore as the PR was closed or merged._

I only made a trivial modification in my commit https://github.com/huggingface/datasets/pull/4431/commits/402c893d35224d7828176717233909ac5f1e7b3e

I have submitted a PR #4434 for the about issue.

> Awesome thanks for adding this dataset :)
> 
> I just have one comment about the licensing.
> 
> Also it seems that you already have the dataset in https://huggingface.co/datasets/silver/personal_dialog, so it's unnecessary to add it here

Thank you very much for your comment.

So, should I close this PR?

Thanks for fixing the licensing section :)

> So, should I close this PR?

Yes you can close this PR, it's better if your dataset is under your namespace at https://huggingface.co/datasets/silver/personal_dialog :)

Don't forget to update the licensing section on https://huggingface.co/datasets/silver/personal_dialog as well
Update builder docstring for deprecated/added arguments
State: closed

This PR updates the builder docstring with deprecated/added directives for arguments name/config_name.

Follow up of:
- #4414 
- huggingface/doc-builder#233

First merge:
- #4432

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@mishig25 is investigating why deprecated/added do not affect the enclosed text format when used in args docstring: no special formatting appears: 
- https://moon-ci-docs.huggingface.co/docs/datasets/pr_4429/en/package_reference/builder_classes#datasets.DatasetBuilder

@albertvillanova please check now 👍 
https://moon-ci-docs.huggingface.co/docs/datasets/pr_4429/en/package_reference/builder_classes#datasets.DatasetBuilder

<img width="500" alt="Screenshot 2022-06-06 at 10 20 34" src="https://user-images.githubusercontent.com/11827707/172123471-fab97138-c903-4a71-ab7f-c90e5e43c58f.png">


Thanks @mishig25.

Just one question: is it expected to have the deprecated box right edge not filling all the page width (contrary to the added box)?

> Just one question: is it expected to have the deprecated box right edge not filling all the page width (contrary to the added box)?

Yes, that is expected 😊 because the depreacted box is being bounded by its parent box (the box for `name` argument in the screenshot above)
Add loading variable number of columns for different splits
State: closed
Labels: description: New feature or request
name: enhancement

**Is your feature request related to a problem? Please describe.**
The original dataset `blended_skill_talk` consists of different sets of columns for the different splits: (test/valid) splits have additional data column `label_candidates` that the (train) doesn't have.
When loading such data, an exception occurs at table.py:cast_table_to_schema, because of mismatched columns.

Comments:
Hi! Indeed the column is missing, but you shouldn't get an error? Have you made some modifications (locally) to the loading script? I've opened a PR to add the missing columns to the script.
Add new dataset MMChat
State: closed

Hi, I am adding a new dataset MMChat. 
It seems that all tests are passed

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks ! As for https://github.com/huggingface/datasets/pull/4431 please also update the licensing section in https://huggingface.co/datasets/silver/mmchat ;)

Then if it's fine for you feel free to close this PR
Cannot load timit_asr data set
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
I am trying to load the timit_asr data set. I have tried with a copy from the LDC, and a copy from deepai. In both cases they fail with a "duplicate key" error. With the LDC version I have to convert the file extensions all to upper-case before I can load it at all.

## Steps to reproduce the bug
```python
timit = datasets.load_dataset("timit_asr", data_dir = "/path/to/dataset")
# Sample code to reproduce the bug
```

## Expected results

The data set should load without error. It worked for me before the LDC url change.

## Actual results
```
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: SA1
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Comments:
Thanks for reporting, @bhaddow.

I'm fixing it.

Thanks for the quick fix!

@bhaddow we have also made a fix so that you don't have to convert to uppercase the file extensions of the LDC data.

Would you mind checking if it works OK now for you and reporting if there are any issues? Thanks. 

Hi @albertvillanova  -It loads fine on a copy of the data from deepai - although I have to remove the copies of the .WAV files (with extension .WAV,wav). On a copy of the data that was obtained from the LDC, the glob still fails to find the files. The LDC copy looks like it was copied from CD, in 2004, so the structure may be different to a current download.

Ah, if I change the train/ and test/ directories to TRAIN/ and TEST/  then it works!

Thanks for your investigation and report, @bhaddow. I'm adding another fix for the TRAIN/train and TEST/test directory names.
Update `unittest` assertions over tuples from `assertEqual` to `assertTupleEqual`
State: closed
Labels: description: New feature or request
name: enhancement

**Is your feature request related to a problem? Please describe.**

So this is more a readability improvement rather than a proposal, wouldn't it be better to use `assertTupleEqual` over the tuples rather than `assertEqual`? As `unittest` added that function in `v3.1`, as detailed at https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual, so maybe it's worth updating.

Find an example of an `assertEqual` over a tuple in 🤗 `datasets` unit tests over an `ArrowDataset` at https://github.com/huggingface/datasets/blob/0bb47271910c8a0b628dba157988372307fca1d2/tests/test_arrow_dataset.py#L570

**Describe the solution you'd like**

Start slowly replacing all the `assertEqual` statements with `assertTupleEqual` if the assertion is done over a Python tuple, as we're doing with the Python lists using `assertListEqual` rather than `assertEqual`.

**Additional context**

If so, please let me know and I'll try to go over the tests and create a PR if applicable, otherwise, if you consider this should stay as `assertEqual` rather than `assertSequenceEqual` feel free to close this issue! Thanks 🤗 


Comments:
Hi! If the only goal is to improve readability, it's better to use `assertTupleEqual` than `assertSequenceEqual` for Python tuples. Also, note that this function is called internally by `assertEqual`, but I guess we can accept a PR to be more verbose.

Hi @mariosasko, right! I'll update the issue title/desc with `assertTupleEqual` even though as you said it seems to be internally using `assertEqual` so I'm not sure whether it's worth it or not...

https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual

I thought we were supposed to move gradually from `unittest` to `pytest`...
Add LCCC dataset
State: closed

Hi, I am trying to add a new dataset lccc.
All tests are passed.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you very much for your help @albertvillanova .

I think I have fixed all the comments.

Please let me know if this PR need further modification ;)

@albertvillanova Thank you very much for your kind help.
These suggestions make the code looks more pythonic.

I have commited these changes

Hi ! The dataset seems to be a duplicate of https://huggingface.co/datasets/silver/lccc - next time no need to add it on github if it's already available on huggingface.co ;)

> Hi ! The dataset seems to be a duplicate of https://huggingface.co/datasets/silver/lccc - next time no need to add it on github if it's already available on huggingface.co ;)

OK, sorry for the inconvenience. I have closed another two PRs since these datasets are already available on huggingface.co

It's fine, thanks @silverriver for adding these datasets !
Update `dataset_infos.json` with new split info in `Dataset.push_to_hub` to avoid verification error
State: closed

Update `dataset_infos.json` when pushing splits one by one via `Dataset.push_to_hub` to avoid the splits verification error. 

TODO:
~~- [ ] handle token + `{Audio, Image}.embed_storage`~~
- [x] tests

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Rename DatasetBuilder config_name
State: closed

This PR renames the DatasetBuilder keyword argument `name` to `config_name` so that:
- it avoids confusion with the attribute `DatasetBuilder.name`, which is different
- it aligns with the Dataset property name `config_name`, defined in `DatasetInfoMixin.config_name`

Other simpler possibility could be to rename it to just `config` instead.

Please note I have only renamed this argument of DatasetBuilder because I think this refactoring has a low impact on users: we can assume this is not a public facing parameter, but private or related to the inners of our library.

It would have a  major impact to rename it also in:
- load_dataset
- load_dataset_builder: although this could also be assumed as inners...
- in our CLI commands

Besides the naming of `name`, I also find really confusing the naming of `path` in `load_dataset`. IMHO, they should have a more simpler and precise meaning (currently, they are too vague). I would propose (maybe for next major release):
```
load_dataset(dataset, config,...
```
instead of
```
load_dataset(path, name,...
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Dataset Viewer issue for ett
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/ett

### Description

Timestamp is not JSON serializable. 

```
Status code:   500
Exception:     Status500Error
Message:       Type is not JSON serializable: Timestamp
```

### Owner

No

Comments:
Thanks for reporting @dgcnz.

I have checked that the dataset works fine in streaming mode.

Additionally, other datasets containing timestamps are properly rendered by the viewer: https://huggingface.co/datasets/blbooks

I have tried to force the refresh of the preview, but the endpoint is not responsive: Connection timed out

CC: @severo 

I've just resent the refresh of the preview to the new endpoint, without success.

CC: @severo 

Fixed!

https://huggingface.co/datasets/ett/viewer/h1/test

<img width="982" alt="Capture d’écran 2022-06-15 à 09 30 22" src="https://user-images.githubusercontent.com/1676121/173769035-a075d753-ecfc-4a43-b54b-973105d464d3.png">
Dataset Viewer issue for conll2012_ontonotesv5
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/conll2012_ontonotesv5

### Description

Dataset viewer outage.

### Owner

No

Comments:
Thanks for reporting, @jiangwy99.

I guess this could be addressed only once we fix our issue with irresponsive backend endpoint.

CC: @severo 

I've just sent the forcing of the refresh of the preview to the new endpoint.

Fixed, thanks for the patience. The issue was the amount of RAM allowed to extract the first rows of the dataset was not sufficient.
Improve language tag for PIAF dataset
State: closed

Hi, 

As pointed out by @lhoestq in this discussion (https://huggingface.co/datasets/asi/wikitext_fr/discussions/1), it is not yet possible to edit datasets outside of a namespace with the Hub PR feature and that you have to go through GitHub.

This modification should allow better referencing since only the xx language tags are currently taken into account and not the xx-xx.

Comments:
Dataset should have a `.name` field
State: closed
Labels: description: New feature or request
name: enhancement

**Is your feature request related to a problem? Please describe.**
If building pipelines that can evaluate on more than one dataset, it would be nice to be able to log results of things like `Evaluating on {dataset.name}` or `results for {dataset.name} are: {results}`

Without some way of concisely identifying a dataset from the dataset object, tools which might run on more than one dataset must be passed the dataset object _and_ the name/id of the dataset being used. 

**Describe the solution you'd like**
The DatasetInfo class should have a `name` field which is the name of a dataset. then for a given dataset if it evolves in time the `version` can be updated but its different versions of the same dataset with a unique `name`. The name could then all be accessed by `dataset.name`

**Describe alternatives you've considered**
For my own purposes I am considering making `NamedDataset[Dataset]` where the subclass just has a .name field. 

**Additional context**
My guess is that most usecases are not working with more than one dataset in a given pipeline so a name is not really needed. This has surprised me though as one of the advantages of a standard dataset interface is to be able to build pipelines which can be passed in a dataset and separate responsibilities of the dataset loading from the train or eval pipeline.


Comments:
Hi! You can already use `dset.builder_name` and `dset.config_name` for that purpose. And when it comes to versioning, it's better to use `dset._fingerprint` than the `version` attribute as the former represents a deterministic hash that encodes all the mutable ops executed on a dataset, and the latter stays the same unless it's manually updated after each op.

@mariosasko  Can we make ._fingerprint not private? seems a critical component for tracking how a model was generated to ensure reproducibility.
Skip identical files in `push_to_hub` instead of overwriting
State: closed

Skip identical files instead of overwriting them to save bandwidth and circumvent (user-side/server-side) errors, which can arise when working with large datasets due to long-lasting HTTP connections, by repeating calls to `push_to_hub` to resume an upload.

To be able to check if an upload can be resumed, this PR modifies the shard naming scheme from:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].parquet
```
to:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]-<SHARD_FINGERPRINT>.parquet
```
cc @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
"NonMatchingChecksumError" when importing 'spider' dataset
State: closed
Labels: name: hosted-on-google-drive

## Describe the bug
When importing 'spider' dataset [https://huggingface.co/datasets/spider] an error occurs

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('spider')
```

## Expected results
Dataset object

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Environment info
- `datasets` version: 2.2.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments:
Thanks for reporting, @OmarAlaaeldein.

Datasets hosted at Google Drive give problems quite often due to a change in their service:
- #3786 

Related to:
- #3906

I'm having a look.

We have made a Pull Request to replace the Google Drive URL. This fix will be accessible in our next `datasets` library release.

In the meantime, once the PR merged into master, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```