helmo/github-issues huggingface dataset:
Load a local cached dataset that has been modified
State: closed | Author: mihail911 (NONE)
Labels: name: bug

## Describe the bug
I have loaded a dataset as follows:
```
d = load_dataset("emotion", split="validation")
```
Afterwards I make some modifications to the dataset via a `map` call:
```
d.map(some_update_func, cache_file_name=modified_dataset)
```
This generates a cached version of the dataset on my local system in the same directory as the original download of the data (/path/to/cache). Running an `ls` returns:
```
modified_dataset
dataset_info.json  
emotion-test.arrow  
emotion-train.arrow  
emotion-validation.arrow
```
as expected. However, when I try to load up the modified cached dataset via a call to 
```
modified = load_dataset("emotion", split="validation", data_files="/path/to/cache/modified_dataset") 
```
it simply redownloads a new version of the dataset and dumps to a new cache rather than loading up the original modified dataset:
```
Using custom data configuration validation-cdbf51685638421b
Downloading and preparing dataset emotion/validation to ...
```

How am I supposed to load the original modified local cache copy of the dataset?

## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Comments: Hi! `datasets` caches every modification/loading, so you can either rerun the pipeline up to the `map` call or use `Dataset.from_file(modified_dataset)` to load the dataset directly from the cache file.

Awesome, hvala Mario! This works.
Fix dummy dataset generation script for handling nested types of _URLs
State: closed | Author: silverriver (CONTRIBUTOR)

It seems that when user specify nested _URLs structures in their dataset script. An error will be raised when generating dummy dataset.

I think the types of all elements in `dummy_data_dict.values()` should be checked because they may have different types.

Linked to issue #4428 

PS: I am not sure whether my code fix this issue in a proper way.
Add personaldialog datasets
State: closed | Author: silverriver (CONTRIBUTOR)

It seems that all tests are passed

Comments: These test errors are related to issue #4428 


_The documentation is not available anymore as the PR was closed or merged._

I only made a trivial modification in my commit https://github.com/huggingface/datasets/pull/4431/commits/402c893d35224d7828176717233909ac5f1e7b3e

I have submitted a PR #4434 for the about issue.

> Awesome thanks for adding this dataset :)
> 
> I just have one comment about the licensing.
> 
> Also it seems that you already have the dataset in https://huggingface.co/datasets/silver/personal_dialog, so it's unnecessary to add it here

Thank you very much for your comment.

So, should I close this PR?

Thanks for fixing the licensing section :)

> So, should I close this PR?

Yes you can close this PR, it's better if your dataset is under your namespace at https://huggingface.co/datasets/silver/personal_dialog :)

Don't forget to update the licensing section on https://huggingface.co/datasets/silver/personal_dialog as well
Update builder docstring for deprecated/added arguments
State: closed | Author: albertvillanova (MEMBER)

This PR updates the builder docstring with deprecated/added directives for arguments name/config_name.

Follow up of:
- #4414 
- huggingface/doc-builder#233

First merge:
- #4432

Comments: _The documentation is not available anymore as the PR was closed or merged._

@mishig25 is investigating why deprecated/added do not affect the enclosed text format when used in args docstring: no special formatting appears: 
- https://moon-ci-docs.huggingface.co/docs/datasets/pr_4429/en/package_reference/builder_classes#datasets.DatasetBuilder

@albertvillanova please check now 👍 
https://moon-ci-docs.huggingface.co/docs/datasets/pr_4429/en/package_reference/builder_classes#datasets.DatasetBuilder

<img width="500" alt="Screenshot 2022-06-06 at 10 20 34" src="https://user-images.githubusercontent.com/11827707/172123471-fab97138-c903-4a71-ab7f-c90e5e43c58f.png">


Thanks @mishig25.

Just one question: is it expected to have the deprecated box right edge not filling all the page width (contrary to the added box)?

> Just one question: is it expected to have the deprecated box right edge not filling all the page width (contrary to the added box)?

Yes, that is expected 😊 because the depreacted box is being bounded by its parent box (the box for `name` argument in the screenshot above)
Add loading variable number of columns for different splits
State: closed | Author: DrMatters (NONE)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
The original dataset `blended_skill_talk` consists of different sets of columns for the different splits: (test/valid) splits have additional data column `label_candidates` that the (train) doesn't have.
When loading such data, an exception occurs at table.py:cast_table_to_schema, because of mismatched columns.

Comments: Hi! Indeed the column is missing, but you shouldn't get an error? Have you made some modifications (locally) to the loading script? I've opened a PR to add the missing columns to the script.
Add new dataset MMChat
State: closed | Author: silverriver (CONTRIBUTOR)

Hi, I am adding a new dataset MMChat. 
It seems that all tests are passed

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thanks ! As for https://github.com/huggingface/datasets/pull/4431 please also update the licensing section in https://huggingface.co/datasets/silver/mmchat ;)

Then if it's fine for you feel free to close this PR
Cannot load timit_asr data set
State: closed | Author: bhaddow (NONE)
Labels: name: bug

## Describe the bug
I am trying to load the timit_asr data set. I have tried with a copy from the LDC, and a copy from deepai. In both cases they fail with a "duplicate key" error. With the LDC version I have to convert the file extensions all to upper-case before I can load it at all.

## Steps to reproduce the bug
```python
timit = datasets.load_dataset("timit_asr", data_dir = "/path/to/dataset")
# Sample code to reproduce the bug
```

## Expected results

The data set should load without error. It worked for me before the LDC url change.

## Actual results
```
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: SA1
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Comments: Thanks for reporting, @bhaddow.

I'm fixing it.

Thanks for the quick fix!

@bhaddow we have also made a fix so that you don't have to convert to uppercase the file extensions of the LDC data.

Would you mind checking if it works OK now for you and reporting if there are any issues? Thanks. 

Hi @albertvillanova  -It loads fine on a copy of the data from deepai - although I have to remove the copies of the .WAV files (with extension .WAV,wav). On a copy of the data that was obtained from the LDC, the glob still fails to find the files. The LDC copy looks like it was copied from CD, in 2004, so the structure may be different to a current download.

Ah, if I change the train/ and test/ directories to TRAIN/ and TEST/  then it works!

Thanks for your investigation and report, @bhaddow. I'm adding another fix for the TRAIN/train and TEST/test directory names.
Update `unittest` assertions over tuples from `assertEqual` to `assertTupleEqual`
State: closed | Author: alvarobartt (MEMBER)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**

So this is more a readability improvement rather than a proposal, wouldn't it be better to use `assertTupleEqual` over the tuples rather than `assertEqual`? As `unittest` added that function in `v3.1`, as detailed at https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual, so maybe it's worth updating.

Find an example of an `assertEqual` over a tuple in 🤗 `datasets` unit tests over an `ArrowDataset` at https://github.com/huggingface/datasets/blob/0bb47271910c8a0b628dba157988372307fca1d2/tests/test_arrow_dataset.py#L570

**Describe the solution you'd like**

Start slowly replacing all the `assertEqual` statements with `assertTupleEqual` if the assertion is done over a Python tuple, as we're doing with the Python lists using `assertListEqual` rather than `assertEqual`.

**Additional context**

If so, please let me know and I'll try to go over the tests and create a PR if applicable, otherwise, if you consider this should stay as `assertEqual` rather than `assertSequenceEqual` feel free to close this issue! Thanks 🤗 


Comments: Hi! If the only goal is to improve readability, it's better to use `assertTupleEqual` than `assertSequenceEqual` for Python tuples. Also, note that this function is called internally by `assertEqual`, but I guess we can accept a PR to be more verbose.

Hi @mariosasko, right! I'll update the issue title/desc with `assertTupleEqual` even though as you said it seems to be internally using `assertEqual` so I'm not sure whether it's worth it or not...

https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual

I thought we were supposed to move gradually from `unittest` to `pytest`...
Add LCCC dataset
State: closed | Author: silverriver (CONTRIBUTOR)

Hi, I am trying to add a new dataset lccc.
All tests are passed.

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thank you very much for your help @albertvillanova .

I think I have fixed all the comments.

Please let me know if this PR need further modification ;)

@albertvillanova Thank you very much for your kind help.
These suggestions make the code looks more pythonic.

I have commited these changes

Hi ! The dataset seems to be a duplicate of https://huggingface.co/datasets/silver/lccc - next time no need to add it on github if it's already available on huggingface.co ;)

> Hi ! The dataset seems to be a duplicate of https://huggingface.co/datasets/silver/lccc - next time no need to add it on github if it's already available on huggingface.co ;)

OK, sorry for the inconvenience. I have closed another two PRs since these datasets are already available on huggingface.co

It's fine, thanks @silverriver for adding these datasets !
Update `dataset_infos.json` with new split info in `Dataset.push_to_hub` to avoid verification error
State: closed | Author: mariosasko (COLLABORATOR)

Update `dataset_infos.json` when pushing splits one by one via `Dataset.push_to_hub` to avoid the splits verification error. 

TODO:
~~- [ ] handle token + `{Audio, Image}.embed_storage`~~
- [x] tests

Comments: _The documentation is not available anymore as the PR was closed or merged._
Rename DatasetBuilder config_name
State: closed | Author: albertvillanova (MEMBER)

This PR renames the DatasetBuilder keyword argument `name` to `config_name` so that:
- it avoids confusion with the attribute `DatasetBuilder.name`, which is different
- it aligns with the Dataset property name `config_name`, defined in `DatasetInfoMixin.config_name`

Other simpler possibility could be to rename it to just `config` instead.

Please note I have only renamed this argument of DatasetBuilder because I think this refactoring has a low impact on users: we can assume this is not a public facing parameter, but private or related to the inners of our library.

It would have a  major impact to rename it also in:
- load_dataset
- load_dataset_builder: although this could also be assumed as inners...
- in our CLI commands

Besides the naming of `name`, I also find really confusing the naming of `path` in `load_dataset`. IMHO, they should have a more simpler and precise meaning (currently, they are too vague). I would propose (maybe for next major release):
```
load_dataset(dataset, config,...
```
instead of
```
load_dataset(path, name,...
```

Comments: _The documentation is not available anymore as the PR was closed or merged._
Dataset Viewer issue for ett
State: closed | Author: dgcnz (NONE)
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/ett

### Description

Timestamp is not JSON serializable. 

```
Status code:   500
Exception:     Status500Error
Message:       Type is not JSON serializable: Timestamp
```

### Owner

No

Comments: Thanks for reporting @dgcnz.

I have checked that the dataset works fine in streaming mode.

Additionally, other datasets containing timestamps are properly rendered by the viewer: https://huggingface.co/datasets/blbooks

I have tried to force the refresh of the preview, but the endpoint is not responsive: Connection timed out

CC: @severo 

I've just resent the refresh of the preview to the new endpoint, without success.

CC: @severo 

Fixed!

https://huggingface.co/datasets/ett/viewer/h1/test

<img width="982" alt="Capture d’écran 2022-06-15 à 09 30 22" src="https://user-images.githubusercontent.com/1676121/173769035-a075d753-ecfc-4a43-b54b-973105d464d3.png">
Dataset Viewer issue for conll2012_ontonotesv5
State: closed | Author: jiangwangyi (NONE)
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/conll2012_ontonotesv5

### Description

Dataset viewer outage.

### Owner

No

Comments: Thanks for reporting, @jiangwy99.

I guess this could be addressed only once we fix our issue with irresponsive backend endpoint.

CC: @severo 

I've just sent the forcing of the refresh of the preview to the new endpoint.

Fixed, thanks for the patience. The issue was the amount of RAM allowed to extract the first rows of the dataset was not sufficient.
Improve language tag for PIAF dataset
State: closed | Author: lbourdois (NONE)

Hi, 

As pointed out by @lhoestq in this discussion (https://huggingface.co/datasets/asi/wikitext_fr/discussions/1), it is not yet possible to edit datasets outside of a namespace with the Hub PR feature and that you have to go through GitHub.

This modification should allow better referencing since only the xx language tags are currently taken into account and not the xx-xx.
Dataset should have a `.name` field
State: closed | Author: f4hy (NONE)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
If building pipelines that can evaluate on more than one dataset, it would be nice to be able to log results of things like `Evaluating on {dataset.name}` or `results for {dataset.name} are: {results}`

Without some way of concisely identifying a dataset from the dataset object, tools which might run on more than one dataset must be passed the dataset object _and_ the name/id of the dataset being used. 

**Describe the solution you'd like**
The DatasetInfo class should have a `name` field which is the name of a dataset. then for a given dataset if it evolves in time the `version` can be updated but its different versions of the same dataset with a unique `name`. The name could then all be accessed by `dataset.name`

**Describe alternatives you've considered**
For my own purposes I am considering making `NamedDataset[Dataset]` where the subclass just has a .name field. 

**Additional context**
My guess is that most usecases are not working with more than one dataset in a given pipeline so a name is not really needed. This has surprised me though as one of the advantages of a standard dataset interface is to be able to build pipelines which can be passed in a dataset and separate responsibilities of the dataset loading from the train or eval pipeline.


Comments: Hi! You can already use `dset.builder_name` and `dset.config_name` for that purpose. And when it comes to versioning, it's better to use `dset._fingerprint` than the `version` attribute as the former represents a deterministic hash that encodes all the mutable ops executed on a dataset, and the latter stays the same unless it's manually updated after each op.

@mariosasko  Can we make ._fingerprint not private? seems a critical component for tracking how a model was generated to ensure reproducibility.
Skip identical files in `push_to_hub` instead of overwriting
State: closed | Author: mariosasko (COLLABORATOR)

Skip identical files instead of overwriting them to save bandwidth and circumvent (user-side/server-side) errors, which can arise when working with large datasets due to long-lasting HTTP connections, by repeating calls to `push_to_hub` to resume an upload.

To be able to check if an upload can be resumed, this PR modifies the shard naming scheme from:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].parquet
```
to:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]-<SHARD_FINGERPRINT>.parquet
```
cc @LysandreJik 

Comments: _The documentation is not available anymore as the PR was closed or merged._
"NonMatchingChecksumError" when importing 'spider' dataset
State: closed | Author: OmarAlaaeldein (NONE)
Labels: name: hosted-on-google-drive

## Describe the bug
When importing 'spider' dataset [https://huggingface.co/datasets/spider] an error occurs

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('spider')
```

## Expected results
Dataset object

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Environment info
- `datasets` version: 2.2.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments: Thanks for reporting, @OmarAlaaeldein.

Datasets hosted at Google Drive give problems quite often due to a change in their service:
- #3786 

Related to:
- #3906

I'm having a look.

We have made a Pull Request to replace the Google Drive URL. This fix will be accessible in our next `datasets` library release.

In the meantime, once the PR merged into master, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
load dataset wikitext-2-raw-v1 failed. Could not reach wikitext-2-raw-v1.py.
State: closed | Author: cailun01 (NONE)
Labels: name: bug

## Describe the bug
Could not reach wikitext-2-raw-v1.py

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("wikitext-2-raw-v1")
```

## Expected results
Download `wikitext-2-raw-v1` dataset successfully.

## Actual results
```
  File "load_datasets.py", line 13, in <module>
    load_dataset("wikitext-2-raw-v1")
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1715, in load_dataset
    **config_kwargs,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1536, in load_dataset_builder
    data_files=data_files,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1282, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1224, in dataset_module_factory
    dynamic_modules_path=dynamic_modules_path,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 559, in get_module
    local_path = self.download_loading_script(revision)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 539, in download_loading_script
    return cached_path(file_path, download_config=download_config)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 246, in cached_path
    download_desc=download_config.download_desc,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 582, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.2.2/datasets/wikitext-2-raw-v1/wikitext-2-raw-v1.py (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Read timed out. (read timeout=100)",),))
```
I tried to download wikitext-2-raw-v1.py by chrome and got:
![image](https://user-images.githubusercontent.com/20658907/170171595-0ca9f1da-c05a-4b57-861e-9530bfa3bdb9.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: CentOS 7
- Python version: 3.6
- PyArrow version: 3.0.0


Comments: I tried in this way.

```python
from datasets import load_dataset
dataset = load_dataset(path="wikitext", name="wikitext-103-v1", split="train")
```
Fix dependency on dill version
State: closed | Author: albertvillanova (MEMBER)

We had to make a hotfix by pinning dill:
- #4380

because from version 0.3.5, our custom `save_function` pickling function was raising an exception:
- #4379

This PR fixes this by implementing our custom `save_function` depending on the version of dill.

CC: @anivegesana 

This PR needs first being merged:
- [x] #4384
  - so that a circular import is fixed

It is also convenient to merge first:
- [x] #4385

Comments: _The documentation is not available anymore as the PR was closed or merged._
Add Pascal VOC dataset
State: closed | Author: nateraw (CONTRIBUTOR)
Labels: name: dataset contribution

This PR adds the Pascal VOC dataset in the same way TFDS has it added. I believe we can iterate on this dataset and in future versions include more data, such as segmentation masks, but for now I think it is a good idea to just add it the same way as TFDS to get a solid first version out there.

Comments: _The documentation is not available anymore as the PR was closed or merged._

Some CI fails are unrelated to your PR and fixed on master, feel free to merge master into your branch :)

Thanks @nateraw for the addition of this dataset.

I would suggest to transfer it to the Hugging Face Hub, under a "pascal" organization namespace: "pascal/voc".

What do you think?

FYI I think this dataset is also available at (internal) https://huggingface.co/datasets/HuggingFaceM4/pascal_voc

@lhoestq @albertvillanova what do you think best path forward is? No idea when I'll get to looking at this again, but would be nice to know plan so when I find time I can just get it done in one sitting. 

My (not strong) opinion on this:
- as we are removing dataset scripts from GitHub, this dataset should be created directly on the Hub
- I proposed doing it under some kind of "official" org namespace, like pascal or pascal2; other suggestions are welcome
- the link given by @lhoestq might serve as inspiration for your implementation (I think yours misses data about action classification): their implementation comprises tasks: classification/detection, segmentation, action classification, person layout; it misses other tasks though

What do you think?
Update CI deprecated legacy image
State: closed | Author: albertvillanova (MEMBER)

Now our CI still uses a deprecated legacy image:
> You’re using a [deprecated Docker convenience image.](https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034) Upgrade to a next-gen Docker convenience image.

This PR updates to next-generation convenience image.

Related to:
- #2955

Comments: _The documentation is not available anymore as the PR was closed or merged._