SurAyush/github_issues huggingface dataset:
Issue: Update `unittest` assertions over tuples from `assertEqual` to `assertTupleEqual`
State: closed
Author: alvarobartt

**Is your feature request related to a problem? Please describe.**

So this is more a readability improvement rather than a proposal, wouldn't it be better to use `assertTupleEqual` over the tuples rather than `assertEqual`? As `unittest` added that function in `v3.1`, as detailed at https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual, so maybe it's worth updating.

Find an example of an `assertEqual` over a tuple in 🤗 `datasets` unit tests over an `ArrowDataset` at https://github.com/huggingface/datasets/blob/0bb47271910c8a0b628dba157988372307fca1d2/tests/test_arrow_dataset.py#L570

**Describe the solution you'd like**

Start slowly replacing all the `assertEqual` statements with `assertTupleEqual` if the assertion is done over a Python tuple, as we're doing with the Python lists using `assertListEqual` rather than `assertEqual`.

**Additional context**

If so, please let me know and I'll try to go over the tests and create a PR if applicable, otherwise, if you consider this should stay as `assertEqual` rather than `assertSequenceEqual` feel free to close this issue! Thanks 🤗 


Comments:
Hi! If the only goal is to improve readability, it's better to use `assertTupleEqual` than `assertSequenceEqual` for Python tuples. Also, note that this function is called internally by `assertEqual`, but I guess we can accept a PR to be more verbose.

Hi @mariosasko, right! I'll update the issue title/desc with `assertTupleEqual` even though as you said it seems to be internally using `assertEqual` so I'm not sure whether it's worth it or not...

https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual

I thought we were supposed to move gradually from `unittest` to `pytest`...
Issue: Add LCCC dataset
State: closed
Author: silverriver

Hi, I am trying to add a new dataset lccc.
All tests are passed.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you very much for your help @albertvillanova .

I think I have fixed all the comments.

Please let me know if this PR need further modification ;)

@albertvillanova Thank you very much for your kind help.
These suggestions make the code looks more pythonic.

I have commited these changes

Hi ! The dataset seems to be a duplicate of https://huggingface.co/datasets/silver/lccc - next time no need to add it on github if it's already available on huggingface.co ;)

> Hi ! The dataset seems to be a duplicate of https://huggingface.co/datasets/silver/lccc - next time no need to add it on github if it's already available on huggingface.co ;)

OK, sorry for the inconvenience. I have closed another two PRs since these datasets are already available on huggingface.co

It's fine, thanks @silverriver for adding these datasets !
Issue: Update `dataset_infos.json` with new split info in `Dataset.push_to_hub` to avoid verification error
State: closed
Author: mariosasko

Update `dataset_infos.json` when pushing splits one by one via `Dataset.push_to_hub` to avoid the splits verification error. 

TODO:
~~- [ ] handle token + `{Audio, Image}.embed_storage`~~
- [x] tests

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Rename DatasetBuilder config_name
State: closed
Author: albertvillanova

This PR renames the DatasetBuilder keyword argument `name` to `config_name` so that:
- it avoids confusion with the attribute `DatasetBuilder.name`, which is different
- it aligns with the Dataset property name `config_name`, defined in `DatasetInfoMixin.config_name`

Other simpler possibility could be to rename it to just `config` instead.

Please note I have only renamed this argument of DatasetBuilder because I think this refactoring has a low impact on users: we can assume this is not a public facing parameter, but private or related to the inners of our library.

It would have a  major impact to rename it also in:
- load_dataset
- load_dataset_builder: although this could also be assumed as inners...
- in our CLI commands

Besides the naming of `name`, I also find really confusing the naming of `path` in `load_dataset`. IMHO, they should have a more simpler and precise meaning (currently, they are too vague). I would propose (maybe for next major release):
```
load_dataset(dataset, config,...
```
instead of
```
load_dataset(path, name,...
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Dataset Viewer issue for ett
State: closed
Author: dgcnz

### Link

https://huggingface.co/datasets/ett

### Description

Timestamp is not JSON serializable. 

```
Status code:   500
Exception:     Status500Error
Message:       Type is not JSON serializable: Timestamp
```

### Owner

No

Comments:
Thanks for reporting @dgcnz.

I have checked that the dataset works fine in streaming mode.

Additionally, other datasets containing timestamps are properly rendered by the viewer: https://huggingface.co/datasets/blbooks

I have tried to force the refresh of the preview, but the endpoint is not responsive: Connection timed out

CC: @severo 

I've just resent the refresh of the preview to the new endpoint, without success.

CC: @severo 

Fixed!

https://huggingface.co/datasets/ett/viewer/h1/test

<img width="982" alt="Capture d’écran 2022-06-15 à 09 30 22" src="https://user-images.githubusercontent.com/1676121/173769035-a075d753-ecfc-4a43-b54b-973105d464d3.png">
Issue: Dataset Viewer issue for conll2012_ontonotesv5
State: closed
Author: jiangwangyi

### Link

https://huggingface.co/datasets/conll2012_ontonotesv5

### Description

Dataset viewer outage.

### Owner

No

Comments:
Thanks for reporting, @jiangwy99.

I guess this could be addressed only once we fix our issue with irresponsive backend endpoint.

CC: @severo 

I've just sent the forcing of the refresh of the preview to the new endpoint.

Fixed, thanks for the patience. The issue was the amount of RAM allowed to extract the first rows of the dataset was not sufficient.
Issue: Improve language tag for PIAF dataset
State: closed
Author: lbourdois

Hi, 

As pointed out by @lhoestq in this discussion (https://huggingface.co/datasets/asi/wikitext_fr/discussions/1), it is not yet possible to edit datasets outside of a namespace with the Hub PR feature and that you have to go through GitHub.

This modification should allow better referencing since only the xx language tags are currently taken into account and not the xx-xx.

Comments:
Issue: Dataset should have a `.name` field
State: closed
Author: f4hy

**Is your feature request related to a problem? Please describe.**
If building pipelines that can evaluate on more than one dataset, it would be nice to be able to log results of things like `Evaluating on {dataset.name}` or `results for {dataset.name} are: {results}`

Without some way of concisely identifying a dataset from the dataset object, tools which might run on more than one dataset must be passed the dataset object _and_ the name/id of the dataset being used. 

**Describe the solution you'd like**
The DatasetInfo class should have a `name` field which is the name of a dataset. then for a given dataset if it evolves in time the `version` can be updated but its different versions of the same dataset with a unique `name`. The name could then all be accessed by `dataset.name`

**Describe alternatives you've considered**
For my own purposes I am considering making `NamedDataset[Dataset]` where the subclass just has a .name field. 

**Additional context**
My guess is that most usecases are not working with more than one dataset in a given pipeline so a name is not really needed. This has surprised me though as one of the advantages of a standard dataset interface is to be able to build pipelines which can be passed in a dataset and separate responsibilities of the dataset loading from the train or eval pipeline.


Comments:
Hi! You can already use `dset.builder_name` and `dset.config_name` for that purpose. And when it comes to versioning, it's better to use `dset._fingerprint` than the `version` attribute as the former represents a deterministic hash that encodes all the mutable ops executed on a dataset, and the latter stays the same unless it's manually updated after each op.

@mariosasko  Can we make ._fingerprint not private? seems a critical component for tracking how a model was generated to ensure reproducibility.
Issue: Skip identical files in `push_to_hub` instead of overwriting
State: closed
Author: mariosasko

Skip identical files instead of overwriting them to save bandwidth and circumvent (user-side/server-side) errors, which can arise when working with large datasets due to long-lasting HTTP connections, by repeating calls to `push_to_hub` to resume an upload.

To be able to check if an upload can be resumed, this PR modifies the shard naming scheme from:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].parquet
```
to:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]-<SHARD_FINGERPRINT>.parquet
```
cc @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: "NonMatchingChecksumError" when importing 'spider' dataset
State: closed
Author: OmarAlaaeldein

## Describe the bug
When importing 'spider' dataset [https://huggingface.co/datasets/spider] an error occurs

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('spider')
```

## Expected results
Dataset object

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Environment info
- `datasets` version: 2.2.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments:
Thanks for reporting, @OmarAlaaeldein.

Datasets hosted at Google Drive give problems quite often due to a change in their service:
- #3786 

Related to:
- #3906

I'm having a look.

We have made a Pull Request to replace the Google Drive URL. This fix will be accessible in our next `datasets` library release.

In the meantime, once the PR merged into master, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Issue: load dataset wikitext-2-raw-v1 failed. Could not reach wikitext-2-raw-v1.py.
State: closed
Author: cailun01

## Describe the bug
Could not reach wikitext-2-raw-v1.py

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("wikitext-2-raw-v1")
```

## Expected results
Download `wikitext-2-raw-v1` dataset successfully.

## Actual results
```
  File "load_datasets.py", line 13, in <module>
    load_dataset("wikitext-2-raw-v1")
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1715, in load_dataset
    **config_kwargs,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1536, in load_dataset_builder
    data_files=data_files,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1282, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1224, in dataset_module_factory
    dynamic_modules_path=dynamic_modules_path,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 559, in get_module
    local_path = self.download_loading_script(revision)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 539, in download_loading_script
    return cached_path(file_path, download_config=download_config)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 246, in cached_path
    download_desc=download_config.download_desc,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 582, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.2.2/datasets/wikitext-2-raw-v1/wikitext-2-raw-v1.py (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Read timed out. (read timeout=100)",),))
```
I tried to download wikitext-2-raw-v1.py by chrome and got:
![image](https://user-images.githubusercontent.com/20658907/170171595-0ca9f1da-c05a-4b57-861e-9530bfa3bdb9.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: CentOS 7
- Python version: 3.6
- PyArrow version: 3.0.0


Comments:
I tried in this way.

```python
from datasets import load_dataset
dataset = load_dataset(path="wikitext", name="wikitext-103-v1", split="train")
```
Issue: Fix dependency on dill version
State: closed
Author: albertvillanova

We had to make a hotfix by pinning dill:
- #4380

because from version 0.3.5, our custom `save_function` pickling function was raising an exception:
- #4379

This PR fixes this by implementing our custom `save_function` depending on the version of dill.

CC: @anivegesana 

This PR needs first being merged:
- [x] #4384
  - so that a circular import is fixed

It is also convenient to merge first:
- [x] #4385

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Add Pascal VOC dataset
State: closed
Author: nateraw

This PR adds the Pascal VOC dataset in the same way TFDS has it added. I believe we can iterate on this dataset and in future versions include more data, such as segmentation masks, but for now I think it is a good idea to just add it the same way as TFDS to get a solid first version out there.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Some CI fails are unrelated to your PR and fixed on master, feel free to merge master into your branch :)

Thanks @nateraw for the addition of this dataset.

I would suggest to transfer it to the Hugging Face Hub, under a "pascal" organization namespace: "pascal/voc".

What do you think?

FYI I think this dataset is also available at (internal) https://huggingface.co/datasets/HuggingFaceM4/pascal_voc

@lhoestq @albertvillanova what do you think best path forward is? No idea when I'll get to looking at this again, but would be nice to know plan so when I find time I can just get it done in one sitting. 

My (not strong) opinion on this:
- as we are removing dataset scripts from GitHub, this dataset should be created directly on the Hub
- I proposed doing it under some kind of "official" org namespace, like pascal or pascal2; other suggestions are welcome
- the link given by @lhoestq might serve as inspiration for your implementation (I think yours misses data about action classification): their implementation comprises tasks: classification/detection, segmentation, action classification, person layout; it misses other tasks though

What do you think?
Issue: Update CI deprecated legacy image
State: closed
Author: albertvillanova

Now our CI still uses a deprecated legacy image:
> You’re using a [deprecated Docker convenience image.](https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034) Upgrade to a next-gen Docker convenience image.

This PR updates to next-generation convenience image.

Related to:
- #2955

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Refactor column mappings for question answering datasets
State: closed
Author: lewtun

This PR tweaks the keys in the metadata that are used to define the column mapping for question answering datasets. This is needed in order to faithfully reconstruct column names like `answers.text` and `answers.answer_start` from the keys in AutoTrain.

As observed in https://github.com/huggingface/datasets/pull/4367 we cannot use periods `.` in the keys of the YAML tags, so a decision was made to use a flat mapping with underscores. For QA datasets, however, it's handy to be able to reconstruct the nesting -- hence this PR.

cc @sashavor 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks.
> 
> I have no visibility about this, but if you say it is more useful for AutoTrain this way...

Thanks for the review @albertvillanova ! Yes, I need some way to reconstruct the original column names with a period because that's how they appear after we flatten the nested columns. In any case, we can adjust this later if needed :)

Does that mean that we need to change the metadata?

> Does that mean that we need to change the metadata?

Yes, but this PR takes care of it :)

Oh good! thanks for the heads up!
Issue: Fix metadata validation
State: closed
Author: albertvillanova

Since Python 3.8, the typing module:
- raises an AttributeError when trying to access `__args__` on any type, e.g.: `List.__args__`
- provides the `get_args` function instead: `get_args(List)`

This PR implements a fix for Python >=3.8 whereas maintaining backward compatibility.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Set builder name from module instead of class
State: closed
Author: albertvillanova

Now the builder name attribute is set from from the builder class name.

This PR sets the builder name attribute from the module name instead. Some motivating reasons:
- The dataset ID is relevant and unique among all datasets and this is directly related to the repository name, i.e., the name of the directory containing the dataset
- The name of the module (i.e. the file containing the loading loading script) is already relevant for loading: it must have the same name as its containing directory (related to the dataset ID), as we search for it using its directory name
- On the other hand, the name of the builder class is not relevant for loading: in our code, we just search for a class which is subclass of `DatasetBuilder` (independently of its name). We do not put any constraint on the naming of the builder class and indeed it can have a name completely different from its module/direcotry/dataset_id

IMO it makes more sense to align the caching directory name with the dataset_id/directory/module name instead of the builder class name.

Fix #4381.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: device/google/accessory/adk2012 - Git at Google
State: closed
Author: Aeckard45

"git clone https://android.googlesource.com/device/google/accessory/adk2012"
 https://android.googlesource.com/device/google/accessory/adk2012/#:~:text=git%20clone%20https%3A//android.googlesource.com/device/google/accessory/adk2012

Comments:
Issue: Test dill
State: closed
Author: albertvillanova

Regression test for future releases of `dill`.

Related to #4379. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I should point out that the hash will be the same if computed twice with the same code on the same version of dill (after adding huggingface's code that removes line numbers and file names, and sorts globals.) My changes in dill 0.3.5 and ones that I will make in 0.3.6 will result in different pickles than the ones dill 0.3.4 was making. This should still be fine for caching.

Just some comments @lhoestq:

The best practice for testing is to have a `test_<filename>.py` for each `<filename>.py`. Therefore in order to have the filenames aligned, I would propose:
- either renaming `fingerprint.py` to `caching.py`
- or renaming `test_caching.py` to `test_fingerprint.py`

On the other hand, my idea when implementing this test was not to test all the functionalities of the `Hasher`, but just to have a regression test that fails if dill version is > 0.3.4 and the pin in our `setup.py` is not present. Just recall that we had no failing test in our CI when the issue with dill was found on `transformers`.

The objective of this PR is just to have a regression test for that case: I tested and I got `AttributeError: module 'dill._dill' has no attribute 'stack'`

For this regression test, I took into account this comment by @gugarosa: https://github.com/huggingface/datasets/issues/4379#issuecomment-1133131825

There is no equivalent test in `test_caching.py` because our CI did not fail before pinning dill.

Ok I see, renaming it to `test_fingerprint.py` sounds like a good idea :)
Issue: L
State: closed
Author: AronCodes21

## Describe the   L
L
## Expected   L
A clear and concise lmll
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:

Comments:
Issue: First time trying
State: closed
Author: Aeckard45

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Issue: Fix checksum and bug in irc_disentangle dataset
State: closed
Author: albertvillanova

There was a bug in filepath segment: 
- wrong: `jkkummerfeld-irc-disentanglement-fd379e9`
- right: `jkkummerfeld-irc-disentanglement-35f0a40`

Also there was a bug in the checksum of the downloaded file.

This PR fixes these issues.

Fix partially #4376.


Comments:
_The documentation is not available anymore as the PR was closed or merged._