jamesLeeeeeee/datasets-github-issues huggingface dataset:
Add ability to load newer, cleaner version of Multi-News 
Author: JohnGiorgi | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**

The [Multi-News dataloader points to the original version of the Multi-News dataset](https://github.com/huggingface/datasets/blob/12540dd75015678ec6019f258d811ee107439a73/datasets/multi_news/multi_news.py#L47), but this has [known errors in it](https://github.com/Alex-Fabbri/Multi-News/issues/11). There exists a [newer version which fixes some of these issues](https://drive.google.com/open?id=1jwBzXBVv8sfnFrlzPnSUBHEEAbpIUnFq).

Unfortunately I don't think you can just replace this old URL with the new one, otherwise this could lead to issues with reproducibility.

**Describe the solution you'd like**

Add a new version to the Multi-News dataloader that points to the updated dataset which has fixes for some known issues.

**Describe alternatives you've considered**

Replace the current URL to the original version to the dataset with the URL to the version with fixes.

**Additional context**

Would be happy to make a PR for this, could someone maybe point me to another dataloader that has multiple versions so I can see how this is handled in `datasets`?
Add loading variable number of columns for different splits
Author: DrMatters | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
The original dataset `blended_skill_talk` consists of different sets of columns for the different splits: (test/valid) splits have additional data column `label_candidates` that the (train) doesn't have.
When loading such data, an exception occurs at table.py:cast_table_to_schema, because of mismatched columns.
Cannot load timit_asr data set
Author: bhaddow | Pull request: no | State: closed

## Describe the bug
I am trying to load the timit_asr data set. I have tried with a copy from the LDC, and a copy from deepai. In both cases they fail with a "duplicate key" error. With the LDC version I have to convert the file extensions all to upper-case before I can load it at all.

## Steps to reproduce the bug
```python
timit = datasets.load_dataset("timit_asr", data_dir = "/path/to/dataset")
# Sample code to reproduce the bug
```

## Expected results

The data set should load without error. It worked for me before the LDC url change.

## Actual results
```
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: SA1
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.2
Update `unittest` assertions over tuples from `assertEqual` to `assertTupleEqual`
Author: alvarobartt | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**

So this is more a readability improvement rather than a proposal, wouldn't it be better to use `assertTupleEqual` over the tuples rather than `assertEqual`? As `unittest` added that function in `v3.1`, as detailed at https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual, so maybe it's worth updating.

Find an example of an `assertEqual` over a tuple in 🤗 `datasets` unit tests over an `ArrowDataset` at https://github.com/huggingface/datasets/blob/0bb47271910c8a0b628dba157988372307fca1d2/tests/test_arrow_dataset.py#L570

**Describe the solution you'd like**

Start slowly replacing all the `assertEqual` statements with `assertTupleEqual` if the assertion is done over a Python tuple, as we're doing with the Python lists using `assertListEqual` rather than `assertEqual`.

**Additional context**

If so, please let me know and I'll try to go over the tests and create a PR if applicable, otherwise, if you consider this should stay as `assertEqual` rather than `assertSequenceEqual` feel free to close this issue! Thanks 🤗
how to convert a dict generator into a huggingface dataset. 
Author: StephennFernandes | Pull request: no | State: closed

### Link

_No response_

### Description

Hey there, I have used seqio to get a well distributed mixture of samples from multiple dataset. However the resultant output from seqio is a python generator dict, which I cannot produce back into huggingface dataset.

The generator contains all the samples needed for training the model but I cannot convert it into a huggingface dataset.

The code looks like this:


```
for ex in seqio_data:
print(ex[“text”])
```

I need to convert the seqio_data (generator) into huggingface dataset.


the complete seqio code goes here:
```
import functools

import seqio
import tensorflow as tf
import t5.data
from datasets import load_dataset
from t5.data import postprocessors
from t5.data import preprocessors
from t5.evaluation import metrics
from seqio import FunctionDataSource, utils

TaskRegistry = seqio.TaskRegistry



def gen_dataset(split, shuffle=False, seed=None, column="text", dataset_params=None):
    dataset = load_dataset(**dataset_params)
    if shuffle:
        if seed:
            dataset = dataset.shuffle(seed=seed)
        else:
            dataset = dataset.shuffle()
    while True:
        for item in dataset[str(split)]:
            yield item[column]


def dataset_fn(split, shuffle_files, seed=None, dataset_params=None):
    return tf.data.Dataset.from_generator(
        functools.partial(gen_dataset, split, shuffle_files, seed, dataset_params=dataset_params),
        output_signature=tf.TensorSpec(shape=(), dtype=tf.string, name=dataset_name)
    )


@utils.map_over_dataset
def target_to_key(x, key_map, target_key):
    """Assign the value from the dataset to target_key in key_map"""
    return {**key_map, target_key: x}



dataset_name = 'oscar-corpus/OSCAR-2109'
subset= 'mr'
dataset_params = {"path": dataset_name, "language":subset, "use_auth_token":True}
dataset_shapes = None

TaskRegistry.add(
    "oscar_marathi_corpus",
    source=seqio.FunctionDataSource(
        dataset_fn=functools.partial(dataset_fn, dataset_params=dataset_params),
        splits=("train", "validation"),
        caching_permitted=False,
        num_input_examples=dataset_shapes,
    ),
preprocessors=[
functools.partial(
target_to_key, key_map={
"targets": None,
}, target_key="targets")],
    output_features={"targets": seqio.Feature(vocabulary=seqio.PassThroughVocabulary, add_eos=False, dtype=tf.string, rank=0)},
    metric_fns=[]
)

dataset = seqio.get_mixture_or_task("oscar_marathi_corpus").get_dataset(
    sequence_length=None,
    split="train",
    shuffle=True,
    num_epochs=1,
    shard_info=seqio.ShardInfo(index=0, num_shards=10),
    use_cached=False,
    seed=42
)
for _, ex in zip(range(5), dataset):
     print(ex['targets'].numpy().decode())
```


### Owner

_No response_
Update `dataset_infos.json` with new split info in `Dataset.push_to_hub` to avoid verification error
Author: mariosasko | Pull request: yes | State: closed

Update `dataset_infos.json` when pushing splits one by one via `Dataset.push_to_hub` to avoid the splits verification error. 

TODO:
~~- [ ] handle token + `{Audio, Image}.embed_storage`~~
- [x] tests
Rename DatasetBuilder config_name
Author: albertvillanova | Pull request: yes | State: closed

This PR renames the DatasetBuilder keyword argument `name` to `config_name` so that:
- it avoids confusion with the attribute `DatasetBuilder.name`, which is different
- it aligns with the Dataset property name `config_name`, defined in `DatasetInfoMixin.config_name`

Other simpler possibility could be to rename it to just `config` instead.

Please note I have only renamed this argument of DatasetBuilder because I think this refactoring has a low impact on users: we can assume this is not a public facing parameter, but private or related to the inners of our library.

It would have a  major impact to rename it also in:
- load_dataset
- load_dataset_builder: although this could also be assumed as inners...
- in our CLI commands

Besides the naming of `name`, I also find really confusing the naming of `path` in `load_dataset`. IMHO, they should have a more simpler and precise meaning (currently, they are too vague). I would propose (maybe for next major release):
```
load_dataset(dataset, config,...
```
instead of
```
load_dataset(path, name,...
```
Update `_format_columns` in `remove_columns`
Author: alvarobartt | Pull request: yes | State: closed

As explained at #4398, when calling `dataset.add_faiss_index` under certain conditions when calling a sequence of operations `cast_column`, `map`, and `remove_columns`, it fails as it's trying to look for already removed columns.

So on, after testing some possible fixes, it seems that setting the dataset format right after removing the columns seems to be working fine, so I had to add a call to `.set_format` in the `remove_columns` function.

Hope this helps!
Update: add using pcm bytes (#4323)
Author: YooSungHyun | Pull request: yes | State: closed

first of all, please look #4323

why i can not use {"path","array","sampling_rate"}
because sf.write(format="wav") and sf.read(BytesIO) is changed my pcm data value
maybe, i think wav got header but, pcm is not.
and variable naming, pcm data is "byte" type. so, "array" name is not fair i think

so, i use scipy lib and numpy (that is huggingface dependency)
and refer to @lhoestq answered,
1. encode -> using sampling_rate and pcm byte -> wav style byte (scipy.wavfile.write to byte)
2. byte converting using fairseq style pcm audio read  [FileAudioDataset](https://github.com/facebookresearch/fairseq/blob/main/fairseq/data/audio/raw_audio_dataset.py)
4. decode -> read wavfile.read

that way is not screw up my pcm byte to float data, and another audio type(wav) safety

please check!
Improve language tag for PIAF dataset
Author: lbourdois | Pull request: yes | State: closed

Hi, 

As pointed out by @lhoestq in this discussion (https://huggingface.co/datasets/asi/wikitext_fr/discussions/1), it is not yet possible to edit datasets outside of a namespace with the Hub PR feature and that you have to go through GitHub.

This modification should allow better referencing since only the xx language tags are currently taken into account and not the xx-xx.
Dataset should have a `.name` field
Author: f4hy | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
If building pipelines that can evaluate on more than one dataset, it would be nice to be able to log results of things like `Evaluating on {dataset.name}` or `results for {dataset.name} are: {results}`

Without some way of concisely identifying a dataset from the dataset object, tools which might run on more than one dataset must be passed the dataset object _and_ the name/id of the dataset being used. 

**Describe the solution you'd like**
The DatasetInfo class should have a `name` field which is the name of a dataset. then for a given dataset if it evolves in time the `version` can be updated but its different versions of the same dataset with a unique `name`. The name could then all be accessed by `dataset.name`

**Describe alternatives you've considered**
For my own purposes I am considering making `NamedDataset[Dataset]` where the subclass just has a .name field. 

**Additional context**
My guess is that most usecases are not working with more than one dataset in a given pipeline so a name is not really needed. This has surprised me though as one of the advantages of a standard dataset interface is to be able to build pipelines which can be passed in a dataset and separate responsibilities of the dataset loading from the train or eval pipeline.
Skip identical files in `push_to_hub` instead of overwriting
Author: mariosasko | Pull request: yes | State: closed

Skip identical files instead of overwriting them to save bandwidth and circumvent (user-side/server-side) errors, which can arise when working with large datasets due to long-lasting HTTP connections, by repeating calls to `push_to_hub` to resume an upload.

To be able to check if an upload can be resumed, this PR modifies the shard naming scheme from:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].parquet
```
to:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]-<SHARD_FINGERPRINT>.parquet
```
cc @LysandreJik
"NonMatchingChecksumError" when importing 'spider' dataset
Author: OmarAlaaeldein | Pull request: no | State: closed

## Describe the bug
When importing 'spider' dataset [https://huggingface.co/datasets/spider] an error occurs

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('spider')
```

## Expected results
Dataset object

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Environment info
- `datasets` version: 2.2.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5
load dataset wikitext-2-raw-v1 failed. Could not reach wikitext-2-raw-v1.py.
Author: cailun01 | Pull request: no | State: closed

## Describe the bug
Could not reach wikitext-2-raw-v1.py

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("wikitext-2-raw-v1")
```

## Expected results
Download `wikitext-2-raw-v1` dataset successfully.

## Actual results
```
  File "load_datasets.py", line 13, in <module>
    load_dataset("wikitext-2-raw-v1")
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1715, in load_dataset
    **config_kwargs,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1536, in load_dataset_builder
    data_files=data_files,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1282, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1224, in dataset_module_factory
    dynamic_modules_path=dynamic_modules_path,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 559, in get_module
    local_path = self.download_loading_script(revision)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 539, in download_loading_script
    return cached_path(file_path, download_config=download_config)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 246, in cached_path
    download_desc=download_config.download_desc,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 582, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.2.2/datasets/wikitext-2-raw-v1/wikitext-2-raw-v1.py (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Read timed out. (read timeout=100)",),))
```
I tried to download wikitext-2-raw-v1.py by chrome and got:
![image](https://user-images.githubusercontent.com/20658907/170171595-0ca9f1da-c05a-4b57-861e-9530bfa3bdb9.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: CentOS 7
- Python version: 3.6
- PyArrow version: 3.0.0
LocalDatasetModuleFactoryWithoutScript extracts invalid builder name
Author: apohllo | Pull request: no | State: closed

## Describe the bug
Trying to load a local dataset raises an error indicating that the config builder has to have a name.
No error should be reported, since the call is completly valid.

## Steps to reproduce the bug
```python
load_dataset("./data/some-dataset/", name="some-name")
```

## Expected results
The dataset should be loaded.

## Actual results
```
Traceback (most recent call last):
  File "train_lquad.py", line 19, in <module>
    load(tokenize_target_function, tokenize_target_function, {}, tokenizer)
  File "train_lquad.py", line 14, in load
    dataset = load_dataset("./data/lquad/", name="lquad")
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/load.py", line 1708, in load_dataset                                                                           
    builder_instance = load_dataset_builder(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/load.py", line 1560, in load_dataset_builder                                                                   
    builder_instance: DatasetBuilder = builder_cls(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/builder.py", line 269, in __init__                                                                             
    self.config, self.config_id = self._create_builder_config(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/builder.py", line 403, in _create_builder_config                                                               
    raise ValueError(f"BuilderConfig must have a name, got {builder_config.name}")
ValueError: BuilderConfig must have a name, got
```

## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-4.18.0-348.20.1.el8_5.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.6
- PyArrow version: 8.0.0
- Pandas version: 1.4.2

The error is probably in line 795 in load.py:

```
 builder_kwargs = {                        
     "hash": hash,
     "data_files": data_files,
     "name": os.path.basename(self.path),
     "base_path": self.path,
     **builder_kwargs,
 }
```

`os.path.basename` for a directory returns an empty string, rather than the name of the directory.
Fix dependency on dill version
Author: albertvillanova | Pull request: yes | State: closed

We had to make a hotfix by pinning dill:
- #4380

because from version 0.3.5, our custom `save_function` pickling function was raising an exception:
- #4379

This PR fixes this by implementing our custom `save_function` depending on the version of dill.

CC: @anivegesana 

This PR needs first being merged:
- [x] #4384
  - so that a circular import is fixed

It is also convenient to merge first:
- [x] #4385
trainer became extremely slow after reload dataset by `load_from_disk`
Author: conan1024hao | Pull request: no | State: open

## Describe the bug
Due to memory problem, I need to save my tokenized datasets locally by CPU and reload it by multi GPU for running training script. However, after I reload it by `load_from_disk` and start training, the speed is extremely slow. It says I need about 1500 hours with 8 A100 cards. Before this, I can run the whole script in one day with a single A100 card.
Since I am try to pre-train a BERT, **my dataset is very large(29058165 rows)**

## Steps to reproduce the bug
```python
tokenized_datasets.save_to_disk(
        "/pathto/dataset"
    )
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"] if training_args.do_train else None,
        eval_dataset=tokenized_datasets["validation"]
        if training_args.do_eval
        else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
 train_result = trainer.train(resume_from_checkpoint=checkpoint)
```

## Expected results
Without the save and reload process, I only need about one day to run the whole script with one A100 card.

## Actual results
```
[INFO|trainer.py:1290] 2022-05-23 22:49:46,266 >> ***** Running training *****
[INFO|trainer.py:1291] 2022-05-23 22:49:46,266 >>   Num examples = 29058165
[INFO|trainer.py:1292] 2022-05-23 22:49:46,266 >>   Num Epochs = 5
[INFO|trainer.py:1293] 2022-05-23 22:49:46,266 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1294] 2022-05-23 22:49:46,266 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1295] 2022-05-23 22:49:46,266 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1296] 2022-05-23 22:49:46,266 >>   Total optimization steps = 567540
  0%|          | 1/567540 [00:09<1544:49:04,  9.80s/it]
  0%|          | 2/567540 [00:17<1320:00:17,  8.37s/it]
  0%|          | 3/567540 [00:26<1393:10:17,  8.84s/it]
  0%|          | 4/567540 [00:34<1344:56:33,  8.53s/it]
  0%|          | 5/567540 [00:43<1359:36:12,  8.62s/it]
```

## Environment info
```
torch              1.11.0+cu113
torchaudio         0.11.0+cu113
torchvision        0.12.0+cu113
transformers       4.18.0
datasets           2.2.2
```
Refactor column mappings for question answering datasets
Author: lewtun | Pull request: yes | State: closed

This PR tweaks the keys in the metadata that are used to define the column mapping for question answering datasets. This is needed in order to faithfully reconstruct column names like `answers.text` and `answers.answer_start` from the keys in AutoTrain.

As observed in https://github.com/huggingface/datasets/pull/4367 we cannot use periods `.` in the keys of the YAML tags, so a decision was made to use a flat mapping with underscores. For QA datasets, however, it's handy to be able to reconstruct the nesting -- hence this PR.

cc @sashavor
Set builder name from module instead of class
Author: albertvillanova | Pull request: yes | State: closed

Now the builder name attribute is set from from the builder class name.

This PR sets the builder name attribute from the module name instead. Some motivating reasons:
- The dataset ID is relevant and unique among all datasets and this is directly related to the repository name, i.e., the name of the directory containing the dataset
- The name of the module (i.e. the file containing the loading loading script) is already relevant for loading: it must have the same name as its containing directory (related to the dataset ID), as we search for it using its directory name
- On the other hand, the name of the builder class is not relevant for loading: in our code, we just search for a class which is subclass of `DatasetBuilder` (independently of its name). We do not put any constraint on the naming of the builder class and indeed it can have a name completely different from its module/direcotry/dataset_id

IMO it makes more sense to align the caching directory name with the dataset_id/directory/module name instead of the builder class name.

Fix #4381.
device/google/accessory/adk2012 - Git at Google
Author: Aeckard45 | Pull request: no | State: closed

"git clone https://android.googlesource.com/device/google/accessory/adk2012"
 https://android.googlesource.com/device/google/accessory/adk2012/#:~:text=git%20clone%20https%3A//android.googlesource.com/device/google/accessory/adk2012