jamesLeeeeeee/datasets-github-issues huggingface dataset:
Loading Data from HDF files
Author: FeryET | Pull request: no | State: open

**Is your feature request related to a problem? Please describe.**
More often than not I come along big HDF datasets, and currently there is no straight forward way to feed them to a dataset.

**Describe the solution you'd like**
I would love to see a `from_h5` method that gets an interface implemented by the user on how items are extracted from dataset (in case of multiple datasets containing elements like arrays and metadata and etc).

**Describe alternatives you've considered**
Currently I manually load hdf files using `h5py` and implement PyTorch dataset interface. For small h5 files I load them into a pandas dataframe and use `from_pandas` function in the `datasets` package to load them, but for big datasets this is not feasible.

**Additional context**
HDF files are widespread throughout different domains and are one of the go to's for many researchers/scientists/engineers who work with numerical data. Given `datasets`' usecases have outgrown NLP use cases, it will make a lot of sense focusing on things like supporting HDF files.
concatenate_datasets removes ClassLabel typing.
Author: Dref360 | Pull request: no | State: closed

## Describe the bug

When concatenating two datasets, we lose typing of ClassLabel columns.

I can work on this if this is a legitimate bug,

## Steps to reproduce the bug
```python
import datasets
from datasets import Dataset, ClassLabel, Value, concatenate_datasets

DS_LEN = 100
my_dataset = Dataset.from_dict(
    {
        "sentence": [f"{chr(i % 10)}" for i in range(DS_LEN)],
        "label": [i % 2 for i in range(DS_LEN)]
    }
)
my_predictions = Dataset.from_dict(
    {
        "pred": [(i + 1) % 2 for i in range(DS_LEN)]
    }
)

my_dataset = my_dataset.cast(datasets.Features({"sentence": Value("string"), "label": ClassLabel(2, names=["POS", "NEG"])}))
print("Original")
print(my_dataset)
print(my_dataset.features)


concat_ds = concatenate_datasets([my_dataset, my_predictions], axis=1)
print("Concatenated")
print(concat_ds)
print(concat_ds.features)


```

## Expected results
The features of `concat_ds` should contain ClassLabel.

## Actual results

On master, I get: 
```
{'sentence': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None), 'pred': Value(dtype='int64', id=None)}
```

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1
Add Google BLEU (aka GLEU) metric
Author: slowwavesleep | Pull request: yes | State: closed

This PR adds the NLTK implementation of Google BLEU metric. This is also a part of an effort to resolve an unfortunate naming collision between GLEU for machine translation and GLEU for grammatical error correction.
I used [this page](https://huggingface.co/docs/datasets/add_metric.html)  for reference. Please, point me to the right direction if I missed anything.
download_mode=`force_redownload` does not work on removed datasets
Author: severo | Pull request: no | State: open

## Describe the bug

If a cached dataset is removed from the library, I don't see how to delete it programmatically. I thought that using `force_redownload` would try to refresh the cache, then raise an exception, but it reuses the cache instead.

## Steps to reproduce the bug

_requires to already have `wit` in the cache_: see https://github.com/huggingface/datasets/pull/2981

```python
import datasets as ds
dataset = ds.load_dataset("wit", split="train", download_mode='force_redownload')
```

## Expected results

It should raise an exception, since the dataset does not exist anymore.

## Actual results

It uses the cached result

```
Using the latest cached version of the module from /home/slesage/.cache/huggingface/modules/datasets_modules/datasets/wit/107afbffd48e058b19101bddc47fbee25fa68eb6d50a733e262875f1285a5171 (last modified on Wed Sep 29 08:21:10 2021) since it couldn't be found locally at wit, or remotely on the Hugging Face Hub.
```

## Environment info

- `datasets` version: 1.13.4.dev0
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1
Replace FSTimeoutError with parent TimeoutError
Author: albertvillanova | Pull request: yes | State: closed

PR #3050 introduced a dependency on `fsspec.FSTiemoutError`. Note that this error only exists from `fsspec` version `2021.06.0` (June 2021).

To fix #3097, there are 2 alternatives:
- Either pinning `fsspec` to versions newer or equal to `2021.06.0`
- Or replacing `fsspec.FSTimeoutError` wth its parent `asyncio.TimeoutError`, which exists from Python 3.8.0 (Sep 2018).

This PR implements the second approach.

Fix #3097.
AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
Author: JTWang2000 | Pull request: no | State: closed

## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4 
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118 
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'



## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0
Push to hub capabilities for `Dataset` and `DatasetDict`
Author: LysandreJik | Pull request: yes | State: closed

This PR implements a `push_to_hub` method on `Dataset` and `DatasetDict`. This does not currently work in `IterableDatasetDict` nor `IterableDataset` as those are simple dicts and I would like your opinion on how you would like to implement this before going ahead and doing it.

This implementation needs to be used with the following `huggingface_hub` branch in order to work correctly: https://github.com/huggingface/huggingface_hub/pull/415

### Implementation

The `push_to_hub` API is entirely based on HTTP requests rather than a git-based workflow:
- This allows pushing changes without firstly cloning the repository, which reduces the time in half for the `push_to_hub` method.
- Collaboration, as well as the system of branches/merges/rebases is IMO less straightforward than for models and spaces. In the situation where such collaboration is needed, I would *heavily* advocate for the `Repository` helper of the `huggingface_hub` to be used instead of the `push_to_hub` method which will always be, by design, limiting in that regard (even if based on a git-workflow instead of HTTP requests)

In order to overcome the limit of 5GB files set by the HTTP requests, dataset sharding is used.

### Testing

The test suite implemented here makes use of the moon-staging instead of the production setup. As several repositories are created and deleted, it is better to use the staging.

It does not require setting an environment variable or any kind of special attention but introduces a new decorator `with_staging_testing` which patches global variables to use the staging endpoint instead of the production endpoint.

### Examples

The tests cover a lot of examples and behavior.
`ModuleNotFoundError: No module named 'fsspec.exceptions'`
Author: VictorSanh | Pull request: no | State: closed

## Describe the bug
I keep runnig into a fsspec ModuleNotFound error

## Steps to reproduce the bug
```python
>>> from datasets import get_dataset_infos
2021-10-15 15:25:37.863206: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-10-15 15:25:37.863252: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 56, in <module>
    from .utils.streaming_download_manager import StreamingDownloadManager
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 11, in <module>
    from fsspec.exceptions import FSTimeoutError
ModuleNotFoundError: No module named 'fsspec.exceptions'
```

Yet, I do have `fsspec`:
```bash
hf@victor-scale:~/dev/promptsource$ pip show fsspec
Name: fsspec
Version: 2021.5.0
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Location: /home/hf/dev/promptsource/.venv/lib/python3.7/site-packages
Requires: 
Required-by: datasets
```

With the same version of fsspec and `datasets==1.9.0`, I don't see this problem....

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
I can't even run `datasets-cli env` actually.., but here's my env:
- `datasets` version: 1.13.3
- Platform: Ubuntu 18.04
- Python version: 3.7.10
- PyArrow version: 3.0.0
`cast_column` makes audio decoding fail
Author: patrickvonplaten | Pull request: no | State: closed

## Describe the bug

After changing the sampling rate automatic decoding fails.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import datasets

ds = load_dataset("common_voice", "ab", split="train")

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))

print(ds[0]["audio"])  # <- this fails currently
```

yields:

```
TypeError: forward() takes 2 positional arguments but 4 were given
```

## Expected results
no failure

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 1.13.2 (master)
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0
Fix JNLBA dataset
Author: bhavitvyamalik | Pull request: yes | State: closed

As mentioned in #3089, I've added more tags and also updated the link for dataset which was earlier using a Google Drive link. 
I'm having problem with generating dummy data as `datasets-cli dummy_data  ./datasets/jnlpba --auto_generate --match_text_files "*.iob2"` is giving `datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
` error. I'll try to add dummy data manually.
`blog_authorship_corpus` is broken
Author: fdtomasi | Pull request: no | State: closed

## Describe the bug
The dataset `blog_authorship_corpus` is broken.
By bypassing the checksum checks, the loading does not return any error but the resulting dataset is empty.
I suspect it is because the data download url is broken (http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip).

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("blog_authorship_corpus", split="train", download_mode='force_redownload')
```

## Expected results
No error.

## Actual results
```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
/tmp/ipykernel_5237/1729238701.py in <module>
      2 ds = load_dataset(
      3     "blog_authorship_corpus", split="train",
----> 4     download_mode='force_redownload'
      5 )

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
   1115         ignore_verifications=ignore_verifications,
   1116         try_from_hf_gcs=try_from_hf_gcs,
-> 1117         use_auth_token=use_auth_token,
   1118     )
   1119 

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    635                     if not downloaded_from_gcs:
    636                         self._download_and_prepare(
--> 637                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    638                         )
    639                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    707         if verify_infos:
    708             verify_checksums(
--> 709                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    710             )
    711 

/opt/conda/lib/python3.7/site-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip']
```

## Environment info
- `datasets` version: 1.13.2
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.10
- PyArrow version: 5.0.0
JNLPBA Dataset
Author: sciarrilli | Pull request: no | State: closed

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
The dataset loading script for this dataset is incorrect. This is a biomedical dataset used for named entity recognition. The entities in the [script](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L81-L83) are: O, B, and I.  The correct entities from the original data file are: 

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

## Actual results
The dataset loader script needs to include the following NER names:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

And the [data](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L46) that is being pulled has been modified from the original dataset and does not include the original NER tags.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
VisibleDeprecationWarning when using `set_format("numpy")`
Author: Rocketknight1 | Pull request: no | State: closed

Code to reproduce:

```
from datasets import load_dataset
dataset = load_dataset("glue", "mnli")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased')

def tokenize_function(dataset):
    return tokenizer(dataset['premise'])

tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=dataset['train'].features)

tokenized_datasets.set_format("numpy")

tokenized_datasets['train'][5:8]
```
Outputs:

```
python3.9/site-packages/datasets/formatting/formatting.py:167: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return np.array(array, copy=False, **self.np_array_kwargs)
```
Datasets with Audio feature raise error when loaded from cache due to _resampler parameter
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
As reported by @patrickvonplaten, when loaded from the cache, datasets containing the Audio feature raise TypeError.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

## Actual results
```
TypeError: __init__() got an unexpected keyword argument '_resampler'
```
[Audio datasets] Adapting all audio datasets
Author: patrickvonplaten | Pull request: yes | State: closed

This PR adds the new `Audio(...)` features - see: https://github.com/huggingface/datasets/pull/2324 to the most important audio datasets:

- Librispeech
- Timit
- Common Voice
- AMI
- ... (others I'm forgetting now)

The PR is curently blocked because the following leads to a problem:

```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

As soon as it's unblocked, I'll adapt the other audio datasets as well.
Error related to timeout keyword argument
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
As reported by @patrickvonplaten, a TypeError is raised when trying to load a dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

## Actual results
```
TypeError: dataset_info() got an unexpected keyword argument 'timeout'
```
Error when loading a metric
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
As reported by @sgugger, after last release, exception is thrown when loading a metric.

## Steps to reproduce the bug
```python
from datasets import load_metric

metric = load_metric("squad_v2")
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-e612a8cab787> in <module>
      1 from datasets import load_metric
----> 2 metric = load_metric("squad_v2")

d:\projects\huggingface\datasets\src\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, script_version, **metric_init_kwargs)
   1336         )
   1337         revision = script_version
-> 1338     metric_module = metric_module_factory(
   1339         path, revision=revision, download_config=download_config, download_mode=download_mode
   1340     ).module_path

d:\projects\huggingface\datasets\src\datasets\load.py in metric_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, **download_kwargs)
   1237                 if not isinstance(e1, FileNotFoundError):
   1238                     raise e1 from None
-> 1239                 raise FileNotFoundError(
   1240                     f"Couldn't find a metric script at {relative_to_absolute_path(combined_path)}. "
   1241                     f"Metric '{path}' doesn't exist on the Hugging Face Hub either."

FileNotFoundError: Couldn't find a metric script at D:\projects\huggingface\datasets\squad_v2\squad_v2.py. Metric 'squad_v2' doesn't exist on the Hugging Face Hub either.
```
Updates LexGLUE and MultiEURLEX README.md files
Author: iliaschalkidis | Pull request: yes | State: closed

Updates LexGLUE and MultiEURLEX README.md files

- Fix leaderboard in LexGLUE.
- Fix an error in the CaseHOLD data example.
- Turn MultiEURLEX dataset statistics table into HTML to nicely render in HF website.
add XCSR dataset
Author: yangxqiao | Pull request: yes | State: closed

Hi,

I wanted to add the [XCSR ](https://inklab.usc.edu//XCSR/xcsr_datasets) dataset to huggingface! :) 

I followed the instructions of adding new dataset to huggingface and have all the required files ready now! It would be super helpful if you can take a look and review them. Thanks in advance for your time and help. Look forward to hearing from you and can't wait to add XCSR to huggingface :D
Fix pathlib patches for streaming
Author: lhoestq | Pull request: yes | State: closed

Fix issue https://github.com/huggingface/datasets/issues/2866 (for good this time)

`counter` now works in both streaming and non-streaming mode.
And the `AttributeError: 'str' object has no attribute 'as_posix'` related to the patch of Path.open is fixed as well

Note : the patches should only affect the datasets module, not the user's ones ! That's why we should probably use something else than patch.object to patch the Path class' methods.

cc @severo @albertvillanova
Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder
Author: zixiliuUSC | Pull request: no | State: closed

## Adding a Dataset
- **Name:** text, json, csv
- **Description:** I am developing a customized dataset loading script. The problem is  mainly about my custom dataset is seperate into many files and I only find a dataset loading template in [https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py](https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py) that can handle my circumstance. I'm afraid these templates are too old to use. Could you re-add these three templates to current master branch?
Add iter_archive
Author: lhoestq | Pull request: yes | State: closed

Added the `iter_archive` method for the StreamingDownloadManager.

It was already implemented in the regular DownloadManager.
Now it can be used to stream from TAR archives as mentioned in https://github.com/huggingface/datasets/issues/2829

I also updated the `food101` dataset as an example.
Any image/audio dataset using TAR archives can be updated to use `iter_archive` in order to be streamable :)

cc @severo 

Fix #2829.