helmo/github-issues huggingface dataset:
RuntimeError: Sharding is ambiguous for this dataset
State: closed | Author: albertvillanova (MEMBER)

### Describe the bug

When loading some datasets, a RuntimeError is raised.

For example, for "ami" dataset: https://huggingface.co/datasets/ami/discussions/3
```
.../huggingface/datasets/src/datasets/builder.py in _prepare_split(self, split_generator, check_duplicate_keys, file_format, num_proc, max_shard_size)
   1415         fpath = path_join(self._output_dir, fname)
   1416 
-> 1417         num_input_shards = _number_of_shards_in_gen_kwargs(split_generator.gen_kwargs)
   1418         if num_input_shards <= 1 and num_proc is not None:
   1419             logger.warning(

.../huggingface/datasets/src/datasets/utils/sharding.py in _number_of_shards_in_gen_kwargs(gen_kwargs)
     10     lists_lengths = {key: len(value) for key, value in gen_kwargs.items() if isinstance(value, list)}
     11     if len(set(lists_lengths.values())) > 1:
---> 12         raise RuntimeError(
     13             (
     14                 "Sharding is ambiguous for this dataset: "

RuntimeError: Sharding is ambiguous for this dataset: we found several data sources lists of different lengths, and we don't know over which list we should parallelize:
    - key samples_paths has length 6
    - key ids has length 7
    - key verification_ids has length 6
To fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, and use tuples otherwise. In the end there should only be one single list, or several lists with the same length.
```

This behavior was introduced when implementing multiprocessing by PR:
- #5107

### Steps to reproduce the bug

```python
ds = load_dataset("ami", "microphone-single", split="train", revision="2d7620bb7c3f1aab9f329615c3bdb598069d907a")
```

### Expected behavior

No error raised.

### Environment info

Since datasets 2.7.0
concatenate_datasets fails when two dataset with shards > 1 and unequal shard numbers
State: closed | Author: ZeguanXiao (NONE)

### Describe the bug

When using `concatenate_datasets([dataset1, dataset2], axis = 1)` to concatenate two datasets with shards > 1, it fails:

```
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/combine.py", line 182, in concatenate_datasets
    return _concatenate_map_style_datasets(dsets, info=info, split=split, axis=axis)
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 5499, in _concatenate_map_style_datasets
    table = concat_tables([dset._data for dset in dsets], axis=axis)
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/table.py", line 1778, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/table.py", line 1483, in from_tables
    blocks = _extend_blocks(blocks, table_blocks, axis=axis)
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/table.py", line 1477, in _extend_blocks
    result[i].extend(row_blocks)
IndexError: list index out of range
```

### Steps to reproduce the bug

dataset = concatenate_datasets([dataset1, dataset2], axis = 1)

### Expected behavior

The datasets are correctly concatenated.

### Environment info

datasets==2.8.0

Comments: Hi ! Thanks for reporting :)

I managed to reproduce the hub using
```python

from datasets import concatenate_datasets, Dataset, load_from_disk

Dataset.from_dict({"a": range(9)}).save_to_disk("tmp/ds1")
ds1 = load_from_disk("tmp/ds1")
ds1 = concatenate_datasets([ds1, ds1])

Dataset.from_dict({"b": range(6)}).save_to_disk("tmp/ds2")
ds2 = load_from_disk("tmp/ds2")
ds2 = concatenate_datasets([ds2, ds2, ds2])

concatenate_datasets([ds1, ds2], axis=1)
```
and I get
```python
Traceback (most recent call last):                                                                                                    
  File "test.py", line 98, in <module>
    dds = concatenate_datasets([ds1, ds2], axis=1)
  File "/Users/.../datasets/combine.py", line 182, in concatenate_datasets
    return _concatenate_map_style_datasets(dsets, info=info, split=split, axis=axis)
  File "/Users/.../datasets/arrow_dataset.py", line 5499, in _concatenate_map_style_datasets
    table = concat_tables([dset._data for dset in dsets], axis=axis)
  File "/Users/.../datasets/table.py", line 1778, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/Users/.../datasets/table.py", line 1483, in from_tables
    blocks = _extend_blocks(blocks, table_blocks, axis=axis)
  File "/Users/.../datasets/table.py", line 1477, in _extend_blocks
    result[i].extend(row_blocks)
IndexError: list index out of range
```

It appears to happen when the two datasets have a number of shards that is not the same
Datasets.from_sql() generates deprecation warning
State: closed | Author: msummerfield (NONE)

### Describe the bug

Calling `Datasets.from_sql()` generates a warning:

`.../site-packages/datasets/builder.py:712: FutureWarning: 'use_auth_token' was deprecated in version 2.7.1 and will be removed in 3.0.0. Pass 'use_auth_token' to the initializer/'load_dataset_builder' instead.`

### Steps to reproduce the bug

Any valid call to `Datasets.from_sql()` will produce the deprecation warning.

### Expected behavior

No warning.

The fix should be simply to remove the parameter `use_auth_token` from the call to `builder.download_and_prepare()` at line 43 of `io/sql.py` (it is set to `None` anyway, and is not needed).

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-4.15.0-169-generic-x86_64-with-glibc2.27
- Python version: 3.9.15
- PyArrow version: 10.0.1
- Pandas version: 1.5.2


Comments: Thanks for reporting @msummerfield. We are fixing it.
size_in_bytes the same for all splits
State: open | Author: Breakend (NONE)

### Describe the bug

Hi, it looks like whenever you pull a dataset and get size_in_bytes, it returns the same size for all splits (and that size is the combined size of all splits). It seems like this shouldn't be the intended behavior since it is misleading. Here's an example:

```
>>> from datasets import load_dataset
>>> x = load_dataset("glue", "wnli")
Found cached dataset glue (/Users/breakend/.cache/huggingface/datasets/glue/wnli/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1097.70it/s]
>>> x["train"].size_in_bytes
186159
>>> x["validation"].size_in_bytes
186159
>>> x["test"].size_in_bytes
186159
>>> 
```

### Steps to reproduce the bug

```
>>> from datasets import load_dataset
>>> x = load_dataset("glue", "wnli")
>>> x["train"].size_in_bytes
186159
>>> x["validation"].size_in_bytes
186159
>>> x["test"].size_in_bytes
186159
```

### Expected behavior

The expected behavior is that it should return the separate sizes for all splits.

### Environment info

- `datasets` version: 2.7.1
- Platform: macOS-12.5-arm64-arm-64bit
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.2

Comments: Hi @Breakend,

Indeed, the attribute `size_in_bytes` refers to the size of the entire dataset configuration, for all splits (size of downloaded files + Arrow files), not the specific split.
This is also the case for `download_size` (downloaded files) and `dataset_size` (Arrow files).

The size of the Arrow files for a specific split can be accessed: e.g. size of the "test" split only
```python
ds["train"].info.splits["test"].num_bytes
```

I agree this is confusing and maybe we should improve it.
Better integration of BIG-bench
State: open | Author: albertvillanova (MEMBER)
Labels: name: enhancement

### Feature request

Ideally, it would be nice to have a maintained PyPI package for `bigbench`.

### Motivation

We'd like to allow anyone to access, explore and use any task.

### Your contribution

@lhoestq has opened an issue in their repo:
- https://github.com/google/BIG-bench/issues/906

Comments: Hi, I made my version : https://huggingface.co/datasets/tasksource/bigbench
Support Dataset conversion from/to Spark
State: open | Author: albertvillanova (MEMBER)

This PR implements Spark integration by supporting `Dataset` conversion from/to Spark `DataFrame`.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5401). All of your documentation changes will be reflected on that endpoint.

Cool thanks !

Spark DataFrame are usually quite big, and I believe here `from_spark` would load everything in the driver node's RAM, which is quite limiting. Same for `to_spark` which would load everything in the driver node's RAM before sending the data to the executor. Maybe we can mention this in the docstring ?

To transfer big datasets from/into the HF ecosystem using Spark maybe we can just make sure that `pyspark` can read/write to the HF Hub, and that `datasets` can read from HDFS/S3/etc.

Yes @lhoestq , consider this as a first integration of the Datasets library with Spark.
- This PR implements the basic conversion between both.
  - And yes, we are using the Spark's `pandas` API (that uses `pyarrow` under the hood): everything is transferred to the driver.
  - Note that we are converting from/to a Datasets dataset: this is not distributed

The next step is to support the integration of the HF Hub with Spark, that I think should be done using `hffs`.

Thinking more about it I don't really see how those two methods help in practice, since one can already do `datasets` <-> pandas <-> spark and those two methods don't add value over this.

However I think it can be good documentation to explain that it's possible to do it and it's super simple
Fix checksum verification
State: closed | Author: daskol (CONTRIBUTOR)

Expected checksum was verified against checksum dict (not checksum).

Comments: Hi ! If I'm not mistaken both `expected_checksums[url]` and `recorded_checksums[url]` are dictionaries with keys "checksum" and "num_bytes". So we need to check whether `expected_checksums[url] != recorded_checksums[url]` (or simply `expected_checksums[url]["checksum"] != recorded_checksums[url]["checksum"]`)

But in your fix you're checking `expected_checksums[url] != recorded_checksums[url]['checksum']`.

So I think it's fine to keep this as is

No, the issue is that there is comparison of sclar value and dictionary.

Acording to [`DatasetInfo`][1], we need specify a dictionary which maps a URL to a checksum as follows.

```python
CHECKSUMS = {
    URL: 'a5dc6bf63ea088ade6e98594bfa386f45211c38b2a3db3dd11b33bd530f3c481',
}

class FancyDataset:
    def _info(self):
        return DatasetInfo(..., download_checksums=CHECKSUMS)
```

However, `load_dataset` fails with this checksum definition.

[1]: https://github.com/huggingface/datasets/blob/main/src/datasets/info.py#L124-L125

I think it has to be formatted like this right now. Maybe the DatasetInfo doc is unclear and we can improve it
```python
CHECKSUMS = {
    URL: {"checksum": checksum, "num_bytes": num_bytes},
}
```

Right. I am not sure that this is a correct way to do it. People usually calculate sha256, md5, or whatever else but not size in bytes. Also, people use only some of checksum algorithms. This means that comparing dictionaries in `verify_checksums` is too strict (requires equality of all items) and raises compatibility issues in the future. Another issue is that a comparison of dictionaries assumes type constraints which imply type equality. 

Since almost noone uses checksums as far as I known, my PR suggests a minimal change to mitigate these issues except support of a specific checksum algorithm which is a separated feature and should be contributed in a separate PRs from my perspective.

Applying this change will break the verification code, since the `expected_checksums` is a dict with those two keys.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5396). All of your documentation changes will be reflected on that endpoint.
CI error: TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers'
State: closed | Author: albertvillanova (MEMBER)

### Describe the bug

While installing the dependencies, the CI raises a TypeError:
```
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/spacy/__init__.py", line 6, in <module>
    from .errors import setup_default_warnings
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/spacy/errors.py", line 2, in <module>
    from .compat import Literal
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/spacy/compat.py", line 3, in <module>
    from thinc.util import copy_array
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/thinc/__init__.py", line 5, in <module>
    from .config import registry
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/thinc/config.py", line 2, in <module>
    import confection
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/confection/__init__.py", line 10, in <module>
    from pydantic import BaseModel, create_model, ValidationError, Extra
  File "pydantic/__init__.py", line 2, in init pydantic.__init__
  File "pydantic/dataclasses.py", line 46, in init pydantic.dataclasses
    # | None    | Attribute is set to None.               |
  File "pydantic/main.py", line 121, in init pydantic.main
TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers'
```

See: https://github.com/huggingface/datasets/actions/runs/3793736481/jobs/6466356565

### Steps to reproduce the bug

```shell
pip install .[tests,metrics-tests]
python -m spacy download en_core_web_sm
```

### Expected behavior

No error.

### Environment info

See: https://github.com/huggingface/datasets/actions/runs/3793736481/jobs/6466356565

Comments: I still getting the same error :

`python -m spacy download fr_core_news_lg
`.
`import spacy`

@MFatnassi, this issue and the corresponding fix only affect our Continuous Integration testing environment.

Note that `datasets` does not depend on `spacy`.
Getting Value Error while loading a dataset..
State: closed | Author: valmetisrinivas (NONE)

### Describe the bug

I am trying to load a dataset using Hugging Face Datasets load_dataset method. I am getting the value error as show below. Can someone help with this? I am using Windows laptop and Google Colab notebook.

```
WARNING:datasets.builder:Using custom data configuration default-a1d9e8eaedd958cd
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-12-5b4fdcb8e6d5>](https://localhost:8080/#) in <module>
      6 )
      7 
----> 8 next(iter(law_dataset_streamed))

17 frames
[/usr/local/lib/python3.8/dist-packages/fsspec/core.py](https://localhost:8080/#) in get_compression(urlpath, compression)
    485         compression = infer_compression(urlpath)
    486     if compression is not None and compression not in compr:
--> 487         raise ValueError("Compression type %s not supported" % compression)
    488     return compression
    489 

ValueError: Compression type zstd not supported
```

### Steps to reproduce the bug

```
!pip install zstandard
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
```

### Expected behavior

I expect an iterable object as the output 'lds' to be created.

### Environment info

Windows laptop with Google Colab notebook

Comments: Hi! I can't reproduce this error locally (Mac) or in Colab. What version of `datasets` are you using?

Hi [mariosasko](https://github.com/mariosasko), the datasets version is '2.8.0'.

@valmetisrinivas you get that error because you imported `datasets` (and thus `fsspec`) before installing `zstandard`.

Please, restart your Colab runtime and execute the install commands before importing `datasets`:
```python
!pip install datasets
!pip install zstandard

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(ds))
```

> @valmetisrinivas you get that error because you imported `datasets` (and thus `fsspec`) before installing `zstandard`.
> 
> Please, restart your Colab runtime and execute the install commands before importing `datasets`:
> 
> ```python
> !pip install datasets
> !pip install zstandard
> 
> from datasets import load_dataset
> 
> ds = load_dataset(
>     "json",
>     data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
>     split="train",
>     streaming=True,
> )
> next(iter(ds))
> ```

I guess that was the problem, importing datasets before the installation of zstandard. Thank you for the feedback.
Missing documentation page : improve-performance
State: closed | Author: astariul (NONE)

### Describe the bug

Trying to access https://huggingface.co/docs/datasets/v2.8.0/en/package_reference/cache#improve-performance, the page is missing.

The link is in here : https://huggingface.co/docs/datasets/v2.8.0/en/package_reference/loading_methods#datasets.load_dataset.keep_in_memory

### Steps to reproduce the bug

Access the page and see it's missing.

### Expected behavior

Not missing page

### Environment info

Doesn't matter

Comments: Hi! Our documentation builder does not support links to sections, hence the bug. This is the link it should point to https://huggingface.co/docs/datasets/v2.8.0/en/cache#improve-performance.
Is `fs=` deprecated in `load_from_disk()` as well?
State: closed | Author: dconathan (CONTRIBUTOR)

### Describe the bug

The `fs=` argument was deprecated from `Dataset.save_to_disk` and `Dataset.load_from_disk` in favor of automagically figuring it out via fsspec:
https://github.com/huggingface/datasets/blob/9a7272cd4222383a5b932b0083a4cc173fda44e8/src/datasets/arrow_dataset.py#L1339-L1340

Is there a reason the same thing shouldn't also apply to `datasets.load.load_from_disk()` as well ?

https://github.com/huggingface/datasets/blob/9a7272cd4222383a5b932b0083a4cc173fda44e8/src/datasets/load.py#L1779



### Steps to reproduce the bug

n/a

### Expected behavior

n/a

### Environment info

n/a

Comments: Hi! Yes, we should deprecate the `fs` param here. Would you be interested in submitting a PR? 

> Hi! Yes, we should deprecate the `fs` param here. Would you be interested in submitting a PR?

Yeah I can do that sometime next week. Should the storage_options be a new arg here?  I’ll look around for anywhere else where fs is an arg.

Closed by #5393.
IterableDataset missing column_names, differs from Dataset interface
State: closed | Author: iceboundflame (NONE)
Labels: name: enhancement

name: good first issue

### Describe the bug

The documentation on [Stream](https://huggingface.co/docs/datasets/v1.18.2/stream.html) seems to imply that IterableDataset behaves just like a Dataset. However, examples like
```
dataset.map(augment_data, batched=True, remove_columns=dataset.column_names, ...)
```
will not work because `.column_names` does not exist on IterableDataset. I cannot find any clear explanation on why this is not available, is it an oversight? We do have `iterable_ds.features` available.

### Steps to reproduce the bug

See above

### Expected behavior

Dataset and IterableDataset would be expected to have the same interface, with any differences noted in the documentation.

### Environment info

n/a

Comments: Another example is that `IterableDataset.map` does not have `fn_kwargs`, among other arguments. It makes it harder to convert code from Dataset to IterableDataset.

Hi! `fn_kwargs` was added to `IterableDataset.map` in `datasets 2.5.0`, so please update your installation (`pip install -U datasets`) to use it.

Regarding `column_names`, I agree we should add this property to `IterableDataset`. In the meantime, you can use `list(dataset.features.keys())` instead.

Thanks! That's great news.

On Thu, Dec 22, 2022, 07:48 Mario Šaško ***@***.***> wrote:

> Hi! fn_kwargs was added to IterableDataset.map in datasets 2.5.0, so
> please update your installation (pip install -U datasets) to use it.
>
> Regarding column_names, I agree we should add this property to
> IterableDataset. In the meantime, you can use
> list(dataset.features.keys()) instead.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5383#issuecomment-1362993633>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHD6N2EQUFEOUFDW3VHSILWORZ45ANCNFSM6AAAAAATGKWVGM>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


I'm marking this issue as a "good first issue", as it makes sense to have `IterableDataset.column_names` in the API. Besides the case when `features` are `None` (e.g., `features` are `None` after `map`), in which we can also return `column_names` as `None`, adding this property should be straightforward,

Hi @mariosasko, I can work on this if that's ok?

Yes! I've assigned you the issue.