MarouaneSanhaji/github-issues huggingface dataset:
Column project operation on `datasets.Dataset`
State: closed

### Feature request

There is no operation to select a subset of columns of original dataset. Expected API follows.
```python
a = Dataset.from_dict({
    'int': [0, 1, 2]
    'char': ['a', 'b', 'c'],
    'none': [None] * 3,
})

b = a.project('int', 'char')  # usually, .select()

print(a.column_names)  # stdout: ['int', 'char', 'none']
print(b.column_names)  # stdout: ['int', 'char']
```
Method project can easily accept not only column names (as a `str)` but univariant function applied to corresponding column as an example. Or keyword arguments can be used in order to rename columns in advance (see `pandas`, `pyspark`, `pyarrow`, and SQL)..

### Motivation

Projection is a typical operation in every data processing library. And it is a basic block of a well-known data manipulation language like SQL. Without this operation `datasets.Dataset` interface is not complete.

### Your contribution

Not sure. Some of my PRs are still open and some do not have any discussions.

Hi ! This would be a nice addition indeed :) This sounds like a duplicate of https://github.com/huggingface/datasets/issues/5468

> Not sure. Some of my PRs are still open and some do not have any discussions.

Sorry to hear that, feel free to ping me on those PRs
Allow opposite of remove_columns on Dataset and DatasetDict
State: closed

### Feature request

In this blog post https://huggingface.co/blog/audio-datasets, I noticed the following code:

```python
COLUMNS_TO_KEEP = ["text", "audio"]
all_columns = gigaspeech["train"].column_names
columns_to_remove = set(all_columns) - set(COLUMNS_TO_KEEP)

gigaspeech = gigaspeech.remove_columns(columns_to_remove)
```

This kind of thing happens a lot when you don't need to keep all columns from the dataset. It would be more convenient (and less error prone) if you could just write:

```python
gigaspeech = gigaspeech.keep_columns(["text", "audio"])
```

Internally, `keep_columns` could still call `remove_columns`, but it expresses more clearly what the user's intent is.

### Motivation

Less code to write for the user of the dataset.

### Your contribution

-

Hi! I agree it would be nice to have a method like that. Instead of `keep_columns`, we can name it `select_columns` to be more aligned with PyArrow's naming convention (`pa.Table.select`).

Hi, I am a newbie to open source and would like to contribute. @mariosasko can I take up this issue ?

Hey, I also want to work on this issue I am a newbie to open source. 

This sounds related to https://github.com/huggingface/datasets/issues/5474

I'm fine with `select_columns`, or we could also override `select` to also accept a list of columns maybe ?

@lhoestq, I am planning to add a member function to the dataset class to perform the selection operation. Do you think its the right way to proceed? or there is a better option ?

Unless @mariosasko thinks otherwise, I think it can go in `Dataset.select()` :)
Though some parameters like keep_in_memory, indices_cache_file_name or writer_batch_size wouldn't when selecting columns, so we would need to update the docstring as well

If someone wants to give it a shot, feel free to comment `#self-assign` and it will assign the issue to you.

Feel free to ping us here if you have questions or if we can help :)

I would rather have this functionality as a separate method. IMO it's always better to be explicit than to have an API where a single method can do different/uncorrelated things (somewhat reminds me of Pandas, and there is probably a good reason why PyArrow is more rigid in this aspect).

In the end I also think it would be nice to have it as a separate method, this way we can also have it for `IterableDataset` (which can't have `select` for indices)
NonMatchingChecksumError for hendrycks_test
State: closed

### Describe the bug

The checksum of the file has likely changed on the remote host.


### Steps to reproduce the bug

`dataset = nlp.load_dataset("hendrycks_test", "anatomy")`


### Expected behavior

no error thrown


### Environment info

- `datasets` version: 2.2.1
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1

Thanks for reporting, @sarahwie.

Please note this issue was already fixed in `datasets` 2.6.0 version:
- #5040

If you update your `datasets` version, you will be able to load the dataset:
```
pip install -U datasets
```

Oops, missed that I needed to upgrade. Thanks!
slice split while streaming
State: closed

### Describe the bug

When using the `load_dataset` function with streaming set to True, slicing splits is apparently not supported.

Did I miss this in the documentation?

### Steps to reproduce the bug

`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train[:3]")`

causes ValueError: Bad split: train[:3]. Available splits: ['train', 'test'] in builder.py, line 1213, in as_streaming_dataset

### Expected behavior

The first 3 entries of the dataset as a stream

### Environment info

- `datasets` version: 2.8.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.9
- PyArrow version: 10.0.1
- Pandas version: 1.5.2


Hi! Yes, that's correct. When `streaming` is `True`, only split names can be specified as `split`, and for slicing, you have to use `.skip`/`.take` instead.

E.g. 
`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train[:3]")`

rewritten with `.skip`/`.take`:
`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train").take(3)`




Thank you for your quick response!
ImageFolder BadZipFile: Bad offset for central directory
State: closed

### Describe the bug

I'm getting the following exception:
```
lib/python3.10/zipfile.py:1353 in _RealGetContents        │
│                                                                                                  │
│   1350 │   │   # self.start_dir:  Position of start of central directory                         │
│   1351 │   │   self.start_dir = offset_cd + concat                                               │
│   1352 │   │   if self.start_dir < 0:                                                            │
│ ❱ 1353 │   │   │   raise BadZipFile("Bad offset for central directory")                          │
│   1354 │   │   fp.seek(self.start_dir, 0)                                                        │
│   1355 │   │   data = fp.read(size_cd)                                                           │
│   1356 │   │   fp = io.BytesIO(data)                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
BadZipFile: Bad offset for central directory                                                                            
Extracting data files:  35%|█████████████████▊                                 | 38572/110812 [00:10<00:20, 3576.26it/s]
```



### Steps to reproduce the bug

```
load_dataset(
    args.dataset_name,
    args.dataset_config_name,
    cache_dir=args.cache_dir,
),
```

### Expected behavior

loads the dataset

### Environment info

datasets==2.8.0
Python 3.10.8
Linux 129-146-3-202 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Hi ! Could you share the full stack trace ? Which dataset did you try to load ?

it may be related to https://github.com/huggingface/datasets/pull/5640

The `BadZipFile` error means the ZIP file is corrupted, so I'm closing this issue as it's not directly related to `datasets`.

For others that find this issue following a `BadZipFile` error, I had the same problem because I had a file in a folder dataset `my-image.target` and the datasets library was incorrectly determining that the (PNG) file was a zip archive. When it tried to extract the file, this error occurred. 

Updating to `datasets==2.12.0` fixed the problem for me.
test v0.12.0.rc0
State: closed

DO NOT MERGE.
Only to test the CI.

cc @lhoestq @albertvillanova 

_The documentation is not available anymore as the PR was closed or merged._

@Wauplin I was testing it in a dedicated branch without opening a PR: https://github.com/huggingface/datasets/commits/test-hfh-0.12.0rc0

Oops, sorry @albertvillanova. I thought for next time I'll start the CIs before pinging everyone.
I'm closing this one.

@Wauplin in your Slack message, you asked people from every major dependent library to check that our CI work. That is why I am checking it... :)

Also, I think for this purpose it is better to test it in a dedicated branch, rather than opening and closing a PR.

Yes, yes I know. Completely my fault on this one
CI tests are broken: AttributeError: 'mappingproxy' object has no attribute 'target'
State: closed

CI tests are broken, raising `AttributeError: 'mappingproxy' object has no attribute 'target'`. See: https://github.com/huggingface/datasets/actions/runs/3966497597/jobs/6797384185
```
...
ERROR tests/test_streaming_download_manager.py::TestxPath::test_xpath_rglob[mock://top_level-date=2019-10-0[1-4]/*-expected_paths4] - AttributeError: 'mappingproxy' object has no attribute 'target'
===== 2076 passed, 19 skipped, 15 warnings, 47 errors in 115.54s (0:01:55) =====
```
OneDrive Integrations  with HF Datasets 
State: closed

### Feature request

First of all , I would like to thank all community who are developed DataSet storage and make it free available  
How to integrate our Onedrive account or any other possible storage clouds (like google drive,...) with the **HF** datasets section.
For example, if  I have **50GB** on my **Onedrive** account and I want to move between drive and Hugging face repo or vis versa 

### Motivation

make the dataset section more flexible with other possible storage 
like the integration between Google Collab and Google drive the storage

### Your contribution

Can be done using Hugging face CLI 

Hi! 

We use [`fsspec`](https://github.com/fsspec/filesystem_spec) to integrate with storage providers. You can find more info (and the usage examples) in [our docs](https://huggingface.co/docs/datasets/v2.8.0/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage).

[`gdrivefs`](https://github.com/fsspec/gdrivefs) makes it possible to use Google Drive as a storage service in Datasets, but this is not the case for OneDrive, since its[ Python SDK](https://github.com/OneDrive/onedrive-sdk-python) is not integrated with `fsspec`. Can you please request the integration with `fsspec` in their repo to address this limitation?

I'm closing this issue as implementing a fsspec-compliant OneDrive filesystem is not our responsibility.
[dataset request] Add Common Voice 12.0
State: closed

### Feature request

Please add the common voice 12_0 datasets. Apart from English, a significant amount of audio-data has been added to the other minor-language datasets.

### Motivation

The dataset link:
https://commonvoice.mozilla.org/en/datasets


@polinaeterna any tentative date on when the Common Voice 12.0 dataset will be added ?

This dataset is now hosted on the Hub here: https://huggingface.co/datasets/mozilla-foundation/common_voice_12_0
Can't load png dataset with 4 channel (RGBA)
State: closed

I try to create dataset which contains about 9000 png images 64x64 in size, and they are all 4-channel (RGBA). When trying to use load_dataset() then a dataset is created from only 2 images. What exactly interferes I can not understand.![Screenshot_20230117_212213.jpg](https://user-images.githubusercontent.com/41611046/212980147-9aa68e30-76e9-4b61-a937-c2fdabd56564.jpg)

Hi! Can you please share the directory structure of your image folder and the `load_dataset` call? We decode images with Pillow, and Pillow supports RGBA PNGs, so this shouldn't be a problem.



> Hi! Can you please share the directory structure of your image folder and the `load_dataset` call? We decode images with Pillow, and Pillow supports RGBA PNGs, so this shouldn't be a problem.
> 
> 

I have only 1 folder that I use in the load_dataset function with the name "IMGDATA" and all my 9000 images are located in this folder.
`
from datasets import load_dataset

dataset = load_dataset("IMGDATA")
`
At the same time, using another data set with images consisting of 3 RGB channels, everything works

Okay, I figured out what was wrong. When uploading my dataset via Google Drive, the images broke and Pillow couldn't open them. As a result, I solved the problem by downloading the ZIP archive
Wrong statement in "Load a Dataset in Streaming mode" leads to data leakage
State: closed

### Describe the bug

In the [Split your dataset with take and skip](https://huggingface.co/docs/datasets/v1.10.2/dataset_streaming.html#split-your-dataset-with-take-and-skip), it states:

> Using take (or skip) prevents future calls to shuffle from shuffling the dataset shards order, otherwise the taken examples could come from other shards. In this case it only uses the shuffle buffer. Therefore it is advised to shuffle the dataset before splitting using take or skip. See more details in the [Shuffling the dataset: shuffle](https://huggingface.co/docs/datasets/v1.10.2/dataset_streaming.html#iterable-dataset-shuffling) section.`
>> \# You can also create splits from a shuffled dataset
>> train_dataset = shuffled_dataset.skip(1000)
>> eval_dataset = shuffled_dataset.take(1000)


Where the shuffled dataset comes from:
`shuffled_dataset = dataset.shuffle(buffer_size=10_000, seed=42)`

At least in Tensorflow 2.9/2.10/2.11, [docs](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle) states the `reshuffle_each_iteration` argument is `True` by default. This means the dataset would be shuffled after each epoch, and as a result **the validation data would leak into training test**.






### Steps to reproduce the bug

N/A

### Expected behavior

The `reshuffle_each_iteration` argument should be set to `False`.

### Environment info

Tensorflow 2.9/2.10/2.11

Just for your information, Tensorflow confirmed this issue [here.](https://github.com/tensorflow/tensorflow/issues/59279)

Thanks for reporting, @HaoyuYang59.

Please note that these are different "dataset" objects: our docs refer to Hugging Face `datasets.Dataset` and not to TensorFlow `tf.data.Dataset`.

Our `datasets.Dataset.shuffle` method does not have a `reshuffle_each_iteration` argument. Therefore, I would say the statement in our docs is True because they refer to `datasets.Dataset.shuffle`, `datasets.Dataset.skip` and `datasets.Dataset.take`.

I think this issue is restricted to TensorFlow dataset, and this would be addressed by them in the issue you opened in their repo: https://github.com/tensorflow/tensorflow/issues/59279

Also note that you are referring to an outdated documentation page: datasets 1.10.2 version

Current datasets version is 2.8.0 and the corresponding documentation page is: https://huggingface.co/docs/datasets/stream#split-dataset

Hi @albertvillanova thanks for your reply and your explaination here. 

Sorry for the confusion as I'm not actually a user of your repo and I just happen to find the thread by Google (and didn't read carefully).

Great to know that and you made everything very clear now.

Thanks for your time and sorry for the consusion.

Wishing you a wonderful time. 

Regards
sample_dataset module not found
State: closed


Hi! Can you describe what the actual error is?

working on the setfit example script

   from setfit import SetFitModel, SetFitTrainer, sample_dataset

ImportError: cannot import name 'sample_dataset' from 'setfit' (C:\Python\Python38\lib\site-packages\setfit\__init__.py)

 apart from that, I also had to hack these loads to import thses modules:
    from datasets.load import load_dataset                        
    from datasets.arrow_dataset import Dataset
    from datasets.dataset_dict import DatasetDict

Hi! This issue is related to the [SetFit](https://github.com/huggingface/setfit) project, so can you please open it there?
Support latest Docker image in CI benchmarks
State: closed

Once we find out the root cause of:
- #5431 

we should revert the temporary pin on the Docker image version introduced by:
- #5432

Sorry, it was us:[^1] https://github.com/iterative/cml/pull/1317 & https://github.com/iterative/cml/issues/1319#issuecomment-1385599559; should be fixed with [v0.18.17](https://github.com/iterative/cml/releases/tag/v0.18.17).

[^1]: More or less, see https://github.com/yargs/yargs/issues/873.

Opened https://github.com/huggingface/datasets/pull/5436 unpinning again the container image.

Hi @0x2b3bfa0, thanks a lot for the investigation, the context about the the root cause and for fixing it!!

We are reviewing your PR to unpin the container image.
CI benchmarks are broken: Unknown arguments: runnerPath, path
State: closed

Our CI benchmarks are broken, raising `Unknown arguments` error: https://github.com/huggingface/datasets/actions/runs/3932397079/jobs/6724905161
```
Unknown arguments: runnerPath, path
```

Stack trace:
```
100%|██████████| 500/500 [00:01<00:00, 338.98ba/s]
Updating lock file 'dvc.lock'

To track the changes with git, run:

	git add dvc.lock

To enable auto staging, run:

	dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.
cml send-comment <markdown file>

Global Options:
  --log                    Logging verbosity
          [string] [choices: "error", "warn", "info", "debug"] [default: "info"]
  --driver                 Git provider where the repository is hosted
    [string] [choices: "github", "gitlab", "bitbucket"] [default: infer from the
                                                                    environment]
  --repo                   Repository URL or slug
                                  [string] [default: infer from the environment]
  --driver-token, --token  CI driver personal/project access token (PAT)
                                  [string] [default: infer from the environment]
  --help                   Show help                                   [boolean]

Options:
  --target                    Comment type (`commit`, `pr`, `commit/f00bar`,
                              `pr/42`, `issue/1337`),default is automatic (`pr`
                              but fallback to `commit`).                [string]
  --watch                     Watch for changes and automatically update the
                              comment                                  [boolean]
  --publish                   Upload any local images found in the Markdown
                              report                   [boolean] [default: true]
  --publish-url               Self-hosted image server URL
                                     [string] [default: "https://asset.cml.dev/"]
  --publish-native, --native  Uses driver's native capabilities to upload assets
                              instead of CML's storage; not available on GitHub
                                                                       [boolean]
  --watermark-title           Hidden comment marker (used for targeting in
                              subsequent `cml comment update`); "{workflow}" &
                              "{run}" are auto-replaced   [string] [default: ""]

Unknown arguments: runnerPath, path
Error: Process completed with exit code 1.
```

Issue reported to iterative/cml:
- iterative/cml#1319
Load/Save FAISS index using fsspec
State: closed

### Feature request

From what I understand `faiss` already support this [link](https://github.com/facebookresearch/faiss/wiki/Index-IO,-cloning-and-hyper-parameter-tuning#generic-io-support)

I would like to use a stream as input to `Dataset.load_faiss_index` and `Dataset.save_faiss_index`.

### Motivation

In my case, I'm saving faiss index in cloud storage and use `fsspec` to load them. It would be ideal if I could send the stream directly instead of copying the file locally (or mounting the bucket) and then load the index.

### Your contribution

I can submit the PR

Hi! Sure, feel free to submit a PR. Maybe if we want to be consistent with the existing API, it would be cleaner to directly add support for `fsspec` paths in `Dataset.load_faiss_index`/`Dataset.save_faiss_index` in the same manner as it was done in `Dataset.load_from_disk`/`Dataset.save_to_disk`.

That's a great idea! I'll do that instead.
Unable to download dataset id_clickbait
State: closed

### Describe the bug

I tried to download dataset `id_clickbait`, but receive this error message.

```
FileNotFoundError: Couldn't find file at https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/k42j7x2kpn-1.zip
```

When i open the link using browser, i got this XML data.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>md-datasets-cache-zipfiles-prod</BucketName><RequestId>NVRM6VEEQD69SD00</RequestId><HostId>W/SPDxLGvlCGi0OD6d7mSDvfOAUqLAfvs9nTX50BkJrjMny+X9Jnqp/Li2lG9eTUuT4MUkAA2jjTfCrCiUmu7A==</HostId></Error>
```

### Steps to reproduce the bug

Code snippet:

```
from datasets import load_dataset

load_dataset('id_clickbait', 'annotated')
load_dataset('id_clickbait', 'raw')
```

Link to Kaggle notebook: https://www.kaggle.com/code/ilosvigil/bug-check-on-id-clickbait-dataset

### Expected behavior

Successfully download and load `id_newspaper` dataset.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 8.0.0
- Pandas version: 1.3.5

Thanks for reporting, @ilos-vigil.

We have transferred this issue to the corresponding dataset on the Hugging Face Hub: https://huggingface.co/datasets/id_clickbait/discussions/1
When applying `ReadInstruction` to custom load it's not DatasetDict but list of Dataset?
State: closed

### Describe the bug

I am loading datasets from custom `tsv` files stored locally and applying split instructions for each split. Although the ReadInstruction is being applied correctly and I was expecting it to be `DatasetDict` but instead it is a list of `Dataset`. 

### Steps to reproduce the bug

Steps to reproduce the behaviour:

1. Import 
   `from datasets import load_dataset, ReadInstruction`
2. Instruction to load the dataset
  ```
instructions = [
      ReadInstruction(split_name="train", from_=0, to=10, unit='%', rounding='closest'),
      ReadInstruction(split_name="dev", from_=0, to=10, unit='%', rounding='closest'),
      ReadInstruction(split_name="test", from_=0, to=5, unit='%', rounding='closest')
  ]
```
3. Load 
`dataset = load_dataset('csv', data_dir="data/", data_files={"train":"train.tsv", "dev":"dev.tsv", "test":"test.tsv"}, delimiter="\t", split=instructions)`


### Expected behavior

**Current behaviour**

![Screenshot from 2023-01-16 10-45-27](https://user-images.githubusercontent.com/25720695/212614754-306898d8-8c27-4475-9bb8-0321bd939561.png)
:
 
**Expected behaviour**

![Screenshot from 2023-01-16 10-45-42](https://user-images.githubusercontent.com/25720695/212614813-0d336bf7-5266-482e-bb96-ef51f64de204.png)


### Environment info

``datasets==2.8.0
``
`Python==3.8.5
`
`Platform - Ubuntu 20.04.4 LTS`

Hi! You can get a `DatasetDict` if you pass a dictionary with read instructions as follows:
```python
instructions = [
    ReadInstruction(split_name="train", from_=0, to=10, unit='%', rounding='closest'),
    ReadInstruction(split_name="dev", from_=0, to=10, unit='%', rounding='closest'),
    ReadInstruction(split_name="test", from_=0, to=5, unit='%', rounding='closest')
]

dataset = load_dataset('csv', data_dir="data/", data_files={"train":"train.tsv", "dev":"dev.tsv", "test":"test.tsv"}, delimiter="\t", split={inst.split_name: inst for inst in instructions})
```
Support case-insensitive Hub dataset name in load_dataset
State: closed

### Feature request

The dataset name on the Hub is case-insensitive (see https://github.com/huggingface/moon-landing/pull/2399, internal issue), i.e., https://huggingface.co/datasets/GLUE redirects to https://huggingface.co/datasets/glue.

Ideally, we could load the glue dataset using the following:

```
from datasets import load_dataset
load_dataset('GLUE', 'cola')
```

It breaks because the loading script `GLUE.py` does not exist (`glue.py` should be selected instead).

Minor additional comment: in other cases without a loading script, we can load the dataset, but the automatically generated config name depends on the casing:
- `load_dataset('severo/danish-wit')` generates the config name `severo--danish-wit-e6fda5b070deb133`, while
- `load_dataset('severo/danish-WIT')` generates the config name `severo--danish-WIT-e6fda5b070deb133`

### Motivation

To follow the same UX on the Hub and in the datasets library.

### Your contribution

...

Closing as case-insensitivity should be only for URL redirection on the Hub. In the APIs, we will only support the canonical name (https://github.com/huggingface/moon-landing/pull/2399#issuecomment-1382085611)
label_column='labels' in datasets.TextClassification and 'label' or 'label_ids' in transformers.DataColator
State: closed

### Describe the bug

When preparing a dataset for a task using `datasets.TextClassification`, the output feature is named `labels`. When preparing the trainer using the `transformers.DataCollator` the default column name is `label` if binary or `label_ids` if multi-class problem.

It is required to rename the column accordingly to the expected name : `label` or `label_ids`

### Steps to reproduce the bug

```python
from datasets import TextClassification, AutoTokenized, DataCollatorWithPadding
ds_prepared = my_dataset.prepare_for_task(TextClassification(text_column='TEXT', label_column='MY_LABEL_COLUMN_1_OR_0'))
print(ds_prepared)

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
ds_tokenized = ds_prepared.map(lambda x: tokenizer(x['text'], truncation=True), batched=True)
print(ds_tokenized)

data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")

tf_data = model.prepare_tf_dataset(ds_tokenized, shuffle=True, batch_size=16, collate_fn=data_collator)

print(tf_data)
```

### Expected behavior

Without renaming the the column, the  target column is not in the final tf_data since it is not in the column name expected by the data_collator.

To correct this, we have to rename the column:

```python
ds_prepared = my_dataset.prepare_for_task(TextClassification(text_column='TEXT', label_column='MY_LABEL_COLUMN_1_OR_0')).rename_column('labels', 'label')
```

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.6
- PyArrow version: 10.0.1
- Pandas version: 1.5.2

- `transformers` version: 4.26.0.dev0
- Platform: Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.11.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Hi! Thanks for pointing out this inconsistency. Changing the default value at this point is probably not worth it, considering we've started discussing the state of the task API internally - we will most likely deprecate the current one and replace it with a more robust solution that relies on the `train_eval_index` field stored in the YAML section of the dataset cards.

The task templates API has been deprecated (will be removed in version 3.0), so I'm closing this issue.
Add ProgressBar for `to_parquet`
State: closed

### Feature request

Add a progress bar for `Dataset.to_parquet`, similar to how `to_json` works.

### Motivation

It's a bit frustrating to not know how long a dataset will take to write to file and if it's stuck or not without a progress bar

### Your contribution

Sure I can help if needed

Thanks for your proposal, @zanussbaum. Yes, I agree that would definitely be a nice feature to have!

@albertvillanova I’m happy to make a quick PR for the feature! let me know 

That would be awesome ! You can comment `#self-assign` to assign you to this issue and open a PR :) Will be happy to review

Closing as this has been merged @lhoestq
concatenate_datasets fails when two dataset with shards > 1 and unequal shard numbers
State: closed

### Describe the bug

When using `concatenate_datasets([dataset1, dataset2], axis = 1)` to concatenate two datasets with shards > 1, it fails:

```
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/combine.py", line 182, in concatenate_datasets
    return _concatenate_map_style_datasets(dsets, info=info, split=split, axis=axis)
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 5499, in _concatenate_map_style_datasets
    table = concat_tables([dset._data for dset in dsets], axis=axis)
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/table.py", line 1778, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/table.py", line 1483, in from_tables
    blocks = _extend_blocks(blocks, table_blocks, axis=axis)
File "/home/xzg/anaconda3/envs/tri-transfer/lib/python3.9/site-packages/datasets/table.py", line 1477, in _extend_blocks
    result[i].extend(row_blocks)
IndexError: list index out of range
```

### Steps to reproduce the bug

dataset = concatenate_datasets([dataset1, dataset2], axis = 1)

### Expected behavior

The datasets are correctly concatenated.

### Environment info

datasets==2.8.0

Hi ! Thanks for reporting :)

I managed to reproduce the hub using
```python

from datasets import concatenate_datasets, Dataset, load_from_disk

Dataset.from_dict({"a": range(9)}).save_to_disk("tmp/ds1")
ds1 = load_from_disk("tmp/ds1")
ds1 = concatenate_datasets([ds1, ds1])

Dataset.from_dict({"b": range(6)}).save_to_disk("tmp/ds2")
ds2 = load_from_disk("tmp/ds2")
ds2 = concatenate_datasets([ds2, ds2, ds2])

concatenate_datasets([ds1, ds2], axis=1)
```
and I get
```python
Traceback (most recent call last):                                                                                                    
  File "test.py", line 98, in <module>
    dds = concatenate_datasets([ds1, ds2], axis=1)
  File "/Users/.../datasets/combine.py", line 182, in concatenate_datasets
    return _concatenate_map_style_datasets(dsets, info=info, split=split, axis=axis)
  File "/Users/.../datasets/arrow_dataset.py", line 5499, in _concatenate_map_style_datasets
    table = concat_tables([dset._data for dset in dsets], axis=axis)
  File "/Users/.../datasets/table.py", line 1778, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/Users/.../datasets/table.py", line 1483, in from_tables
    blocks = _extend_blocks(blocks, table_blocks, axis=axis)
  File "/Users/.../datasets/table.py", line 1477, in _extend_blocks
    result[i].extend(row_blocks)
IndexError: list index out of range
```

It appears to happen when the two datasets have a number of shards that is not the same
Datasets.from_sql() generates deprecation warning
State: closed

### Describe the bug

Calling `Datasets.from_sql()` generates a warning:

`.../site-packages/datasets/builder.py:712: FutureWarning: 'use_auth_token' was deprecated in version 2.7.1 and will be removed in 3.0.0. Pass 'use_auth_token' to the initializer/'load_dataset_builder' instead.`

### Steps to reproduce the bug

Any valid call to `Datasets.from_sql()` will produce the deprecation warning.

### Expected behavior

No warning.

The fix should be simply to remove the parameter `use_auth_token` from the call to `builder.download_and_prepare()` at line 43 of `io/sql.py` (it is set to `None` anyway, and is not needed).

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-4.15.0-169-generic-x86_64-with-glibc2.27
- Python version: 3.9.15
- PyArrow version: 10.0.1
- Pandas version: 1.5.2


Thanks for reporting @msummerfield. We are fixing it.
size_in_bytes the same for all splits
State: open

### Describe the bug

Hi, it looks like whenever you pull a dataset and get size_in_bytes, it returns the same size for all splits (and that size is the combined size of all splits). It seems like this shouldn't be the intended behavior since it is misleading. Here's an example:

```
>>> from datasets import load_dataset
>>> x = load_dataset("glue", "wnli")
Found cached dataset glue (/Users/breakend/.cache/huggingface/datasets/glue/wnli/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1097.70it/s]
>>> x["train"].size_in_bytes
186159
>>> x["validation"].size_in_bytes
186159
>>> x["test"].size_in_bytes
186159
>>> 
```

### Steps to reproduce the bug

```
>>> from datasets import load_dataset
>>> x = load_dataset("glue", "wnli")
>>> x["train"].size_in_bytes
186159
>>> x["validation"].size_in_bytes
186159
>>> x["test"].size_in_bytes
186159
```

### Expected behavior

The expected behavior is that it should return the separate sizes for all splits.

### Environment info

- `datasets` version: 2.7.1
- Platform: macOS-12.5-arm64-arm-64bit
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.2

Hi @Breakend,

Indeed, the attribute `size_in_bytes` refers to the size of the entire dataset configuration, for all splits (size of downloaded files + Arrow files), not the specific split.
This is also the case for `download_size` (downloaded files) and `dataset_size` (Arrow files).

The size of the Arrow files for a specific split can be accessed: e.g. size of the "test" split only
```python
ds["train"].info.splits["test"].num_bytes
```

I agree this is confusing and maybe we should improve it.
Better integration of BIG-bench
State: open

### Feature request

Ideally, it would be nice to have a maintained PyPI package for `bigbench`.

### Motivation

We'd like to allow anyone to access, explore and use any task.

### Your contribution

@lhoestq has opened an issue in their repo:
- https://github.com/google/BIG-bench/issues/906

Hi, I made my version : https://huggingface.co/datasets/tasksource/bigbench
Support Dataset conversion from/to Spark
State: open

This PR implements Spark integration by supporting `Dataset` conversion from/to Spark `DataFrame`.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5401). All of your documentation changes will be reflected on that endpoint.

Cool thanks !

Spark DataFrame are usually quite big, and I believe here `from_spark` would load everything in the driver node's RAM, which is quite limiting. Same for `to_spark` which would load everything in the driver node's RAM before sending the data to the executor. Maybe we can mention this in the docstring ?

To transfer big datasets from/into the HF ecosystem using Spark maybe we can just make sure that `pyspark` can read/write to the HF Hub, and that `datasets` can read from HDFS/S3/etc.

Yes @lhoestq , consider this as a first integration of the Datasets library with Spark.
- This PR implements the basic conversion between both.
  - And yes, we are using the Spark's `pandas` API (that uses `pyarrow` under the hood): everything is transferred to the driver.
  - Note that we are converting from/to a Datasets dataset: this is not distributed

The next step is to support the integration of the HF Hub with Spark, that I think should be done using `hffs`.

Thinking more about it I don't really see how those two methods help in practice, since one can already do `datasets` <-> pandas <-> spark and those two methods don't add value over this.

However I think it can be good documentation to explain that it's possible to do it and it's super simple
Fix checksum verification
State: closed

Expected checksum was verified against checksum dict (not checksum).

Hi ! If I'm not mistaken both `expected_checksums[url]` and `recorded_checksums[url]` are dictionaries with keys "checksum" and "num_bytes". So we need to check whether `expected_checksums[url] != recorded_checksums[url]` (or simply `expected_checksums[url]["checksum"] != recorded_checksums[url]["checksum"]`)

But in your fix you're checking `expected_checksums[url] != recorded_checksums[url]['checksum']`.

So I think it's fine to keep this as is

No, the issue is that there is comparison of sclar value and dictionary.

Acording to [`DatasetInfo`][1], we need specify a dictionary which maps a URL to a checksum as follows.

```python
CHECKSUMS = {
    URL: 'a5dc6bf63ea088ade6e98594bfa386f45211c38b2a3db3dd11b33bd530f3c481',
}

class FancyDataset:
    def _info(self):
        return DatasetInfo(..., download_checksums=CHECKSUMS)
```

However, `load_dataset` fails with this checksum definition.

[1]: https://github.com/huggingface/datasets/blob/main/src/datasets/info.py#L124-L125

I think it has to be formatted like this right now. Maybe the DatasetInfo doc is unclear and we can improve it
```python
CHECKSUMS = {
    URL: {"checksum": checksum, "num_bytes": num_bytes},
}
```

Right. I am not sure that this is a correct way to do it. People usually calculate sha256, md5, or whatever else but not size in bytes. Also, people use only some of checksum algorithms. This means that comparing dictionaries in `verify_checksums` is too strict (requires equality of all items) and raises compatibility issues in the future. Another issue is that a comparison of dictionaries assumes type constraints which imply type equality. 

Since almost noone uses checksums as far as I known, my PR suggests a minimal change to mitigate these issues except support of a specific checksum algorithm which is a separated feature and should be contributed in a separate PRs from my perspective.

Applying this change will break the verification code, since the `expected_checksums` is a dict with those two keys.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5396). All of your documentation changes will be reflected on that endpoint.
CI error: TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers'
State: closed

### Describe the bug

While installing the dependencies, the CI raises a TypeError:
```
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/spacy/__init__.py", line 6, in <module>
    from .errors import setup_default_warnings
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/spacy/errors.py", line 2, in <module>
    from .compat import Literal
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/spacy/compat.py", line 3, in <module>
    from thinc.util import copy_array
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/thinc/__init__.py", line 5, in <module>
    from .config import registry
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/thinc/config.py", line 2, in <module>
    import confection
  File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/confection/__init__.py", line 10, in <module>
    from pydantic import BaseModel, create_model, ValidationError, Extra
  File "pydantic/__init__.py", line 2, in init pydantic.__init__
  File "pydantic/dataclasses.py", line 46, in init pydantic.dataclasses
    # | None    | Attribute is set to None.               |
  File "pydantic/main.py", line 121, in init pydantic.main
TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers'
```

See: https://github.com/huggingface/datasets/actions/runs/3793736481/jobs/6466356565

### Steps to reproduce the bug

```shell
pip install .[tests,metrics-tests]
python -m spacy download en_core_web_sm
```

### Expected behavior

No error.

### Environment info

See: https://github.com/huggingface/datasets/actions/runs/3793736481/jobs/6466356565

I still getting the same error :

`python -m spacy download fr_core_news_lg
`.
`import spacy`

@MFatnassi, this issue and the corresponding fix only affect our Continuous Integration testing environment.

Note that `datasets` does not depend on `spacy`.
Getting Value Error while loading a dataset..
State: closed

### Describe the bug

I am trying to load a dataset using Hugging Face Datasets load_dataset method. I am getting the value error as show below. Can someone help with this? I am using Windows laptop and Google Colab notebook.

```
WARNING:datasets.builder:Using custom data configuration default-a1d9e8eaedd958cd
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-12-5b4fdcb8e6d5>](https://localhost:8080/#) in <module>
      6 )
      7 
----> 8 next(iter(law_dataset_streamed))

17 frames
[/usr/local/lib/python3.8/dist-packages/fsspec/core.py](https://localhost:8080/#) in get_compression(urlpath, compression)
    485         compression = infer_compression(urlpath)
    486     if compression is not None and compression not in compr:
--> 487         raise ValueError("Compression type %s not supported" % compression)
    488     return compression
    489 

ValueError: Compression type zstd not supported
```

### Steps to reproduce the bug

```
!pip install zstandard
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
```

### Expected behavior

I expect an iterable object as the output 'lds' to be created.

### Environment info

Windows laptop with Google Colab notebook

Hi! I can't reproduce this error locally (Mac) or in Colab. What version of `datasets` are you using?

Hi [mariosasko](https://github.com/mariosasko), the datasets version is '2.8.0'.

@valmetisrinivas you get that error because you imported `datasets` (and thus `fsspec`) before installing `zstandard`.

Please, restart your Colab runtime and execute the install commands before importing `datasets`:
```python
!pip install datasets
!pip install zstandard

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(ds))
```

> @valmetisrinivas you get that error because you imported `datasets` (and thus `fsspec`) before installing `zstandard`.
> 
> Please, restart your Colab runtime and execute the install commands before importing `datasets`:
> 
> ```python
> !pip install datasets
> !pip install zstandard
> 
> from datasets import load_dataset
> 
> ds = load_dataset(
>     "json",
>     data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
>     split="train",
>     streaming=True,
> )
> next(iter(ds))
> ```

I guess that was the problem, importing datasets before the installation of zstandard. Thank you for the feedback.
Missing documentation page : improve-performance
State: closed

### Describe the bug

Trying to access https://huggingface.co/docs/datasets/v2.8.0/en/package_reference/cache#improve-performance, the page is missing.

The link is in here : https://huggingface.co/docs/datasets/v2.8.0/en/package_reference/loading_methods#datasets.load_dataset.keep_in_memory

### Steps to reproduce the bug

Access the page and see it's missing.

### Expected behavior

Not missing page

### Environment info

Doesn't matter

Hi! Our documentation builder does not support links to sections, hence the bug. This is the link it should point to https://huggingface.co/docs/datasets/v2.8.0/en/cache#improve-performance.
Is `fs=` deprecated in `load_from_disk()` as well?
State: closed

### Describe the bug

The `fs=` argument was deprecated from `Dataset.save_to_disk` and `Dataset.load_from_disk` in favor of automagically figuring it out via fsspec:
https://github.com/huggingface/datasets/blob/9a7272cd4222383a5b932b0083a4cc173fda44e8/src/datasets/arrow_dataset.py#L1339-L1340

Is there a reason the same thing shouldn't also apply to `datasets.load.load_from_disk()` as well ?

https://github.com/huggingface/datasets/blob/9a7272cd4222383a5b932b0083a4cc173fda44e8/src/datasets/load.py#L1779



### Steps to reproduce the bug

n/a

### Expected behavior

n/a

### Environment info

n/a

Hi! Yes, we should deprecate the `fs` param here. Would you be interested in submitting a PR? 

> Hi! Yes, we should deprecate the `fs` param here. Would you be interested in submitting a PR?

Yeah I can do that sometime next week. Should the storage_options be a new arg here?  I’ll look around for anywhere else where fs is an arg.

Closed by #5393.
IterableDataset missing column_names, differs from Dataset interface
State: closed

### Describe the bug

The documentation on [Stream](https://huggingface.co/docs/datasets/v1.18.2/stream.html) seems to imply that IterableDataset behaves just like a Dataset. However, examples like
```
dataset.map(augment_data, batched=True, remove_columns=dataset.column_names, ...)
```
will not work because `.column_names` does not exist on IterableDataset. I cannot find any clear explanation on why this is not available, is it an oversight? We do have `iterable_ds.features` available.

### Steps to reproduce the bug

See above

### Expected behavior

Dataset and IterableDataset would be expected to have the same interface, with any differences noted in the documentation.

### Environment info

n/a

Another example is that `IterableDataset.map` does not have `fn_kwargs`, among other arguments. It makes it harder to convert code from Dataset to IterableDataset.

Hi! `fn_kwargs` was added to `IterableDataset.map` in `datasets 2.5.0`, so please update your installation (`pip install -U datasets`) to use it.

Regarding `column_names`, I agree we should add this property to `IterableDataset`. In the meantime, you can use `list(dataset.features.keys())` instead.

Thanks! That's great news.

On Thu, Dec 22, 2022, 07:48 Mario Šaško ***@***.***> wrote:

> Hi! fn_kwargs was added to IterableDataset.map in datasets 2.5.0, so
> please update your installation (pip install -U datasets) to use it.
>
> Regarding column_names, I agree we should add this property to
> IterableDataset. In the meantime, you can use
> list(dataset.features.keys()) instead.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5383#issuecomment-1362993633>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHD6N2EQUFEOUFDW3VHSILWORZ45ANCNFSM6AAAAAATGKWVGM>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


I'm marking this issue as a "good first issue", as it makes sense to have `IterableDataset.column_names` in the API. Besides the case when `features` are `None` (e.g., `features` are `None` after `map`), in which we can also return `column_names` as `None`, adding this property should be straightforward,

Hi @mariosasko, I can work on this if that's ok?

Yes! I've assigned you the issue.