konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Lossy json serialization - deserialization of dataset info 

### Describe the bug

Saving a dataset to disk as json (using `to_json`) and then loading it again (using `load_dataset`) results in features whose labels are not type-cast correctly. In the code snippet below, `features.label` should have a label of type `ClassLabel` but has type `Value` instead. 

### Steps to reproduce the bug

```
from datasets import load_dataset

def test_serdes_from_json(d):
    dataset = load_dataset(d, split="train")
    dataset.to_json('_test')
    dataset_loaded = load_dataset("json", data_files='_test', split='train')
    try:
        assert dataset_loaded.info.features == dataset.info.features, "features unequal!"
    except Exception as ex:
        print(f'{ex}')
        print(f'expected {dataset.info.features}, \nactual { dataset_loaded.info.features }')
    
test_serdes_from_json('rotten_tomatoes')
```

Output
```
features unequal!
expected {'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)}, 
actual {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}
```


### Expected behavior

The deserialized `features.label` should have type `ClassLabel`.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.10.144-127.601.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.2.3

Hi ! JSON is a lossy format indeed. If you want to keep the feature types or other metadata I'd encourage you to store them as well. For example you can use `dataset.info.write_to_directory` and `DatasetInfo.from_directory` to store the feature types, split info, description, license etc.
Save file name in embed_storage

Having the file name is useful in case we need to check the extension of the file (e.g. mp3), or in general in case it includes some metadata information (track id, image id etc.)

Related to https://github.com/huggingface/datasets/issues/5276

_The documentation is not available anymore as the PR was closed or merged._

I updated the tests, met le know if it sounds good to you now :)
Import error

https://github.com/huggingface/datasets/blob/cd3d8e637cfab62d352a3f4e5e60e96597b5f0e9/src/datasets/__init__.py#L28

Hy, 
I have error at the above line. I have python version 3.8.13, the message says I need python>=3.7, which is True, but I think the if statement not working properly (or the message wrong)

Hi ! Can you 
```python
import platform
print(platform.python_version())
```
to see that it returns ?

Hi,

3.8.13

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:37:02 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)


Hi ! Can you

import platform
print(platform.python_version())

to see that it returns ?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323691385>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F5YGG32W6WABYC25NJTWJTD75ANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


Then it should work as expected if you use the same python when using `datasets`

Please make sure you're running your code in the right environment

It's the right environment. But in if statement I have
"3.8.13" < 3.7
And in the error message is Python>=3.7 which is true in my case (3.8.13 is greater then 3.7), so I don't understand my python should be below the 3.7 which case the if statement is right, but the message is wrong, or above 3.7 which case if statement is wrong, error message is right.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:41:43 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)


Then it should work as expected if you use the same python when using datasets

Please make sure you're running your code in the right environment

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323697094>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F54JURTAJJWWDO2QGI3WJTERPANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


If you're having an error then you're not running your code in the right environment.
Warn about checksums

It takes a lot of time on big datasets to compute the checksums, we should at least add a warning to notify the user about this step. I also mentioned how to disable it, and added a tqdm bar (delay=5 seconds)

cc @ola13 

_The documentation is not available anymore as the PR was closed or merged._

I'm also in favor of disabling this by default - it's kinda impractical

Great, thanks for the quick turnaround on this!
load_dataset does not read jsonl metadata file properly

### Describe the bug

Hi, I'm following [this page](https://huggingface.co/docs/datasets/image_dataset) to create a dataset of images and captions via an image folder and a metadata.json file, but I can't seem to get the dataloader to recognize the "text" column. It just spits out "image" and "label" as features. 

Below is code to reproduce my exact example/problem.

### Steps to reproduce the bug

```ruby 
dataset_link="19Unu89Ih_kP6zsE7f9Mkw8dy3NwHopRF"
id = dataset_link
output = 'Godardv01.zip'
gdown.download(id=id, output=output, quiet=False)

ds = load_dataset("imagefolder", data_dir="/kaggle/working/Volumes/TOSHIBA/Godard_imgs/Volumes/TOSHIBA/Godard_imgs/Full/train", split="train", drop_labels=False)
print(ds)
```

### Expected behavior

I would expect that it returned "image" and "text" columns from the code above.

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 5.0.0
- Pandas version: 1.3.5

Can you try to remove "drop_labels=false" ? It may force the loader to infer the labels instead of reading the metadata

Hi, thanks for responding. I tried that, but it does not change anything.

Can you try updating `datasets` ? Metadata support was added in `datasets` 2.4

Probably the issue, will report back asap!

Okay, now it seems to actually load the metadata and create the train_split, but it still says only returns "image" and "label", which is always 0 since all images are from same folder

> Can you try updating `datasets` ? Metadata support was added in `datasets` 2.4

Update: This was the issue.
load_dataset possibly broken for gated datasets?

### Describe the bug

When trying to download the [winoground dataset](https://huggingface.co/datasets/facebook/winoground), I get this error unless I roll back the version of huggingface-hub:

```
[/usr/local/lib/python3.7/dist-packages/huggingface_hub/utils/_validators.py](https://localhost:8080/#) in validate_repo_id(repo_id)
    165     if repo_id.count("/") > 1:
    166         raise HFValidationError(
--> 167             "Repo id must be in the form 'repo_name' or 'namespace/repo_name':"
    168             f" '{repo_id}'. Use `repo_type` argument if needed."
    169         )

HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'datasets/facebook/winoground'. Use `repo_type` argument if needed
```

### Steps to reproduce the bug

Install requirements:

```
pip install transformers
pip install datasets
# It works if you uncomment the following line, rolling back huggingface hub:
# pip install huggingface-hub==0.10.1
```

Then:

```
from datasets import load_dataset
auth_token = ""  # Replace with an auth token, which you can get from your huggingface account: Profile -> Settings -> Access Tokens -> New Token
winoground = load_dataset("facebook/winoground", use_auth_token=auth_token)["test"]
```





### Expected behavior

Downloading of the datset

### Environment info

Just a google colab; see here: https://colab.research.google.com/drive/15wwOSte2CjTazdnCWYUm2VPlFbk2NGc0?usp=sharing

@BradleyHsu

Btw, thanks very much for finding the hub rollback temporary fix and bringing the issue to our attention @KhoomeiK!

I see the same issue when calling `load_dataset('poloclub/diffusiondb', 'large_random_1k')` with `datasets==2.7.1` and `huggingface-hub=0.11.0`. No issue with `datasets=2.6.1` and `huggingface_hub==0.10.1`.

https://github.com/poloclub/diffusiondb/issues/7

I fixed my issue by specifying `repo_type` in `hf_hub_url()`. https://github.com/poloclub/diffusiondb/commit/9eb91c79aaca98b0515a0ce45778b8af65b84652

I opened a PR on the Winoground's repo: https://huggingface.co/datasets/facebook/winoground/discussions/2

This is a bug in the script, indeed. The most robust fix is to use a relative path instead of `hf_hub_url`, which does not depend on `huggingface_hub`'s version 🙂. I've opened a PR here: https://huggingface.co/datasets/facebook/winoground/discussions/3.

Awesome, big thanks to both @xiaohk and @mariosasko!

so, if i reproduce the bug, what should i do ? with huggingface_hub0.13.3  dataset2.6.1

huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name':

    tokenizer = AutoTokenizer.from_pretrained(ARGS.model_path, trust_remote_code=True)

Please handle automatically for local path and repo name inside, otherwise users always get confused about this

I think I'm also hitting this error, trying to load a model from a local path.
Sharded save_to_disk + multiprocessing

Added `num_shards=` and `num_proc=` to `save_to_disk()`

EDIT: also added `max_shard_size=` to `save_to_disk()`, and also `num_shards=` to `push_to_hub`

I also:
- deprecated the fs parameter in favor of storage_options (for consistency with the rest of the lib) in save_to_disk and load_from_disk
- always embed the image/audio data in arrow when doing `save_to_disk`
- added a tqdm bar in `save_to_disk`
- Use the MockFileSystem in tests for `save_to_disk` and `load_from_disk`
- removed the unused integration tests with S3, since we can now test with `mockfs` instead of `s3fs`

TODO:
- [x] implem save_to_disk for dataset dict
- [x] save_to_disk for dataset dict tests
- [x] deprecate fs in dataset dict load_from_disk as well
- [x] update docs

Close #5263 
Close https://github.com/huggingface/datasets/issues/4196
Close https://github.com/huggingface/datasets/issues/4351

_The documentation is not available anymore as the PR was closed or merged._

Added both num_shards and max_shard_size in push_to_hub/save_to_disk. Will take care of updating the tests later

It's ready for a final review @mariosasko and @albertvillanova, let me know what you think :)

Took your comments into account, and also changed `iflatmap_unordered` to take an iterable of kwargs to make the code more redable :)
Specify arguments as keywords in librosa.reshape to avoid future errors 

Fixes a warning and future deprecation from `librosa.reshape`:
```
FutureWarning: Pass orig_sr=16000, target_sr=48000 as keyword args. From version 0.10 passing these as positional arguments will result in an error

   array = librosa.resample(array, sampling_rate, self.sampling_rate, res_type="kaiser_best")
```

_The documentation is not available anymore as the PR was closed or merged._
Get an IterableDataset from a map-style Dataset

This is useful to leverage iterable datasets specific features like:
- fast approximate shuffling
- lazy map, filter etc.

Iterating over the resulting iterable dataset should be at least as fast at iterating over the map-style dataset.

Here are some ideas regarding the API:
```python
# 1. 
# - consistency with load_dataset(..., streaming=True)
# - gives intuition that map/filter/etc. are done on-the-fly
ids = ds.stream()

# 2.
# - more explicit on the output type
# - but maybe sounds like a conversion tool rather than a step in a processing pipeline
ids = ds.as_iterable_dataset()
```

I think `stream` could be misleading since the data is not being streamed from remote endpoints (one could think that's the case when they see `load_dataset` followed by `stream`). Hence, I prefer the second option.

PS: When we resolve https://github.com/huggingface/datasets/issues/4542, we could add `as_tf_dataset` to the API for consistency and deprecate `to_tf_dataset`.
AttributeError: 'Value' object has no attribute 'names'

Hello

I'm trying to build a model for custom token classification 
I already followed the token classification course on huggingface

while adapting the code to my work, this message occures : 
'Value' object has no attribute 'names'

Here's my code:
`raw_datasets` 
generates
DatasetDict({
    train: Dataset({
        features: ['isDisf', 'pos', 'tokens', 'id'],
        num_rows: 14
    })
})

`raw_datasets["train"][3]["isDisf"]`
generates 
['B_RM', 'I_RM', 'I_RM', 'B_RP', 'I_RP', 'O', 'O']

`dis_feature = raw_datasets["train"].features["isDisf"]
dis_feature`
generates
Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)

and
`label_names = dis_feature.feature.names
label_names`
generates
AttributeError                            Traceback (most recent call last)
[<ipython-input-28-972fd54a869a>](https://localhost:8080/#) in <module>
----> 1 label_names = dis_feature.feature.names
      2 label_names

AttributeError: 'Value' object has
AttributeError: 'Value' object has no attribute 'names'

Thank you for your help

Hi ! It looks like your "isDif" column is a Sequence of Value("string"), not a Sequence of ClassLabel.

You can convert your Value("string") feature type to a ClassLabel feature type this way:
```python
from datasets import ClassLabel, Sequence

# provide the label_names yourself
label_names = [...]
# OR get them from the dataset
label_names = sorted(set(label for labels in raw_datasets["train"]["isDif"] for label in labels))

# Cast to ClassLabel
raw_datasets = raw_datasets.cast_column("isDif", Sequence(ClassLabel(names=label_names)))
```


thank you 
it works 💯
Add PubTables-1M

### Name

PubTables-1M

### Paper

https://openaccess.thecvf.com/content/CVPR2022/html/Smock_PubTables-1M_Towards_Comprehensive_Table_Extraction_From_Unstructured_Documents_CVPR_2022_paper.html

### Data

https://github.com/microsoft/table-transformer

### Motivation

Table Transformer is now available in 🤗 Transformer, and it was trained on PubTables-1M. It's a large dataset for table extraction and structure recognition in unstructured documents.

cc @albertvillanova the author would like to add this dataset to the hub: https://github.com/microsoft/table-transformer/issues/68#issuecomment-1319114621. Could you help him out?
Restore order of split names in dataset_info for canonical datasets

After a bulk edit of canonical datasets to create the YAML `dataset_info` metadata, the split names were accidentally sorted alphabetically. See for example:
- https://huggingface.co/datasets/bc2gm_corpus/commit/2384629484401ecf4bb77cd808816719c424e57c

Note that this order is the one appearing in the preview of the datasets.

I'm making a bulk edit to align the order of the splits appearing in the metadata info with the order appearing in the loading script.

Related to:
- #5202

The bulk edit is running...

See for example: 
- A single config: https://huggingface.co/datasets/acronym_identification/discussions/2
- Multiple configs: https://huggingface.co/datasets/babi_qa/discussions/1

TODO: Add "dataset_info" YAML metadata to:
- [x] "chr_en" has no metadata JSON file, nor "dataset_info" YAML tag in its card
  - Fixing PR: https://huggingface.co/datasets/chr_en/discussions/1 
- [x] "conll2000" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "crime_and_punish" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "dart" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "iwslt2017" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [ ] "mc4" has no metadata JSON file, nor "dataset_info" YAML tag in its card
- [ ] "the_pile" has no metadata JSON file, nor "dataset_info" YAML tag in its card
- [ ] "timit_asr" has no metadata JSON file, nor "dataset_info" YAML tag in its card

The bulk edit is finished.
Support for decoding Image/Audio types in map when format type is not default one

Add support for decoding the `Image`/`Audio` types in `map` for the formats (Numpy, TF, Jax, PyTorch) other than the default one (Python).

Additional improvements:
* make `Dataset`'s "iter" API  cleaner by removing `_iter` and replacing `_iter_batches` with `iter(batch_size)` (also implemented for `IterableDataset`)
* iterate over arrow tables in `map` to avoid `_getitem` calls, which are much slower than `__iter__`/`iter(batch_size)`, when the `format_type` is not Python
* fix `_iter_batches` (now named `iter`) when `drop_last_batch=True` and `pyarrow<=8.0.0` is installed
* lazily extract and decode arrow data in the default format 

TODO:
* [x] update the `iter` benchmark in the docs (the `BeamBuilder` cannot load the preprocessed datasets from our bucket, so wait for this to be fixed (cc @lhoestq))

Fix https://github.com/huggingface/datasets/issues/3992, fix https://github.com/huggingface/datasets/issues/3756

_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5252). All of your documentation changes will be reflected on that endpoint.

Yes, if the image column is the first in the batch keys, it will decode the images because it reads the actual values. We could avoid this by checking the batch type, and if it's `LazyDict`, `num_examples` is equal to `len(batch.pa_table)`, which doesn't lead to decoding.

Good idea. This can be done in a subsequent PR btw, since it's out of scope of the original goal of this PR

Just fixed a small bug where it would show the pyarrow 10 warning about None -> empty lists conversions even with an Array2D with no nulls

Fixed another bug when your map function returns a mix of LazyDict or regular dict and added some tests
Docs are not generated after latest release

After the latest `datasets` release version 0.7.0, the docs were not generated.

As we have changed the release procedure (so that now we do not push directly to main branch), maybe we should also change the corresponding GitHub action:

https://github.com/huggingface/datasets/blob/edf1902f954c5568daadebcd8754bdad44b02a85/.github/workflows/build_documentation.yml#L3-L8

Related to:
- #5250

CC: @mishig25 

After a discussion with @mishig25:
- He said that this action should be triggered if we call our release branch according to the regex `v*-release`, as transformers does
- I said that our procedure is different: our release branch is *temporary* and it is deleted just after the release PR is merged to main
  - Indeed the release tag is not yet created when we make the release PR (not event when this is merged to main), but when we make the Release itself.

I was thinking that maybe we could change the triggering event: use `release` instead of `push`.

What do you think, @huggingface/datasets?

Why is it an issue if our branch is temporary ?

He says not; but the branch has no tag yet; does the doc building require the tag? Or just the version number in `__init__.py` or setup.py?

It uses `module.__version__` (i.e. the one defined in `__init__.py`) - no need to have a tag

https://github.com/huggingface/doc-builder/blob/81575cf081964c30ea5fd39450f4820db963f18e/src/doc_builder/commands/build.py#L69

Thanks, @lhoestq.

@mishig25 has manually forced the generation of the docs, that are live for 2.7.0 version: https://huggingface.co/docs/datasets/v2.7.0/en/index 

Cool ! this can be closed then ?

I was waiting for #5250 to be merged to close this.

just to confirm, is there anything I need to do from my side ? Or is everything good here ?
Protect the main branch from inadvertent direct pushes

We have decided to implement a protection mechanism in this repository, so that nobody (not even administrators) can inadvertently push accidentally directly to the main branch.

See context here:
- d7c942228b8dcf4de64b00a3053dce59b335f618

To do:
- [x] Protect main branch
  - Settings > Branches > Branch protection rules > main > Edit
    - [x] Check: Do not allow bypassing the above settings 
      - The above settings will apply to administrators and custom roles with the "bypass branch protections" permission.
    - [x] Additionally, uncheck: Require approvals [under "Require a pull request before merging", which was already checked]
      - Before, we could exceptionally merge a non-approved PR, using Administrator bypass
      - Now that Administrator bypass is no longer possible, we would always need an approval to be able to merge; and pull request authors cannot approve their own pull requests. This could be an inconvenient in some exceptional circumstances when an urgent fix is needed
      - Nevertheless, although it is no longer enforced, it is strongly recommended to merge PRs only if they have at least one approval
- [x] #5250
  - So that direct pushes to main branch are no longer necessary

It seems all the tasks have been addressed, meaning this issue can be closed, no?
Complete doc migration

Reverts huggingface/datasets#5214

Everything is handled on the doc-builder side now 😊

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5248). All of your documentation changes will be reflected on that endpoint.

Thanks for the fix @mishig25.

I guess this is the reason why the docs are not generated for the latest release version 2.7.0? https://huggingface.co/docs/datasets/index
Unable to rename columns in streaming dataset

### Describe the bug

Trying to rename column in a streaming datasets, destroys the features object.

### Steps to reproduce the bug

The following code illustrates the error:

```
from datasets import load_dataset
dataset = load_dataset('mc4', 'en', streaming=True, split='train')
dataset.info.features
# {'text': Value(dtype='string', id=None), 'timestamp': Value(dtype='string', id=None), 'url': Value(dtype='string', id=None)}
dataset = dataset.rename_column("text", "content")
dataset.info.features
# This returned object is now None!
```



### Expected behavior

This should just alter the renamed column.

### Environment info

datasets 2.6.1

Hi @peregilk this bug is directly related to https://github.com/huggingface/datasets/issues/3888, and still not fixed... But I'll try to have a look!

Thanks @alvarobartt. It is great if you are able to fix it, but when reading the explanation it seems like it is possible to work around it.

We also tried keeping the 'info.features' and then adding a modified version back after the remove/rename. Unforutunately that leads to a dataset that is not possible to iterate over.

So if you iterate over the `IterableDataset` as `next(iter(ds))` and then run `rename_columns` when checking that data it will work, but in the end, it's just renaming the column one example/batch at a time, not renaming the column name for all the entries in the dataset, which is the ideal.

@alvarobartt Thanks. My use case was that I wanted to do multiple things, ie removing all unnecessary columns, renaming some valid columns, and then using cast (in my case checking if the audio is not 16K and casting it). It is just convenient to look into the info.features between each of these operations. Alternatively, I will just plan ahead...;) To me it seems like all the operations are working.

Thanks for the advice. It was very useful.

If we know the features before renaming, then we know the features after renaming, so we can pass the new features to the returned dataset in `rename_column` indeed ! If anyone is interested in contributing, feel free to open a PR and I'd be happy to help / give some pointers :)

Sure @lhoestq thanks! I’ll try to work on that

#self-assign
Failed Data Processing upon upload with zip file full of images

I went to autotrain and under image classification arrived where it was time to prepare my dataset. Screenshot below
![image](https://user-images.githubusercontent.com/82735473/201814099-3cc5ff8a-88dc-4f5f-8140-f19560641d83.png)


I chose the method 2 option. I have a csv file with two columns. ~23,000 files. 

I uploaded this and chose the image_relpath, and target columns. 

The image uploader said that I could only upload 10,000 singular images at a time so the 2nd option was to zip the images up and upload a zip archive which I did. 

That all uploaded. 

Now I have the message below. It appears the zip archive does just uncompress on the Hugging Face end? 
What am I missing here?



![image](https://user-images.githubusercontent.com/82735473/201813838-b50dbbbc-34e8-4d73-9c07-12f9e41c62eb.png)


cc @abhishekkrthakur @SBrandeis
Add num_proc to from_csv/generator/json/parquet/text

Allow multiprocessing to from_* methods

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5239). All of your documentation changes will be reflected on that endpoint.

I ended up moving `num_proc` to `AbstractDatasetReader.__init__` :)

Let me know if it sounds good to you now
Make `Version` hashable

Add `__hash__` to the `Version` class to make it hashable (and remove the unneeded methods), as  `Version("0.0.0")` is the default value of `BuilderConfig.version` and the default fields of a dataclass need to be hashable in Python 3.11.

Fix https://github.com/huggingface/datasets/issues/5230

_The documentation is not available anymore as the PR was closed or merged._
Handle ArrowNotImplementedError caused by try_type being Image or Audio in cast

Handle the `ArrowNotImplementedError` thrown when `try_type` is `Image` or `Audio` and the input array cannot be converted to their storage formats.

Reproducer:
```python
from datasets import Dataset
from PIL import Image
import requests

ds = Dataset.from_dict({"image": [Image.open(requests.get("https://upload.wikimedia.org/wikipedia/commons/e/e9/Felis_silvestris_silvestris_small_gradual_decrease_of_quality.png", stream=True).raw)]})
ds.map(lambda x: {"image": True}) # ArrowNotImplementedError 
```

PS: This could also be fixed by raising `TypeError` in `{Image, Audio}.cast_storage` for unsupported types instead of passing the array to `array_cast.`

_The documentation is not available anymore as the PR was closed or merged._

> Not sure how we can have a test that is relevant for this though - feel free to add one if you have ideas

Yes, this was my reasoning for not adding a test. This change is pretty simple, so I think it's OK not to have a test for it.
fix: dataset path should be absolute

cache_file_name depends on dataset's path.

A simple way where this could cause a problem:
```
import os
import datasets


def add_prefix(example):
    example["text"] = "Review: " + example["text"]
    return example


ds = datasets.load_from_disk("a/relative/path")
os.chdir("/tmp")

ds_1 = ds.map(add_prefix)
```

while it may feel that the `chdir` is quite constructed, there are many scenarios when the current working dir can/will change...

Good catch thanks ! Have you tried to use the absolue path in `MemoryMappedTable.__init__` in `table.py`?

I think it can fix issues with relative paths at more levels than just fixing it `load_from_disk`. If it works I think it would be a more robust fix to this issue

@lhoestq right, that actually fixed it indeed. I've pushed the changes (one-liner). lemme know if there's anything else you need for this fix

_The documentation is not available anymore as the PR was closed or merged._
Incompatible dill versions in datasets 2.6.1

### Describe the bug

datasets version 2.6.1 has a dependency on dill<0.3.6. This causes a conflict with dill>=0.3.6 used by multiprocess dependency in datasets 2.6.1

This issue is already fixed in https://github.com/huggingface/datasets/pull/5166/files, but not yet been released. Please release a new version of the datasets library to fix this.

### Steps to reproduce the bug

1. Create requirements.in with only dependency being datasets (or datasets[s3])
2. Run pip-compile
3. The output is as follows:

```
Could not find a version that matches dill<0.3.6,>=0.3.6 (from datasets[s3]==2.6.1->-r requirements.in (line 1))
Tried: 0.2, 0.2, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.2.5, 0.2.5, 0.2.6, 0.2.7, 0.2.7.1, 0.2.8, 0.2.8.1, 0.2.8.2, 0.2.9, 0.3.0, 0.3.1, 0.3.1.1, 0.3.2, 0.3.3, 0.3.3, 0.3.4, 0.3.4, 0.3.5, 0.3.5, 0.3.5.1, 0.3.5.1, 0.3.6, 0.3.6
Skipped pre-versions: 0.1a1, 0.2a1, 0.2a1, 0.2b1, 0.2b1
There are incompatible versions in the resolved dependencies:
  dill<0.3.6 (from datasets[s3]==2.6.1->-r requirements.in (line 1))
  dill>=0.3.6 (from multiprocess==0.70.14->datasets[s3]==2.6.1->-r requirements.in (line 1))
```

### Expected behavior

pip-compile produces requirements.txt without any conflicts

### Environment info

datasets version 2.6.1

Thanks for reporting, @vinaykakade.

We are discussing about making a release early this week.

Please note that in the meantime, in your specific case (as we also pointed out here: https://github.com/huggingface/datasets/issues/5162#issuecomment-1291720293), you can circumvent the issue by pinning `multiprocess` to 0.70.13 version (instead of using latest 0.70.14).

Duplicate of:
- https://github.com/huggingface/datasets/issues/5162

You can also make `pip-compile` work by using the backtracking resolver (instead of the legacy one): https://pip-tools.readthedocs.io/en/latest/#a-note-on-resolvers
```
pip-compile --resolver=backtracking requirements.in
```
This resolver will automatically use `multiprocess` 0.70.13 version.
Using `set_format(type='torch', columns=columns)` makes Array2D/3D columns stop formatting correctly

I have a Dataset with two Features defined as follows:

```
'image': Array3D(dtype="int64", shape=(3, 224, 224)),
'bbox': Array2D(dtype="int64", shape=(512, 4)),
```

On said dataset, if I `dataset.set_format(type='torch')` and then use the dataset in a dataloader, these columns are correctly cast to Tensors of (batch_size, 3, 224, 244) for example.

However, if I `dataset.set_format(type='torch', columns=['image', 'bbox'])` these columns are cast to Lists of tensors and miss the batch size completely (the 3 dimension is the list length).

I'm currently digging through datasets formatting code to try and find out why, but was curious if someone knew an immediate solution for this.

In case others find this, the problem was not with set_format, but my usages of `to_pandas()` and `from_pandas()` which I was using during dataset splitting; somewhere in the chain of converting to and from pandas the `Array2D/Array3D` types get converted to series of `Sequence()` types
Type error when calling `map` over dataset containing 0-d tensors

### Describe the bug

0-dimensional tensors in a dataset lead to `TypeError: iteration over a 0-d array` when calling `map`.  It is easy to generate such tensors by using `.with_format("...")` on the whole dataset.

### Steps to reproduce the bug

```
ds = datasets.Dataset.from_list([{"a": 1}, {"a": 1}]).with_format("torch")
ds.map(None)
```

### Expected behavior

Getting back `ds` without errors.

### Environment info

Python 3.10.8
datasets 2.6.
torch 1.13.0

Hi! 

We could address this by calling `.item()` on such tensors to extract the value, but this would lose us the type, which could lead to storing the generated dataset in a suboptimal format. Considering this, I think the only proper fix would be implementing support for 0-D tensors on Apache Arrow's side (Arrow is the underlying format we use to store datasets on disk/in memory). WDYT @lhoestq?

I think we can just convert the item to a numpy typed scalar using `.numpy()` ?

For example this works:
```python
import numpy as np
import pyarrow as pa

assert pa.array([np.float64(1.0)]).type == pa.float64()
assert pa.array([np.float32(1.0)]).type == pa.float32()
assert pa.array([np.int32(1)]).type == pa.int32()
assert pa.array([np.int64(1)]).type == pa.int64()
```

And therefore it would work the same as for PyTorch N-D Tensors: convert to Numpy Array to keep the type in `_cast_to_python_objects`, then convert to Arrow
datasets.data_files.EmptyDatasetError: The directory at wikisql doesn't contain any data files

### Describe the bug

From these lines:

from datasets import list_datasets, load_dataset
dataset = load_dataset("wikisql","binary")

I get error message:
datasets.data_files.EmptyDatasetError: The directory at wikisql doesn't contain any data files

And yet the 'wikisql' is reported to exist via the list_datasets(). 

Any help appreciated. 



### Steps to reproduce the bug

From these lines:

from datasets import list_datasets, load_dataset
dataset = load_dataset("wikisql","binary")

I get error message:
datasets.data_files.EmptyDatasetError: The directory at wikisql doesn't contain any data files

And yet the 'wikisql' is reported to exist via the list_datasets(). 

Any help appreciated. 


### Expected behavior

Dataset should load. This same code used to work.

### Environment info

Mac OS

Fixed. Please close.

how to fix?i need your help
Q: Memory release when removing the column?

### Describe the bug

How do I release memory when I use methods like `.remove_columns()` or `clear()` in notebooks?
```python
from datasets import load_dataset
common_voice = load_dataset("mozilla-foundation/common_voice_11_0", "ja", use_auth_token=True)
# check memory -> RAM Used (GB):  0.704 / Total (GB)  33.670
common_voice = common_voice.remove_columns(column_names=common_voice.column_names['train'])
common_voice.clear()
# check memory -> RAM Used (GB):  0.705 / Total (GB)  33.670
```

I tried `gc.collect()` but did not help

### Steps to reproduce the bug

1. load dataset
2. remove all the columns
3. check memory is reduced or not


[link to reproduce](https://www.kaggle.com/code/bayartsogtya/huggingface-dataset-memory-issue/notebook?scriptVersionId=110630567)

### Expected behavior

Memory released when I remove the column

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 8.0.0
- Pandas version: 1.3.5

Hi ! Datasets are memory mapped from your disk, i.e. they're not loaded in RAM. This is possible thanks to the Arrow data format.

Therefore the column you remove is not in RAM, so removing it doesn't cause the RAM to decrease.

Thanks for the explanation! @lhoestq 
I wonder since it is memory mapped, can we reduce or remove this memory map?

Yes you can `del common_voice` for example or wait for it to be garbage collected
Add SQL guide

This PR adapts @nateraw's awesome SQL notebook as a guide for the docs!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5223). All of your documentation changes will be reflected on that endpoint.

I think we may want more content on this page that's not SQL related. Some of that content probably already lives in the main `load` docs page, but might be bad to remove major things like csv/pandas from there...WDYT we should do @lhoestq ?

Maybe the main load page can only show one example and redirect to this page for more details ?

We can do the same for pandas stuff: have one example in load, and redirect to this page for more details

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5223). All of your documentation changes will be reflected on that endpoint.
HuggingFace website is incorrectly reporting that my datasets are pickled

### Describe the bug

HuggingFace is incorrectly reporting that my datasets are pickled. They are not picked, they are simple ZIP files containing PNG images.

Hopefully this is the right location to report this bug.

### Steps to reproduce the bug

Inspect my dataset respository here: https://huggingface.co/datasets/ProGamerGov/StableDiffusion-v1-5-Regularization-Images

### Expected behavior

They should not be reported as being pickled.

### Environment info

N/A

cc @McPatate maybe you know what's happening ?

Yes I think I know what is happening. We check in zips for pickles, and the UI must display the pickle jar when a scan has an associated list of imports, even when empty.
~I'll fix ASAP !~

>  I'll fix ASAP !

Actually I'd rather leave it like that for now, as it indicates that we checked for pickles and nothing dangerous appeared :)

Closing the issue with the typical "feature not a bug"
Cannot push 

### Describe the bug

I am facing the issue when I try to push the tar.gz file around 11G to HUB.

```
(venv) ╭─laptop@laptop ~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ du -sh *
4.0K	README.md
 13G	data
516K	test.jsonl
 18M	train.jsonl
4.0K	ulaanbal_v0.py
 11G	ulaanbal_v0.tar.gz
452K	validation.jsonl

(venv) ╭─laptop@laptop~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ git add ulaanbal_v0.tar.gz && git commit -m 'large version'

(venv) ╭─laptop@laptop ~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ git push 
EOFoading LFS objects:   0% (0/1), 0 B | 0 B/s                                                                                                                                                      
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
error: failed to push some refs to 'https://huggingface.co/datasets/bayartsogt/ulaanbal_v0'
```

I have already tried pushing a small version of this and it was working fine. So my guess it is probably because of the big file.

Following I run before the commit:
```
╰─$ git lfs install
╰─$ huggingface-cli lfs-enable-largefiles .
```

### Steps to reproduce the bug

Create a private dataset on huggingface and push 12G tar.gz file

### Expected behavior

To be pushed with no issue

### Environment info


- `datasets` version: 2.6.1
- Platform: Darwin-21.6.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 10.0.0
- Pandas version: 1.3.5


Did you run `huggingface-cli lfs-enable-largefiles` before committing or before adding ? Maybe you can try before adding

Anyway I'd encourage you to split your data into several TAR archives if possible, this way the dataset can loaded faster using multiprocessing (by giving each process a subset of shards to process)

@lhoestq 
Thanks for the help!
> Maybe you can try before adding

It did not help

But I totally got your point about split into multiple TAR archives. It really helped!
Implicit type conversion of lists in to_pandas

### Describe the bug

```
ds = Dataset.from_list([{'a':[1,2,3]}])
ds.to_pandas().a.values[0]
```
Results in `array([1, 2, 3])`  -- a rather unexpected conversion of types which made downstream tools expecting lists not happy.

### Steps to reproduce the bug

See snippet

### Expected behavior

Keep the original type

### Environment info

datasets 2.6.1
python 3.8.10

I think this behavior comes from PyArrow:
```python
import pyarrow as pa
t = pa.table({"a": [[0]]})
t.to_pandas().a.values[0]
# array([0])
```

I believe this has to do with zero-copy: you can get a pandas DataFrame without copying the buffers from arrow, and therefore end up with numpy arrays.

That's interesting, I guess not much to do here then.
Delta Tables usage using Datasets Library

### Feature request

Adding compatibility of Datasets library with Delta Format. Elevating the utilities of Datasets library from Machine Learning Scope to Data Engineering Scope as well.

### Motivation

We know datasets library can absorb csv, json, parquet, etc. file formats but it would be great if Datasets library could work with Delta Tables (with delta format) as it has different features such as time travelling, layout optimization, query performance, aids in Data Engineering.

This will help and enhance Datasets library from Machine Learning utility to Data Engineering utilities and expand horizons thereafter. I am totally using Datasets library in all my usecases and as my role expands so does the work, compatibility with Datasets library is something I don't want to lose.

### Your contribution

Would love to work on this feature, even if this has to picked up from scratch, including design paradigms and patterns.
I have basic idea about Delta Live Tables, would brush it easily for this feature.

Hi ! Interesting :) Can you provide concrete examples of cases where it can be useful ?

Few example blogs and posts that might help on this - 

1. https://hevodata.com/learn/databricks-delta-tables/
2. https://docs.databricks.com/delta/index.html

Basically, we are looking at utility of Datasets library with Delta Lake Tables.


`datasets` can already read/write from parquet from/to a cloud storage using fsspec, if I understand correctly it's should be possible to load parquet files as delat lake tables no ? :) Or is there someting missing ?

@lhoestq Per my understanding, delta lake table is a bunch of paruqet files together with the meta to support ACID. For example file 1 contains v0.1 of record A while file 2 contains v0.2 of record A. I am assuming the Hugging face dataset would delegate the read/write delta table to 3rd party lib, maybe pyarrow. Correct me if I was wrong @reichenbch 

And I am assuming, people are asking the versioning of Hugging face datasets. But I am assuming Hugging face delegate this function to github and it is not the key requirement for Public Data set. It actually the key function of ML Ops, I am not sure whether hugging face would like expand to that area.
Delta Tables usage using Datasets Library

### Feature request

Adding compatibility of Datasets library with Delta Format. Elevating the utilities of Datasets library from Machine Learning Scope to Data Engineering Scope as well.

### Motivation

We know datasets library can absorb csv, json, parquet, etc. file formats but it would be great if Datasets library could work with Delta Tables (with delta format) as it has different features such as time travelling, layout optimization, query performance, aids in Data Engineering.

This will help and enhance Datasets library from Machine Learning utility to Data Engineering utilities and expand horizons thereafter. I am totally using Datasets library in all my usecases and as my role expands so does the work, compatibility with Datasets library is something I don't want to lose.

### Your contribution

Would love to work on this feature, even if this has to picked up from scratch, including design paradigms and patterns.
I have basic idea about Delta Live Tables, would brush it easily for this feature.
save_elasticsearch_index

Hi,

I am new to Dataset and elasticsearch. I was wondering is there any equivalent approach to save elasticsearch index as of save_faiss_index locally for later use, to remove the need to re-index a dataset?

Hi ! I think there exist tools to dump and reload an index in your elastic search but I'm not super familiar with it.

Anyway after reloading an index in elastic search you can call `ds.load_elasticsearch_index` which will connect the index to the dataset without re-indexing
Fix CI require_beam maximum compatible dill version

A previous commit to main branch introduced an additional requirement on maximum compatible `dill` version with `apache-beam` in our CI `require_beam`:
- d7c942228b8dcf4de64b00a3053dce59b335f618
- ec222b220b79f10c8d7b015769f0999b15959feb

This PR fixes the maximum compatible `dill` version with `apache-beam`, which is <0.3.2 (and not 0.3.6): https://github.com/apache/beam/blob/v2.42.0/sdks/python/setup.py#L219

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5212). All of your documentation changes will be reflected on that endpoint.
Update Overview.ipynb google colab

- removed metrics stuff
- added image example
- added audio example (with ffmpeg instructions)
- updated the "add a new dataset" section

_The documentation is not available anymore as the PR was closed or merged._

WDYT @albertvillanova ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5211). All of your documentation changes will be reflected on that endpoint.
Tweak readme

Tweaked some paragraphs mentioning the modalities we support + added a paragraph on security

_The documentation is not available anymore as the PR was closed or merged._

Nit: We should also update the `Disclaimers` section to let the dataset owners know they should use Hub discussions rather than GH issues for removal requests/updates

Updated the disclaimers section, thanks !

Does it sound good to you @albertvillanova ?
Use logging instead of printing to console

### Describe the bug

Some logs ([here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L778), [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L786), and [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L830)) generated by the `DatasetBuilder` are printed to the console instead of passed to `datasets` logger.


### Steps to reproduce the bug

```python
>> import datasets

>> datasets.load_dataset("some-dataset")

Downloading and preparing dataset csv/data to <path>...
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7729.06it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 527.23it/s]
Dataset csv downloaded and prepared to <path>. Subsequent calls will reuse this data.
```

### Expected behavior

The logs should not be printed to the console directly but passed to the logger so that the user can redirect them wherever he wants.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-x86_64-i386-64bit
- Python version: 3.9.15
- PyArrow version: 10.0.0
- Pandas version: 1.5.1

Actually upon closer inspection, it is documented in the code that this behavior is intentional, so I'll close this.