Minimartzz/github-issues huggingface dataset:
Type error when calling `map` over dataset containing 0-d tensors
State: closed

### Describe the bug

0-dimensional tensors in a dataset lead to `TypeError: iteration over a 0-d array` when calling `map`.  It is easy to generate such tensors by using `.with_format("...")` on the whole dataset.

### Steps to reproduce the bug

```
ds = datasets.Dataset.from_list([{"a": 1}, {"a": 1}]).with_format("torch")
ds.map(None)
```

### Expected behavior

Getting back `ds` without errors.

### Environment info

Python 3.10.8
datasets 2.6.
torch 1.13.0

Comments:
Hi! 

We could address this by calling `.item()` on such tensors to extract the value, but this would lose us the type, which could lead to storing the generated dataset in a suboptimal format. Considering this, I think the only proper fix would be implementing support for 0-D tensors on Apache Arrow's side (Arrow is the underlying format we use to store datasets on disk/in memory). WDYT @lhoestq?

I think we can just convert the item to a numpy typed scalar using `.numpy()` ?

For example this works:
```python
import numpy as np
import pyarrow as pa

assert pa.array([np.float64(1.0)]).type == pa.float64()
assert pa.array([np.float32(1.0)]).type == pa.float32()
assert pa.array([np.int32(1)]).type == pa.int32()
assert pa.array([np.int64(1)]).type == pa.int64()
```

And therefore it would work the same as for PyTorch N-D Tensors: convert to Numpy Array to keep the type in `_cast_to_python_objects`, then convert to Arrow
datasets.data_files.EmptyDatasetError: The directory at wikisql doesn't contain any data files
State: closed

### Describe the bug

From these lines:

from datasets import list_datasets, load_dataset
dataset = load_dataset("wikisql","binary")

I get error message:
datasets.data_files.EmptyDatasetError: The directory at wikisql doesn't contain any data files

And yet the 'wikisql' is reported to exist via the list_datasets(). 

Any help appreciated. 



### Steps to reproduce the bug

From these lines:

from datasets import list_datasets, load_dataset
dataset = load_dataset("wikisql","binary")

I get error message:
datasets.data_files.EmptyDatasetError: The directory at wikisql doesn't contain any data files

And yet the 'wikisql' is reported to exist via the list_datasets(). 

Any help appreciated. 


### Expected behavior

Dataset should load. This same code used to work.

### Environment info

Mac OS

Comments:
Fixed. Please close.

how to fix?i need your help
Q: Memory release when removing the column?
State: closed

### Describe the bug

How do I release memory when I use methods like `.remove_columns()` or `clear()` in notebooks?
```python
from datasets import load_dataset
common_voice = load_dataset("mozilla-foundation/common_voice_11_0", "ja", use_auth_token=True)
# check memory -> RAM Used (GB):  0.704 / Total (GB)  33.670
common_voice = common_voice.remove_columns(column_names=common_voice.column_names['train'])
common_voice.clear()
# check memory -> RAM Used (GB):  0.705 / Total (GB)  33.670
```

I tried `gc.collect()` but did not help

### Steps to reproduce the bug

1. load dataset
2. remove all the columns
3. check memory is reduced or not


[link to reproduce](https://www.kaggle.com/code/bayartsogtya/huggingface-dataset-memory-issue/notebook?scriptVersionId=110630567)

### Expected behavior

Memory released when I remove the column

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 8.0.0
- Pandas version: 1.3.5

Comments:
Hi ! Datasets are memory mapped from your disk, i.e. they're not loaded in RAM. This is possible thanks to the Arrow data format.

Therefore the column you remove is not in RAM, so removing it doesn't cause the RAM to decrease.

Thanks for the explanation! @lhoestq 
I wonder since it is memory mapped, can we reduce or remove this memory map?

Yes you can `del common_voice` for example or wait for it to be garbage collected
Add SQL guide
State: closed

This PR adapts @nateraw's awesome SQL notebook as a guide for the docs!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5223). All of your documentation changes will be reflected on that endpoint.

I think we may want more content on this page that's not SQL related. Some of that content probably already lives in the main `load` docs page, but might be bad to remove major things like csv/pandas from there...WDYT we should do @lhoestq ?

Maybe the main load page can only show one example and redirect to this page for more details ?

We can do the same for pandas stuff: have one example in load, and redirect to this page for more details

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5223). All of your documentation changes will be reflected on that endpoint.
HuggingFace website is incorrectly reporting that my datasets are pickled
State: closed

### Describe the bug

HuggingFace is incorrectly reporting that my datasets are pickled. They are not picked, they are simple ZIP files containing PNG images.

Hopefully this is the right location to report this bug.

### Steps to reproduce the bug

Inspect my dataset respository here: https://huggingface.co/datasets/ProGamerGov/StableDiffusion-v1-5-Regularization-Images

### Expected behavior

They should not be reported as being pickled.

### Environment info

N/A

Comments:
cc @McPatate maybe you know what's happening ?

Yes I think I know what is happening. We check in zips for pickles, and the UI must display the pickle jar when a scan has an associated list of imports, even when empty.
~I'll fix ASAP !~

>  I'll fix ASAP !

Actually I'd rather leave it like that for now, as it indicates that we checked for pickles and nothing dangerous appeared :)

Closing the issue with the typical "feature not a bug"
Cannot push 
State: closed

### Describe the bug

I am facing the issue when I try to push the tar.gz file around 11G to HUB.

```
(venv) ╭─laptop@laptop ~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ du -sh *
4.0K	README.md
 13G	data
516K	test.jsonl
 18M	train.jsonl
4.0K	ulaanbal_v0.py
 11G	ulaanbal_v0.tar.gz
452K	validation.jsonl

(venv) ╭─laptop@laptop~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ git add ulaanbal_v0.tar.gz && git commit -m 'large version'

(venv) ╭─laptop@laptop ~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ git push 
EOFoading LFS objects:   0% (0/1), 0 B | 0 B/s                                                                                                                                                      
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
error: failed to push some refs to 'https://huggingface.co/datasets/bayartsogt/ulaanbal_v0'
```

I have already tried pushing a small version of this and it was working fine. So my guess it is probably because of the big file.

Following I run before the commit:
```
╰─$ git lfs install
╰─$ huggingface-cli lfs-enable-largefiles .
```

### Steps to reproduce the bug

Create a private dataset on huggingface and push 12G tar.gz file

### Expected behavior

To be pushed with no issue

### Environment info


- `datasets` version: 2.6.1
- Platform: Darwin-21.6.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 10.0.0
- Pandas version: 1.3.5


Comments:
Did you run `huggingface-cli lfs-enable-largefiles` before committing or before adding ? Maybe you can try before adding

Anyway I'd encourage you to split your data into several TAR archives if possible, this way the dataset can loaded faster using multiprocessing (by giving each process a subset of shards to process)

@lhoestq 
Thanks for the help!
> Maybe you can try before adding

It did not help

But I totally got your point about split into multiple TAR archives. It really helped!
Implicit type conversion of lists in to_pandas
State: closed

### Describe the bug

```
ds = Dataset.from_list([{'a':[1,2,3]}])
ds.to_pandas().a.values[0]
```
Results in `array([1, 2, 3])`  -- a rather unexpected conversion of types which made downstream tools expecting lists not happy.

### Steps to reproduce the bug

See snippet

### Expected behavior

Keep the original type

### Environment info

datasets 2.6.1
python 3.8.10

Comments:
I think this behavior comes from PyArrow:
```python
import pyarrow as pa
t = pa.table({"a": [[0]]})
t.to_pandas().a.values[0]
# array([0])
```

I believe this has to do with zero-copy: you can get a pandas DataFrame without copying the buffers from arrow, and therefore end up with numpy arrays.

That's interesting, I guess not much to do here then.
Delta Tables usage using Datasets Library
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Adding compatibility of Datasets library with Delta Format. Elevating the utilities of Datasets library from Machine Learning Scope to Data Engineering Scope as well.

### Motivation

We know datasets library can absorb csv, json, parquet, etc. file formats but it would be great if Datasets library could work with Delta Tables (with delta format) as it has different features such as time travelling, layout optimization, query performance, aids in Data Engineering.

This will help and enhance Datasets library from Machine Learning utility to Data Engineering utilities and expand horizons thereafter. I am totally using Datasets library in all my usecases and as my role expands so does the work, compatibility with Datasets library is something I don't want to lose.

### Your contribution

Would love to work on this feature, even if this has to picked up from scratch, including design paradigms and patterns.
I have basic idea about Delta Live Tables, would brush it easily for this feature.

Comments:
Hi ! Interesting :) Can you provide concrete examples of cases where it can be useful ?

Few example blogs and posts that might help on this - 

1. https://hevodata.com/learn/databricks-delta-tables/
2. https://docs.databricks.com/delta/index.html

Basically, we are looking at utility of Datasets library with Delta Lake Tables.


`datasets` can already read/write from parquet from/to a cloud storage using fsspec, if I understand correctly it's should be possible to load parquet files as delat lake tables no ? :) Or is there someting missing ?

@lhoestq Per my understanding, delta lake table is a bunch of paruqet files together with the meta to support ACID. For example file 1 contains v0.1 of record A while file 2 contains v0.2 of record A. I am assuming the Hugging face dataset would delegate the read/write delta table to 3rd party lib, maybe pyarrow. Correct me if I was wrong @reichenbch 

And I am assuming, people are asking the versioning of Hugging face datasets. But I am assuming Hugging face delegate this function to github and it is not the key requirement for Public Data set. It actually the key function of ML Ops, I am not sure whether hugging face would like expand to that area.
Delta Tables usage using Datasets Library
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

Adding compatibility of Datasets library with Delta Format. Elevating the utilities of Datasets library from Machine Learning Scope to Data Engineering Scope as well.

### Motivation

We know datasets library can absorb csv, json, parquet, etc. file formats but it would be great if Datasets library could work with Delta Tables (with delta format) as it has different features such as time travelling, layout optimization, query performance, aids in Data Engineering.

This will help and enhance Datasets library from Machine Learning utility to Data Engineering utilities and expand horizons thereafter. I am totally using Datasets library in all my usecases and as my role expands so does the work, compatibility with Datasets library is something I don't want to lose.

### Your contribution

Would love to work on this feature, even if this has to picked up from scratch, including design paradigms and patterns.
I have basic idea about Delta Live Tables, would brush it easily for this feature.

Comments:
save_elasticsearch_index
State: open

Hi,

I am new to Dataset and elasticsearch. I was wondering is there any equivalent approach to save elasticsearch index as of save_faiss_index locally for later use, to remove the need to re-index a dataset?

Comments:
Hi ! I think there exist tools to dump and reload an index in your elastic search but I'm not super familiar with it.

Anyway after reloading an index in elastic search you can call `ds.load_elasticsearch_index` which will connect the index to the dataset without re-indexing
Fix CI require_beam maximum compatible dill version
State: closed

A previous commit to main branch introduced an additional requirement on maximum compatible `dill` version with `apache-beam` in our CI `require_beam`:
- d7c942228b8dcf4de64b00a3053dce59b335f618
- ec222b220b79f10c8d7b015769f0999b15959feb

This PR fixes the maximum compatible `dill` version with `apache-beam`, which is <0.3.2 (and not 0.3.6): https://github.com/apache/beam/blob/v2.42.0/sdks/python/setup.py#L219

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5212). All of your documentation changes will be reflected on that endpoint.
Update Overview.ipynb google colab
State: closed

- removed metrics stuff
- added image example
- added audio example (with ffmpeg instructions)
- updated the "add a new dataset" section

Comments:
_The documentation is not available anymore as the PR was closed or merged._

WDYT @albertvillanova ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5211). All of your documentation changes will be reflected on that endpoint.
Tweak readme
State: closed

Tweaked some paragraphs mentioning the modalities we support + added a paragraph on security

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Nit: We should also update the `Disclaimers` section to let the dataset owners know they should use Hub discussions rather than GH issues for removal requests/updates

Updated the disclaimers section, thanks !

Does it sound good to you @albertvillanova ?
Use logging instead of printing to console
State: closed

### Describe the bug

Some logs ([here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L778), [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L786), and [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L830)) generated by the `DatasetBuilder` are printed to the console instead of passed to `datasets` logger.


### Steps to reproduce the bug

```python
>> import datasets

>> datasets.load_dataset("some-dataset")

Downloading and preparing dataset csv/data to <path>...
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7729.06it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 527.23it/s]
Dataset csv downloaded and prepared to <path>. Subsequent calls will reuse this data.
```

### Expected behavior

The logs should not be printed to the console directly but passed to the logger so that the user can redirect them wherever he wants.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-x86_64-i386-64bit
- Python version: 3.9.15
- PyArrow version: 10.0.0
- Pandas version: 1.5.1

Comments:
Actually upon closer inspection, it is documented in the code that this behavior is intentional, so I'll close this.
`push_to_hub` not propagating `token` through `DownloadConfig`
State: closed

### Describe the bug

When trying to upload a new 🤗 Dataset to the Hub via Python, and providing the `token` as a parameter to the `Dataset.push_to_hub` function, it just works for the first time, assuming that the dataset didn't exist before.

But when trying to run `Dataset.push_to_hub` again over the same dataset, instead of updating it, it throws a `ConnectionError` when trying to retrieve the `README.md` that may contain some metadata about the dataset, so as to also update it, but since the `token` is not propagated, the `DownloadConfig` provided to the `datasets.utils.file_utils.get_from_cache` function doesn't contain the `use_auth_token` value set to `token`, it's just using the default one which is None/False.

So on, when uploading a dataset via Python with `push_to_hub` with the `token` as a parameter with the HuggingFace API Token as value, it can just be uploaded when the dataset is new, otherwise it fails with to `ConnectionError` due to the `token` not being propagated as `use_auth_token`.

### Steps to reproduce the bug

Let's create a new dataset in our HF account via Python as:

```python
from datasets import Dataset

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
ds = Dataset.from_dict(data)
ds.push_to_hub(repo_id=<HF_USERNAME>/<HF_DATASET>, private=private, token=<HF_TOKEN_HERE>)
```

When we create the `Dataset` for the first time it works and there are no issues, but when trying to actually upload a new version of the same dataset (same name under the same username), we encounter the following issue:

```python
from datasets import Dataset

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
ds = Dataset.from_dict(data)
ds.push_to_hub(repo_id=<HF_USERNAME>/<HF_DATASET>, private=private, token=<HF_TOKEN_HERE>)
>>> ConnectionError: Couldn't reach https://huggingface.co/datasets/alvarobartt/demo/resolve/main/README.md (ConnectionError('Unauthorized for URL https://huggingface.co/datasets/<HF_USERNAME>/<HF_DATASET>/resolve/main/README.md. Please use the parameter `use_auth_token=True` after logging in with `huggingface-cli login`'))
```

### Expected behavior

Ideally, the `token` parameter provided to `push_to_hub` should be propagated and used to download the `README.md` when trying to update a `Dataset`, instead of throwing that exception, so that the authentication can be done directly through code without running `huggingface-cli login`as mentioned at https://huggingface.co/docs/datasets/upload_dataset#upload-with-python.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-arm64-arm-64bit
- Python version: 3.10.8
- PyArrow version: 10.0.0
- Pandas version: 1.5.1

Comments:
#self-assign

@lhoestq can you close this issue as part of the recent #5205 merge? Thanks 🤗 

Thank you :)
Do not sort splits in dataset info
State: closed

I suggest not to sort splits by their names in dataset_info in README so that they are displayed in the order specified in the loading script. Otherwise `test` split is displayed first, see this repo: https://huggingface.co/datasets/paws
What do you think?

But I added sorting in tests to fix CI (for the same dataset).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

It would be coherent with https://github.com/huggingface/datasets-server/issues/614#issuecomment-1290534153

I think we started working on this issue nearly at the same time... :sweat_smile: 
- CI was fixed with this: https://huggingface.co/datasets/paws/discussions/1

Related issue:
- #5202

@albertvillanova yeah I noticed it right after the PR :smile: thank you! the fix of the dataset info yaml fixes tests on CI, but in general order of splits in yaml influences the order in which they are displayed in the viewer, if I understand it correctly. So I suggest not to sort splits in yaml initially to avoid this for other datasets in the future. I think [this change](https://github.com/huggingface/datasets/pull/5201/files#diff-198ba4fdf2f94cb3e1aba8a0170a43b08d4ab5636d682374321c5a383a8be24dR571) should work for it. 

Changes to tests here maybe can be reverted considering that order in yaml now corresponds to the one in tests, thanks to your change in the dataset info.

Hehe, @polinaeterna, we make comments nearly at the same time as well... :laughing:
Some links to canonical datasets in the docs are outdated
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

As we don't have canonical datasets in the github repo anymore, some old links to them doesn't work. I don't know how many of them are there, I found link to SuperGlue here: https://huggingface.co/docs/datasets/dataset_script#multiple-configurations, probably there are more of them. These links should be replaced by links to the corresponding datasets on the Hub. 

Comments:
Thanks for catching this, I can go through the docs and replace the links to their corresponding datasets on the Hub!
[zstd] Use max window log size
State: open

ZstdDecompressor has a parameter `max_window_size` to limit max memory usage when decompressing zstd files. The default `max_window_size` is not enough when files are compressed by `zstd --ultra` flags.

Change `max_window_size` to the zstd's max window size. NOTE, the `zstd.WINDOWLOG_MAX` is the log_2 value of the max window size.

Comments:
@albertvillanova Please take a review.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5197). All of your documentation changes will be reflected on that endpoint.