NyssaW/github-issues huggingface dataset:
HuggingFace website is incorrectly reporting that my datasets are pickled

### Describe the bug

HuggingFace is incorrectly reporting that my datasets are pickled. They are not picked, they are simple ZIP files containing PNG images.

Hopefully this is the right location to report this bug.

### Steps to reproduce the bug

Inspect my dataset respository here: https://huggingface.co/datasets/ProGamerGov/StableDiffusion-v1-5-Regularization-Images

### Expected behavior

They should not be reported as being pickled.

### Environment info

N/A

Comments:
cc @McPatate maybe you know what's happening ?

Yes I think I know what is happening. We check in zips for pickles, and the UI must display the pickle jar when a scan has an associated list of imports, even when empty.
~I'll fix ASAP !~

>  I'll fix ASAP !

Actually I'd rather leave it like that for now, as it indicates that we checked for pickles and nothing dangerous appeared :)

Closing the issue with the typical "feature not a bug"
Cannot push 

### Describe the bug

I am facing the issue when I try to push the tar.gz file around 11G to HUB.

```
(venv) ╭─laptop@laptop ~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ du -sh *
4.0K	README.md
 13G	data
516K	test.jsonl
 18M	train.jsonl
4.0K	ulaanbal_v0.py
 11G	ulaanbal_v0.tar.gz
452K	validation.jsonl

(venv) ╭─laptop@laptop~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ git add ulaanbal_v0.tar.gz && git commit -m 'large version'

(venv) ╭─laptop@laptop ~/PersonalProjects/data/ulaanbal_v0 ‹main●› 
╰─$ git push 
EOFoading LFS objects:   0% (0/1), 0 B | 0 B/s                                                                                                                                                      
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
error: failed to push some refs to 'https://huggingface.co/datasets/bayartsogt/ulaanbal_v0'
```

I have already tried pushing a small version of this and it was working fine. So my guess it is probably because of the big file.

Following I run before the commit:
```
╰─$ git lfs install
╰─$ huggingface-cli lfs-enable-largefiles .
```

### Steps to reproduce the bug

Create a private dataset on huggingface and push 12G tar.gz file

### Expected behavior

To be pushed with no issue

### Environment info


- `datasets` version: 2.6.1
- Platform: Darwin-21.6.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 10.0.0
- Pandas version: 1.3.5


Comments:
Did you run `huggingface-cli lfs-enable-largefiles` before committing or before adding ? Maybe you can try before adding

Anyway I'd encourage you to split your data into several TAR archives if possible, this way the dataset can loaded faster using multiprocessing (by giving each process a subset of shards to process)

@lhoestq 
Thanks for the help!
> Maybe you can try before adding

It did not help

But I totally got your point about split into multiple TAR archives. It really helped!
Implicit type conversion of lists in to_pandas

### Describe the bug

```
ds = Dataset.from_list([{'a':[1,2,3]}])
ds.to_pandas().a.values[0]
```
Results in `array([1, 2, 3])`  -- a rather unexpected conversion of types which made downstream tools expecting lists not happy.

### Steps to reproduce the bug

See snippet

### Expected behavior

Keep the original type

### Environment info

datasets 2.6.1
python 3.8.10

Comments:
I think this behavior comes from PyArrow:
```python
import pyarrow as pa
t = pa.table({"a": [[0]]})
t.to_pandas().a.values[0]
# array([0])
```

I believe this has to do with zero-copy: you can get a pandas DataFrame without copying the buffers from arrow, and therefore end up with numpy arrays.

That's interesting, I guess not much to do here then.
Delta Tables usage using Datasets Library

### Feature request

Adding compatibility of Datasets library with Delta Format. Elevating the utilities of Datasets library from Machine Learning Scope to Data Engineering Scope as well.

### Motivation

We know datasets library can absorb csv, json, parquet, etc. file formats but it would be great if Datasets library could work with Delta Tables (with delta format) as it has different features such as time travelling, layout optimization, query performance, aids in Data Engineering.

This will help and enhance Datasets library from Machine Learning utility to Data Engineering utilities and expand horizons thereafter. I am totally using Datasets library in all my usecases and as my role expands so does the work, compatibility with Datasets library is something I don't want to lose.

### Your contribution

Would love to work on this feature, even if this has to picked up from scratch, including design paradigms and patterns.
I have basic idea about Delta Live Tables, would brush it easily for this feature.

Comments:
Hi ! Interesting :) Can you provide concrete examples of cases where it can be useful ?

Few example blogs and posts that might help on this - 

1. https://hevodata.com/learn/databricks-delta-tables/
2. https://docs.databricks.com/delta/index.html

Basically, we are looking at utility of Datasets library with Delta Lake Tables.


`datasets` can already read/write from parquet from/to a cloud storage using fsspec, if I understand correctly it's should be possible to load parquet files as delat lake tables no ? :) Or is there someting missing ?

@lhoestq Per my understanding, delta lake table is a bunch of paruqet files together with the meta to support ACID. For example file 1 contains v0.1 of record A while file 2 contains v0.2 of record A. I am assuming the Hugging face dataset would delegate the read/write delta table to 3rd party lib, maybe pyarrow. Correct me if I was wrong @reichenbch 

And I am assuming, people are asking the versioning of Hugging face datasets. But I am assuming Hugging face delegate this function to github and it is not the key requirement for Public Data set. It actually the key function of ML Ops, I am not sure whether hugging face would like expand to that area.
Delta Tables usage using Datasets Library

### Feature request

Adding compatibility of Datasets library with Delta Format. Elevating the utilities of Datasets library from Machine Learning Scope to Data Engineering Scope as well.

### Motivation

We know datasets library can absorb csv, json, parquet, etc. file formats but it would be great if Datasets library could work with Delta Tables (with delta format) as it has different features such as time travelling, layout optimization, query performance, aids in Data Engineering.

This will help and enhance Datasets library from Machine Learning utility to Data Engineering utilities and expand horizons thereafter. I am totally using Datasets library in all my usecases and as my role expands so does the work, compatibility with Datasets library is something I don't want to lose.

### Your contribution

Would love to work on this feature, even if this has to picked up from scratch, including design paradigms and patterns.
I have basic idea about Delta Live Tables, would brush it easily for this feature.

Comments:
save_elasticsearch_index

Hi,

I am new to Dataset and elasticsearch. I was wondering is there any equivalent approach to save elasticsearch index as of save_faiss_index locally for later use, to remove the need to re-index a dataset?

Comments:
Hi ! I think there exist tools to dump and reload an index in your elastic search but I'm not super familiar with it.

Anyway after reloading an index in elastic search you can call `ds.load_elasticsearch_index` which will connect the index to the dataset without re-indexing
Fix CI require_beam maximum compatible dill version

A previous commit to main branch introduced an additional requirement on maximum compatible `dill` version with `apache-beam` in our CI `require_beam`:
- d7c942228b8dcf4de64b00a3053dce59b335f618
- ec222b220b79f10c8d7b015769f0999b15959feb

This PR fixes the maximum compatible `dill` version with `apache-beam`, which is <0.3.2 (and not 0.3.6): https://github.com/apache/beam/blob/v2.42.0/sdks/python/setup.py#L219

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5212). All of your documentation changes will be reflected on that endpoint.
Update Overview.ipynb google colab

- removed metrics stuff
- added image example
- added audio example (with ffmpeg instructions)
- updated the "add a new dataset" section

Comments:
_The documentation is not available anymore as the PR was closed or merged._

WDYT @albertvillanova ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5211). All of your documentation changes will be reflected on that endpoint.
Tweak readme

Tweaked some paragraphs mentioning the modalities we support + added a paragraph on security

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Nit: We should also update the `Disclaimers` section to let the dataset owners know they should use Hub discussions rather than GH issues for removal requests/updates

Updated the disclaimers section, thanks !

Does it sound good to you @albertvillanova ?
Use logging instead of printing to console

### Describe the bug

Some logs ([here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L778), [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L786), and [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L830)) generated by the `DatasetBuilder` are printed to the console instead of passed to `datasets` logger.


### Steps to reproduce the bug

```python
>> import datasets

>> datasets.load_dataset("some-dataset")

Downloading and preparing dataset csv/data to <path>...
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7729.06it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 527.23it/s]
Dataset csv downloaded and prepared to <path>. Subsequent calls will reuse this data.
```

### Expected behavior

The logs should not be printed to the console directly but passed to the logger so that the user can redirect them wherever he wants.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-x86_64-i386-64bit
- Python version: 3.9.15
- PyArrow version: 10.0.0
- Pandas version: 1.5.1

Comments:
Actually upon closer inspection, it is documented in the code that this behavior is intentional, so I'll close this.
`push_to_hub` not propagating `token` through `DownloadConfig`

### Describe the bug

When trying to upload a new 🤗 Dataset to the Hub via Python, and providing the `token` as a parameter to the `Dataset.push_to_hub` function, it just works for the first time, assuming that the dataset didn't exist before.

But when trying to run `Dataset.push_to_hub` again over the same dataset, instead of updating it, it throws a `ConnectionError` when trying to retrieve the `README.md` that may contain some metadata about the dataset, so as to also update it, but since the `token` is not propagated, the `DownloadConfig` provided to the `datasets.utils.file_utils.get_from_cache` function doesn't contain the `use_auth_token` value set to `token`, it's just using the default one which is None/False.

So on, when uploading a dataset via Python with `push_to_hub` with the `token` as a parameter with the HuggingFace API Token as value, it can just be uploaded when the dataset is new, otherwise it fails with to `ConnectionError` due to the `token` not being propagated as `use_auth_token`.

### Steps to reproduce the bug

Let's create a new dataset in our HF account via Python as:

```python
from datasets import Dataset

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
ds = Dataset.from_dict(data)
ds.push_to_hub(repo_id=<HF_USERNAME>/<HF_DATASET>, private=private, token=<HF_TOKEN_HERE>)
```

When we create the `Dataset` for the first time it works and there are no issues, but when trying to actually upload a new version of the same dataset (same name under the same username), we encounter the following issue:

```python
from datasets import Dataset

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
ds = Dataset.from_dict(data)
ds.push_to_hub(repo_id=<HF_USERNAME>/<HF_DATASET>, private=private, token=<HF_TOKEN_HERE>)
>>> ConnectionError: Couldn't reach https://huggingface.co/datasets/alvarobartt/demo/resolve/main/README.md (ConnectionError('Unauthorized for URL https://huggingface.co/datasets/<HF_USERNAME>/<HF_DATASET>/resolve/main/README.md. Please use the parameter `use_auth_token=True` after logging in with `huggingface-cli login`'))
```

### Expected behavior

Ideally, the `token` parameter provided to `push_to_hub` should be propagated and used to download the `README.md` when trying to update a `Dataset`, instead of throwing that exception, so that the authentication can be done directly through code without running `huggingface-cli login`as mentioned at https://huggingface.co/docs/datasets/upload_dataset#upload-with-python.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-arm64-arm-64bit
- Python version: 3.10.8
- PyArrow version: 10.0.0
- Pandas version: 1.5.1

Comments:
#self-assign

@lhoestq can you close this issue as part of the recent #5205 merge? Thanks 🤗 

Thank you :)
Do not sort splits in dataset info

I suggest not to sort splits by their names in dataset_info in README so that they are displayed in the order specified in the loading script. Otherwise `test` split is displayed first, see this repo: https://huggingface.co/datasets/paws
What do you think?

But I added sorting in tests to fix CI (for the same dataset).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

It would be coherent with https://github.com/huggingface/datasets-server/issues/614#issuecomment-1290534153

I think we started working on this issue nearly at the same time... :sweat_smile: 
- CI was fixed with this: https://huggingface.co/datasets/paws/discussions/1

Related issue:
- #5202

@albertvillanova yeah I noticed it right after the PR :smile: thank you! the fix of the dataset info yaml fixes tests on CI, but in general order of splits in yaml influences the order in which they are displayed in the viewer, if I understand it correctly. So I suggest not to sort splits in yaml initially to avoid this for other datasets in the future. I think [this change](https://github.com/huggingface/datasets/pull/5201/files#diff-198ba4fdf2f94cb3e1aba8a0170a43b08d4ab5636d682374321c5a383a8be24dR571) should work for it. 

Changes to tests here maybe can be reverted considering that order in yaml now corresponds to the one in tests, thanks to your change in the dataset info.

Hehe, @polinaeterna, we make comments nearly at the same time as well... :laughing:
Some links to canonical datasets in the docs are outdated

As we don't have canonical datasets in the github repo anymore, some old links to them doesn't work. I don't know how many of them are there, I found link to SuperGlue here: https://huggingface.co/docs/datasets/dataset_script#multiple-configurations, probably there are more of them. These links should be replaced by links to the corresponding datasets on the Hub. 

Comments:
Thanks for catching this, I can go through the docs and replace the links to their corresponding datasets on the Hub!
[zstd] Use max window log size

ZstdDecompressor has a parameter `max_window_size` to limit max memory usage when decompressing zstd files. The default `max_window_size` is not enough when files are compressed by `zstd --ultra` flags.

Change `max_window_size` to the zstd's max window size. NOTE, the `zstd.WINDOWLOG_MAX` is the log_2 value of the max window size.

Comments:
@albertvillanova Please take a review.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5197). All of your documentation changes will be reflected on that endpoint.
Drop labels in Image and Audio folders if files are on different levels in directory or if there is only one label

Will close https://github.com/huggingface/datasets/issues/5153

Drop labels by default (`drop_labels=None`) when:
* there are files on different levels of directory hierarchy by checking their path depth
* all files are in the same directory (=only one label was inferred)

First one fixes cases like this:
```
repo
   image3.jpg
   image4.jpg
   data
      image1.jpg
      image2.jpg
```
Second one fixes cases like this:
```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
This is mostly to fix the viewer for people who just drop images in the Hub interface into the root dir.

I added tests for both of the cases on local and remote files. **I also changed data files for old test on drop_labels** (`test_generate_examples_drop_labels`). The files I provide to `test_generate_examples_drop_labels` now has "canonical" classification structure (two dirs) in order not to change the logic of the test (=not to check these two cases addressed in the PR).



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

> Nit: maybe we can use the count_path_segments function from this file for counting (updated with your logic to make it faster).

@mariosasko just to make sure I understood you correctly - are you okay with this change? (actually `os.path.normpath` is redundant here as paths from `data_files` should be already normalized but just in case)
https://github.com/huggingface/datasets/pull/5192/files#diff-1f09f7a178211f7539b1499b64b69793bd53b30c8b7b34cfcc5835e25d31929fR33
If you are, we can merge.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

awesome ! :D
Make torch.Tensor and spacy models cacheable

Override `Pickler.save` to implement deterministic reduction (lazily registered; inspired by https://github.com/uqfoundation/dill/blob/master/dill/_dill.py#L343) functions for `torch.Tensor` and spaCy models.

Fix https://github.com/huggingface/datasets/issues/5170, fix https://github.com/huggingface/datasets/issues/3178


Comments:
_The documentation is not available anymore as the PR was closed or merged._
`path` is `None` when downloading a custom audio dataset from the Hub

### Describe the bug

I've created an [audio dataset](https://huggingface.co/datasets/lewtun/audio-test-push) using the `audiofolder` feature desribed in the [docs](https://huggingface.co/docs/datasets/audio_dataset#audiofolder) and then pushed it to the Hub. 

Locally, I can see the `audio.path` feature is of the expected form `path/to/data_dir`, but when I download the dataset from the Hub, I see `audio.path` is `None`

Here's an example:

```python
from datasets import load_dataset

ds = load_dataset("lewtun/audio-test-push")
ds["train"][0]
# {
#     "audio": {
#         "path": None,    <-- Is this expected?
#         "array": array(
#             [
#                 3.97140226e-07,
#                 7.30310290e-07,
#                 7.56406735e-07,
#                 ...,
#                 -1.19636677e-01,
#                 -1.16811886e-01,
#                 -1.12441722e-01,
#             ]
#         ),
#         "sampling_rate": 44100,
#     },
#     "song_id": 0,
#     "genre_id": 0,
#     "genre": "Electronic",
# }
```

Is this expected behaviour? If yes, feel free to close this issue as it's not a true bug then :)

### Steps to reproduce the bug

1. Create an audio dataset with the `audiofolder` feature
2. Push the dataset to the Hub with `push_to_hub()`
3. Download the Hub dataset and inspect the `audio.path` feature

### Expected behavior

`audio.path` points to the file associated with the audio data

### Environment info

- `datasets` version: 2.6.2.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1

Comments:
Hi! Yes, this is expected behavior - we do this as a security measure to not leak local paths (this info would be useless on other users' machines anyways) and only push audio bytes.
add: segmentation guide.

Closes #5181 

I have opened a PR on Hub (https://huggingface.co/datasets/huggingface/documentation-images/discussions/5) to include the images in our central Hub repository. Once the PR is merged I will edit the image links. 

I have also prepared a [Colab Notebook](https://colab.research.google.com/drive/1BMDCfOTBnyshoME5RSxn5iQy-TWeFbOA?usp=sharing) in case anyone wants to play. 

- [x] Replace the image links 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @osanseviero. Am I good to merge? 

I would wait for a second approval just in case :) 

Sure :) 

Merging since the images have been pushed as LFS files ([PR](https://huggingface.co/datasets/huggingface/documentation-images/discussions/8)).
chore: add notebook links to img cls and obj det.

Closes https://github.com/huggingface/datasets/issues/5182

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@nateraw I guess the failing test is unrelated.  

@sayakpaul Yea failures are unrelated. 

Alright. Will wait for @osanseviero's take and then merge. 

FYI @stevhliu 

@osanseviero @stevhliu @nateraw thank you for your comments. Acted on them.

Thanks! Can I merge? Or should we wait for approvals from the others?

Since @stevhliu approved as well, I think you're good to go

Alright!

Merging as a Member for the first time 🫀
Incorrect error message when Dataset.from_sql fails and sqlalchemy not installed

### Describe the bug

When calling `Dataset.from_sql` (in my case, with sqlite3), it fails with a message ```ValueError: Please pass `features` or at least one example when writing data``` when I don't have `sqlalchemy` installed. 


### Steps to reproduce the bug

Make a new sqlite db with `sqlite3` and `pandas` from a remote [URL](https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv).

```python
import sqlite3
import pandas as pd
from datasets import Dataset

conn = sqlite3.connect('us_covid_data.db')
df = pd.read_csv('https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv')
df.to_sql('states', conn, if_exists='replace')
```

Then if you try to query this DB like this:

```python
ds = Dataset.from_sql('''SELECT * from states WHERE state=="New York";''', "sqlite:///us_covid_data.db")
```

You run into the error I described above:

```ValueError: Please pass `features` or at least one example when writing data```

However, if you try to pass features, as the error suggests, then you get an error that tells you the underlying problem...

```python
from datasets import Dataset, Features, Value

features = Features({
    'date': Value('date32'),
    'label': Value('string'),
    'fips': Value('int32'),
    'cases': Value('int32'),
    'deaths': Value('int32')
})
ds = Dataset.from_sql(
    '''SELECT * from states WHERE state=="New York";''',
    "sqlite:///us_covid_data.db",
    features=features
)
```

Which results in the actual underlying error: `ImportError: Using URI string without sqlalchemy installed.`

### Expected behavior

Instead of `ValueError` about needing to pass features, we should provide the actual underlying error about not having SQLAlchemy installed when it isn't found in the environment. 

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.10
- PyArrow version: 10.0.0
- Pandas version: 1.2.5

Comments:
Hi! The first `Dataset.from_sql` call also outputs the "ImportError: Using URI string without sqlalchemy installed." message, but you also get "During handling of the above exception another exception occurred: ..." after which the ValueError is printed. I agree that this behavior makes it easy to miss the original error. 

I think we can improve this by not throwing the writer's ValueError if the error from a dataset script is already being handled to make debugging easier. @lhoestq @albertvillanova wdyt?

Yup ! Alternatively the error can be raised in sql.py before generating the examples ? In `_info` for example

yea @lhoestq that would probably be good. The 2nd error is useless if the 1st error is the real reason it failed.
Allow passing a subset of output features to Dataset.map

### Feature request

Currently, map does one of two things to the features (if I'm not mistaken):

* when you do not pass features, types are assumed to be equal to the input if they can be cast, and inferred otherwise
* when you pass a full specification of features, output features are set to this

However, sometimes you want to just pass some of the output types, particularly when the first of these modes makes an incorrect type. This currently crashes.

### Motivation

To give a little background: this problem appears in converting labels to ids, where the labels happen to be floats rather than strings

Consider the following use of map to convert from float to int
```python
data = Dataset.from_dict({'y':[1.0,2.0,3.0]})
mapped = data.map(lambda r: {'y': int(r['y'])})
mapped['y'] # is floats, not ints
```

The result is a float again, since after the mapping operation it forces the old datatypes back on the data. 

Passing `features=Features({"y": Value(dtype="int64")})` to map works in principle, but then extending it a little to e.g.
```python
def format_data(r):
    return {**tokenizer(r["text"]), "y": int(r["y"])}

data = Dataset.from_dict({"y": [1.0, 2.0, 3.0], "text": ["one", "two", "three"]})
mapped = data.map(
    format_data,
    features=Features({'y': Value(dtype="int64")}),
    remove_columns=["text"],
)
```

Results in a crash in dataset internals, as it expects either all or no output features to be specified.
Of course one can pass a full feature specification, but this becomes tokenizer specific and very awkward.


### Your contribution

I've looked at `write_batch` and particularly `col_type = features[col] if features else None`, but checking for `col in features` here makes it fail elsewhere, but the structure makes it hard to understand how and why. I do not think I would have the time myself to get to the bottom of this anytime soon.

Comments:
Loading an external dataset in a format similar to conll2003

I'm trying to load a custom dataset in a Dataset object, it's similar to conll2003 but with 2 columns only (word entity), I used the following script:

features = datasets.Features(
            {"tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=["B-PER", .... etc.]))}
        )


from datasets import Dataset

INPUT_COLUMNS = "tokens ner_tags".split(" ")

def read_conll(file):
    #all_labels = []
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
          if line:
            if line.startswith("-DOCSTART-") and example["tokens"] != []:
                print(idx, example)
                yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            elif line == "\n" or (line.startswith("-DOCSTART-") and example["tokens"] == []):
              continue
            else:
                row_cols = line.split(" ")
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "/content/new_train.txt"}, features = features)   

The following error happened:
[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <genexpr>(.0)
    285     for key in unique_values(itertools.chain(*dicts)):  # set merge all keys
    286         # Will raise KeyError if the dict don't have the same keys
--> 287         yield key, tuple(d[key] for d in dicts)
    288 
    

TypeError: tuple indices must be integers or slices, not str

What does this mean and what should I modify?

Comments:
Add notebook / other resource links to the task-specific data loading guides

Does it make sense to include links to notebooks / scripts that show how to use a dataset for training / fine-tuning a model? 

For example, here in [https://huggingface.co/docs/datasets/image_classification] we could include a mention of https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb.

Applies to https://huggingface.co/docs/datasets/object_detection as well.

Cc: @osanseviero @nateraw 

Comments:
Yea this would be great! We would need an object detection tutorial notebook too if it doesn't already exist there. 

There is one: https://huggingface.co/docs/datasets/object_detection.

I will start the work.
Add a guide for semantic segmentation

Currently, we have these guides for object detection and image classification:

* https://huggingface.co/docs/datasets/object_detection
* https://huggingface.co/docs/datasets/image_classification

I am proposing adding a similar guide for semantic segmentation. 

I am happy to contribute a PR for it. 

Cc: @osanseviero @nateraw 

Comments:
Sure this sounds great! Would this be pure torchvision, albumentations, or something else?

I am considering `torchvision` and `albumentations`. Also [works with TensorFlow](https://github.com/deep-diver/segformer-tf-transformers/blob/main/notebooks/TFSegFormer_Finetune.ipynb). 

I am assigning the issue to myself then.
An example or recommendations for creating large image datasets?

I know that Apache Beam and `datasets` have [some connector utilities](https://huggingface.co/docs/datasets/beam). But it's a little unclear what we mean by "But if you want to run your own Beam pipeline with Dataflow, here is how:". What does that pipeline do? 

As a user, I was wondering if we have this support for creating large image datasets. If so, we should mention that [here](https://huggingface.co/docs/datasets/image_dataset). 

Cc @lhoestq 

Comments:
The beam utilities allow to prepare a dataset as parquet in your cloud storage. From my perspective this CLI is not super easy to use, but we've been working on a new python API to prepare a dataset in your cloud storage:
```python
from datasets import load_dataset_builder

builder = load_dataset_builder("c4", "en")
builder.download_and_prepapre("s3://my-bucket/c4", file_format="parquet")
```

And to use Beam you can do:
```python
beam_runner = ...  # one of "SparkRunner", "DataFlowRunner", "DirectRunner", etc.
beam_options = ...

builder.download_and_prepapre(
    "s3://my-bucket/c4",
    file_format="parquet",
    beam_runner=beam_runner,
    beam_options=beam_options
)
```

Though Beam can be used ONLY if there is a dataset script based on the `BeamBasedBuilder` right now - it doesn't work on an arbitrary dataset (see [wikipedia.py](https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py) for example).

Thanks! 

Would be nice to have something similar for creating large image datasets.
Loading an external NER dataset

I need to use huggingface datasets to load a custom dataset similar to conll2003 but with more entities and each the files contain only two columns: word and ner tag.

I tried this code snnipet that I found here as an answer to a similar issue:

from datasets import Dataset

INPUT_COLUMNS = "ID Text NER".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

train = Dataset.from_generator(read_conll, gen_kwargs={"file": "some_path"})   

But the following error happened:

ValueError: Please pass `features` or at least one example when writing data

Comments:
Raise ffmpeg warnings only once

Our warnings looks nice now.

`librosa` warning that was raised at each decoding: 
```
/usr/local/lib/python3.7/dist-packages/librosa/core/audio.py:165: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn("PySoundFile failed. Trying audioread instead.")
```
is suppressed with `filterwarnings("ignore")` in a context manager. That means the first warning is also ignored (setting `filterwarnings("once")` didn't work!), so I added info that audioread is used for decoding to our message. Hope it's enough.


Tests failed at first because they used to check if the warning was raised at (each) decoding in `librosa` case but now we throw only one warning (at first decoding). I removed this check for warnings, do you think it's fine? 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Inconsistency behavior between handling local file protocol and other FS protocols

### Describe the bug

These lines us used during load_from_disk:
```
if is_remote_filesystem(fs):
    dest_dataset_dict_path = extract_path_from_uri(dataset_dict_path)
else:
    fs = fsspec.filesystem("file")
    dest_dataset_dict_path = dataset_dict_path
```
If a local FS is given, then it will the URL as the path name. If a remote Fs is given, then it will use the path of the URL. This is an inconsistent behavior when handling a file: when using remote FS, you must write a URL, but for local FS, even if you passed LocalFileSystem as `fs` you still can't use a `file://` URL. It will be recognized as a directory named `file:`.

### Steps to reproduce the bug

```
import fsspec.core
url = "hdfs:///somewhere/MNIST"
# url = "file:///somewhere/MNIST"
fs, path = fsspec.core.url_to_fs(url)

fs.ls(path) # this will always work
load_from_disk(path, fs) # only works for local FS
load_from_disk(url, fs) # only works for remote FS

```

### Expected behavior

one of `url` or `path` should always work

I think we extract path from given URL by using `fsspec.core.url_to_fs` instead of using `is_remote_filesystem` and `extract_path_from_uri` will fix this, since:
```
fsspec.core.url_to_fs("/somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("file:///somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("hdfs:///somewhere/MNIST") ->  HDFS, '/somewhere/MNIST'
```
and
```
fsspec.core.url_to_fs("file:///somewhere/MNIST") == fsspec.core.url_to_fs("/somewhere/MNIST")
```
In theory, this wouldn't break anything, since giving local path and remote uri still works. It will only affect local URI (make it works too)

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-5.4.205.1**HIDDEN**
- Python version: 3.7.10
- PyArrow version: 8.0.0
- Pandas version: 1.2.4


Comments:
[Caching] Deterministic hashing of torch tensors

Currently this fails
```python
import torch
from datasets.fingerprint import Hasher

t = torch.tensor([1.])

def func(x):
    return t + x

hash1 = Hasher.hash(func)
t = torch.tensor([1.])
hash2 = Hasher.hash(func)
assert hash1 == hash2
```

Also as noticed in https://discuss.huggingface.co/t/dataset-cant-cache-models-outputs/24945, using a model in a `map` function doesn't work well with caching. Indeed the `bert-base-uncased` model has a different hash every time you reload it. Supporting torch tensors may also help in this case.

This can be fixed by registering a custom pickling functions for torch tensors - as we did for other objects such as CodeType, FunctionType and Regex in `py_utils.py`

Comments:
Add "ipykernel" to list of `co_filename`s to remove

Should resolve #5157 

Comments:
I don't know how I could add some tests for this, although jupyter is not among the dependencies so at least that would need to be added. If someone can tell a recommended way I will try to do it!

So testing by myself and looking around the jupyter codebase it looks like the `co_filename` of objects created within jupyter is of the form `f"{tempfile.tempdir}/ipykernel_{id1}/{id2}.py"` however I can't find the exact command setting it so I [asked in discourse](https://discourse.jupyter.org/t/co-filename-within-notebooks/16538). For now adapted the `co_filename` filter and added tests according to this I hope to get an answer and possibly fix based on that.

Ok ! I think it's fine to just check if the parent folder is named like `ipykernel_*` then

see the source code of how it's created:

https://github.com/ipython/ipykernel/blob/7f73ff705510b35d1e2faad7f5a676c620ce08d4/ipykernel/compiler.py#L72-L75

Should look better now didn't notice the duplicated tests

_The documentation is not available anymore as the PR was closed or merged._

Should work now on windows too

I did the changes you suggested and tried to rebase, the first part went fine, the second less so :( 

If you have time to spare, can you tell me what should I do now to fix this? thanks

Instead of rebasing you can just merge `main` into your branch, otherwise the GitHub preview of your PR shows changes of from `main`.

Feel free to close this PR and create a new one. Or alternatively your can force push to this PR with a new clean git history.

I have force-pushed and merged main, only shows the right changes, if you can run CI one more time it should be ok now

Hi, sorry I have been busy, the thing is I can't really understand why the test fail, besides the ugly thing I had done in the last commit to check if within CI smth stange happened with `os`, locally tests pass

The CI wasn't passing when using the latest version `dill==0.3.6`. We have a separate function to dump CodeType objects for 0.3.6

I applied the same changes you did to this other function as well - it should be all good now

> The CI wasn't passing when using the latest version `dill==0.3.6`. We have a separate function to dump CodeType objects for 0.3.6
> 
> I applied the same changes you did to this other function as well - it should be all good now

Thanks, it would have taken a long time to figure out :)
Fix CI require beam

This PR:
- Fixes the CI `require_beam`: before it was requiring PyTorch instead
  ```python
  def require_beam(test_case):
      if not config.TORCH_AVAILABLE:
          test_case = unittest.skip("test requires PyTorch")(test_case)
      return test_case
  ```
- Fixes a missing `require_beam` in `test_beam_based_builder_download_and_prepare_as_parquet`
- Refactors `require_beam` to use `pytest` (`skipif`) instead

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm merging this PR because it is quite a trivial fix and this is required by:
- #5166
Add ffmpeg4 installation instructions in warnings

Adds instructions on how to install `ffmpeg=4` on Linux (relevant for Colab users).

Looks pretty ugly because I didn't find a way to check `ffmpeg` version from python (without `subprocess.call()`; `ctypes.util.find_library` doesn't work`), so the warning is raised on each decoding. Any suggestions on how to make it look nice are welcome!

This is how it looks on Colab:
![image](https://user-images.githubusercontent.com/16348744/198052412-d48018d1-4416-4aa5-9114-f7f9b4af031f.png)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

To make it warn only once, feel free to use a global counter in python - and if the warning has already been done, you don't do it again

> Added the same formatting for the error message :)

nice!! thank you! 

> Oh and regarding the warning counter, you can do it in another PR maybe ?

Yes, more warnings is better then no warnings.... I'll merge when the CI passes
Memory explosion when trying to access 4d tensors in datasets cast to torch or np

### Describe the bug

When trying to access an item by index, in a datasets.Dataset cast to torch/np using `set_format` or `with_format`, we get a memory explosion if the item contains 4d (or above) tensors. 

### Steps to reproduce the bug

MWE:

```python
from datasets import load_dataset
import numpy as np

def create_4d_tensor(item):  
    i = item["num_nodes"]
    item["x_big"] = np.random.rand(i, 2*i, int(i/2), 1) + 1 # we create a big 4d tensor
    return item

if __name__ == "__main__":
    dataset = load_dataset(path=f"graphs-datasets/PROTEINS")

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset = dataset.map( 
        create_4d_tensor, 
        batched=False, 
        writer_batch_size=100, 
    )

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset.set_format("torch")
    
    print(dataset["train"].format)
    # This gets killed :(
    print(dataset["train"][0].keys())
```

The problem likely comes from `format_table` [here](https://cs.github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/src/datasets/arrow_dataset.py#L2328)

### Expected behavior

No memory explosion when trying to access dataset items after cast.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Comments:
WIP: drop labels in Image and Audio folders by default

will fix https://github.com/huggingface/datasets/issues/5153 and redundant labels displaying for most of the images datasets on the Hub (which are used just to store files)

TODO: discuss adding `drop_labels` (and `drop_metadata`) params to yaml

Comments:
_The documentation is not available anymore as the PR was closed or merged._

close in favor of https://github.com/huggingface/datasets/pull/5192
Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6

### Describe the bug

When using `pip-compile` (part of `pip-tools`) to generate a pinned requirements file that includes `datasets`, a version conflict of `dill` appears.

It is caused by a transitive dependency conflict between `datasets` and `multiprocess`.

### Steps to reproduce the bug

```bash
$ echo "datasets" > requirements.in
$ pip install pip-tools
$ pip-compile requirements.in 
Could not find a version that matches dill<0.3.6,>=0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
Tried: 0.2, 0.2, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.2.5, 0.2.5, 0.2.6, 0.2.7, 0.2.7.1, 0.2.8, 0.2.8.1, 0.2.8.2, 0.2.9, 0.3.0, 0.3.1, 0.3.1.1, 0.3.2, 0.3.3, 0.3.3, 0.3.4, 0.3.4, 0.3.5, 0.3.5, 0.3.5.1, 0.3.5.1, 0.3.6, 0.3.6
Skipped pre-versions: 0.1a1, 0.2a1, 0.2a1, 0.2b1, 0.2b1
There are incompatible versions in the resolved dependencies:
  dill<0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
  dill>=0.3.6 (from multiprocess==0.70.14->datasets==2.6.1->-r requirements.in (line 1))
```

### Expected behavior

A correctly generated file `requirements.txt` with pinned dependencies

### Environment info

Tested with versions `2.6.1, 2.6.0, 2.5.2` on Python 3.8 and 3.10 on Ubuntu 20.04LTS and Python 3.10 on MacOS 12.6 (M1).

Comments:
Thanks for reporting, @Rijgersberg.

We were waiting for the release of `dill` 0.3.6, that happened 2 days ago (24 Oct 2022): https://github.com/uqfoundation/dill/releases/tag/dill-0.3.6
- See comment: https://github.com/huggingface/datasets/pull/4397#discussion_r880629543

Also `multiprocess` 0.70.14 was released 2 days ago: https://github.com/uqfoundation/multiprocess/releases/tag/multiprocess-0.70.14

We are addressing this issue to align dependencies.

In your specific setup, I guess the compatible configuration is with `multiprocess` 0.70.13 (instead of 0.70.14).

@Rijgersberg this issue is fixed. It will be available in our next `datasets` release.

Thanks!

> @Rijgersberg this issue is fixed. It will be available in our next `datasets` release.

Any chance you have a eta? 

@StefanSamba we are disussing about making a release early this week.

@Rijgersberg, please also that you can make `pip-compile` work by using the backtracking resolver (instead of the legacy one): https://pip-tools.readthedocs.io/en/latest/#a-note-on-resolvers
```
pip-compile --resolver=backtracking requirements.in
```
This resolver will automatically use `multiprocess` 0.70.13 version.
Dataset can’t cache model’s outputs

### Describe the bug

Hi,

I try to cache some outputs of teacher model( Knowledge Distillation ) by using map function of Dataset library, while every time I run my code, I still recompute all the sequences. I tested Bert Model like this, I got different hash every single run, so any idea to deal with this?

### Steps to reproduce the bug

1. run below code
2. get different hash
```
from transformers import BertModel
from transformers import AutoTokenizer
import torch
token = ['hello']
model = BertModel.from_pretrained("bert-base-uncased").eval()
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
def abcd():
    with torch.no_grad():
        
        out = model(**tok(token,return_tensors='pt'))[0]
        # out = tok(token)
    return out
from datasets.fingerprint import Hasher
my_func = abcd
print(Hasher.hash(my_func))

print(abcd())
```

### Expected behavior

I wanna cache all the model output 

### Environment info

datasets:2.5.0

Comments:
Addressed in https://github.com/huggingface/datasets/pull/5191 (torch.Tensor objects now produce deterministic hashes)
Fix language and license tag names in all Hub datasets

While working on this:
- #5137

we realized there are still many datasets with deprecated "languages" and "licenses" tag names (instead of "language" and "license").

This is a blocking issue: no subsequent PR can be opened to modify their metadata: a ValueError will be thrown.

We should fix the "language" and "license" tag names in all Hub datasets.

TODO:
- [x] Fix language and license tag names in 402 Hub datasets

CC: @julien-c 

Comments:
There are currently 402 datasets with deprecated "languages" or "licenses".

hey @albertvillanova ,i would love to work on this issue if you like.

Hi @ayushthe1, thanks for your offer.

But as you can see, I self-assigned this issue.

I have already fixed 200 out of the 402 datasets. My script is still running and fixing the rest.

For example: https://huggingface.co/datasets/fhamborg/news_sentiment_newsmtsc/discussions/2/files

Thanks for your time. Will try next time. 😇

@ayushthe1 feel free to take one of the non-assigned open issues: https://github.com/huggingface/datasets/issues

This is done.