KRadim/github-issues-minimal huggingface dataset:
Add support for different configs with `push_to_hub`
State: closed
Labels: name: enhancement

will solve  #5151

@lhoestq @albertvillanova @mariosasko 
This is still a super draft so please ignore code issues but I want to discuss some conceptually important things.

I suggest a way to do `.push_to_hub("repo_id", "config_name")` with pushing parquet files to directories named as `config_name` (inside `data/` dir as it is now), for example: 
```
data
   |__config-v1
        train-00000-00002-...-.parquet
        train-00001-00002-...-.parquet
        ...
   |__config-v2
        ....
```

When loading a dataset, I parse these configs from repository data files (only for `"data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"` pattern that is used for parquet datasets pushed with `.push_to_hub`). 
Therefore, 
- when user tries to load a dataset that has configs parsed from data files dir names without providing a config (like `load_dataset("repo")` instead of `load_dataset("repo", "config-v1")`) - raise error and asks for config - to be aligned with how it works in datasets with scripts.
- for backward compatibility: if user tries to `.push_to_hub(""repo", "config_name")` to an existing parquet repo with no configurations (all parquet files are directly in `data/` dir) - raise error. My initial idea was to raise a warning and move these files to another dir with name (config) like "default" or smth but in a PR and suggest user to merge it on the Hub. But there is no support for renaming (moving) files via `HfApi` yet so it would require deleting and pushing again if I understand it right.

This parsing approach can be extended to other Hub packaged modules, and to local packaged modules and other data files patterns
(except for cases when splits are in dir names `KEYWORDS_IN_DIR_NAME_BASE_PATTERNS` because we allow for arbitrary depth of directory hierarchy).
Do you think it's reasonable? Not sure how to provide flexibility (and backward compatibility) to not parsing configs and load all the data in a single config as it is now.

I also thought about getting information about configs from Readme.md `dataset_info` ([example](https://huggingface.co/datasets/polinaeterna/test_push_two_configs/blob/main/README.md)). But that way we 
are dependent on if it exists. It is created automatically with `.push_to_hub` but what if it is 
accidentally deleted or smth). 

Also, what I don't like is that this parsing is a part of Module/DataFiles logic, not Builder's one, which is not aligned with datasets with custom scripts. But I don't know to implement the second approach in current library's logic. 

 What do you think about this all? Am I missing smth?


TODO:

- [ ] save cache in the same dir for configs of the same datasets
- [ ] fix verification errors
- [ ] correctly update `dataset_infos.json` too
- [ ] ...
Fix CI require_beam maximum compatible dill version
State: closed

A previous commit to main branch introduced an additional requirement on maximum compatible `dill` version with `apache-beam` in our CI `require_beam`:
- d7c942228b8dcf4de64b00a3053dce59b335f618
- ec222b220b79f10c8d7b015769f0999b15959feb

This PR fixes the maximum compatible `dill` version with `apache-beam`, which is <0.3.2 (and not 0.3.6): https://github.com/apache/beam/blob/v2.42.0/sdks/python/setup.py#L219
Implement ability to define splits in metadata section of dataset card
State: closed
Labels: name: enhancement

### Feature request

If you go here: https://huggingface.co/datasets/inria-soda/tabular-benchmark/tree/main you will see bunch of folders that has various CSV files. I’d like dataset viewer to show these files instead of only one dataset like it currently does. (and also people to be able to load them as splits instead of loading through `data_files`) 
e.g GLUE has various splits on viewer but it’s too overkill to ask people to implement loading script, so it would be better to let them define these in the README file instead.

Also pinging @polinaeterna @lhoestq @adrinjalali
Use logging instead of printing to console
State: closed

### Describe the bug

Some logs ([here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L778), [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L786), and [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L830)) generated by the `DatasetBuilder` are printed to the console instead of passed to `datasets` logger.


### Steps to reproduce the bug

```python
>> import datasets

>> datasets.load_dataset("some-dataset")

Downloading and preparing dataset csv/data to <path>...
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7729.06it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 527.23it/s]
Dataset csv downloaded and prepared to <path>. Subsequent calls will reuse this data.
```

### Expected behavior

The logs should not be printed to the console directly but passed to the logger so that the user can redirect them wherever he wants.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-x86_64-i386-64bit
- Python version: 3.9.15
- PyArrow version: 10.0.0
- Pandas version: 1.5.1
`push_to_hub` not propagating `token` through `DownloadConfig`
State: closed

### Describe the bug

When trying to upload a new 🤗 Dataset to the Hub via Python, and providing the `token` as a parameter to the `Dataset.push_to_hub` function, it just works for the first time, assuming that the dataset didn't exist before.

But when trying to run `Dataset.push_to_hub` again over the same dataset, instead of updating it, it throws a `ConnectionError` when trying to retrieve the `README.md` that may contain some metadata about the dataset, so as to also update it, but since the `token` is not propagated, the `DownloadConfig` provided to the `datasets.utils.file_utils.get_from_cache` function doesn't contain the `use_auth_token` value set to `token`, it's just using the default one which is None/False.

So on, when uploading a dataset via Python with `push_to_hub` with the `token` as a parameter with the HuggingFace API Token as value, it can just be uploaded when the dataset is new, otherwise it fails with to `ConnectionError` due to the `token` not being propagated as `use_auth_token`.

### Steps to reproduce the bug

Let's create a new dataset in our HF account via Python as:

```python
from datasets import Dataset

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
ds = Dataset.from_dict(data)
ds.push_to_hub(repo_id=<HF_USERNAME>/<HF_DATASET>, private=private, token=<HF_TOKEN_HERE>)
```

When we create the `Dataset` for the first time it works and there are no issues, but when trying to actually upload a new version of the same dataset (same name under the same username), we encounter the following issue:

```python
from datasets import Dataset

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
ds = Dataset.from_dict(data)
ds.push_to_hub(repo_id=<HF_USERNAME>/<HF_DATASET>, private=private, token=<HF_TOKEN_HERE>)
>>> ConnectionError: Couldn't reach https://huggingface.co/datasets/alvarobartt/demo/resolve/main/README.md (ConnectionError('Unauthorized for URL https://huggingface.co/datasets/<HF_USERNAME>/<HF_DATASET>/resolve/main/README.md. Please use the parameter `use_auth_token=True` after logging in with `huggingface-cli login`'))
```

### Expected behavior

Ideally, the `token` parameter provided to `push_to_hub` should be propagated and used to download the `README.md` when trying to update a `Dataset`, instead of throwing that exception, so that the authentication can be done directly through code without running `huggingface-cli login`as mentioned at https://huggingface.co/docs/datasets/upload_dataset#upload-with-python.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-arm64-arm-64bit
- Python version: 3.10.8
- PyArrow version: 10.0.0
- Pandas version: 1.5.1
Some links to canonical datasets in the docs are outdated
State: closed
Labels: name: documentation

As we don't have canonical datasets in the github repo anymore, some old links to them doesn't work. I don't know how many of them are there, I found link to SuperGlue here: https://huggingface.co/docs/datasets/dataset_script#multiple-configurations, probably there are more of them. These links should be replaced by links to the corresponding datasets on the Hub.
[zstd] Use max window log size
State: open

ZstdDecompressor has a parameter `max_window_size` to limit max memory usage when decompressing zstd files. The default `max_window_size` is not enough when files are compressed by `zstd --ultra` flags.

Change `max_window_size` to the zstd's max window size. NOTE, the `zstd.WINDOWLOG_MAX` is the log_2 value of the max window size.
Use hfh hf_hub_url function
State: closed

Small refactoring to use `hf_hub_url` function from `huggingface_hub`.

This PR also creates the `hub` module that will contain all `huggingface_hub` functionalities relevant to `datasets`.

This is a necessary stage before implementing the use of the `hfh` caching system (which uses its `hf_hub_url` under the hood).

EDIT:
~~Finally, we use our `config.HUB_DATASETS_URL` when using `hfh.hf_hub_url`~~
There is a breaking change:  the `hfh` `hf_hub_url` function uses
- `hfh` `HUGGINGFACE_CO_URL_TEMPLATE` URL template, different from the `datasets` `config.HUB_DATASETS_URL`
- also, `hfh` `DEFAULT_REVISION`, instead of `datasets` `config.HUB_DEFAULT_VERSION`
Drop labels in Image and Audio folders if files are on different levels in directory or if there is only one label
State: closed
Labels: name: bug

Will close https://github.com/huggingface/datasets/issues/5153

Drop labels by default (`drop_labels=None`) when:
* there are files on different levels of directory hierarchy by checking their path depth
* all files are in the same directory (=only one label was inferred)

First one fixes cases like this:
```
repo
   image3.jpg
   image4.jpg
   data
      image1.jpg
      image2.jpg
```
Second one fixes cases like this:
```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
This is mostly to fix the viewer for people who just drop images in the Hub interface into the root dir.

I added tests for both of the cases on local and remote files. **I also changed data files for old test on drop_labels** (`test_generate_examples_drop_labels`). The files I provide to `test_generate_examples_drop_labels` now has "canonical" classification structure (two dirs) in order not to change the logic of the test (=not to check these two cases addressed in the PR).
Make torch.Tensor and spacy models cacheable
State: closed

Override `Pickler.save` to implement deterministic reduction (lazily registered; inspired by https://github.com/uqfoundation/dill/blob/master/dill/_dill.py#L343) functions for `torch.Tensor` and spaCy models.

Fix https://github.com/huggingface/datasets/issues/5170, fix https://github.com/huggingface/datasets/issues/3178
`path` is `None` when downloading a custom audio dataset from the Hub
State: closed

### Describe the bug

I've created an [audio dataset](https://huggingface.co/datasets/lewtun/audio-test-push) using the `audiofolder` feature desribed in the [docs](https://huggingface.co/docs/datasets/audio_dataset#audiofolder) and then pushed it to the Hub. 

Locally, I can see the `audio.path` feature is of the expected form `path/to/data_dir`, but when I download the dataset from the Hub, I see `audio.path` is `None`

Here's an example:

```python
from datasets import load_dataset

ds = load_dataset("lewtun/audio-test-push")
ds["train"][0]
# {
#     "audio": {
#         "path": None,    <-- Is this expected?
#         "array": array(
#             [
#                 3.97140226e-07,
#                 7.30310290e-07,
#                 7.56406735e-07,
#                 ...,
#                 -1.19636677e-01,
#                 -1.16811886e-01,
#                 -1.12441722e-01,
#             ]
#         ),
#         "sampling_rate": 44100,
#     },
#     "song_id": 0,
#     "genre_id": 0,
#     "genre": "Electronic",
# }
```

Is this expected behaviour? If yes, feel free to close this issue as it's not a true bug then :)

### Steps to reproduce the bug

1. Create an audio dataset with the `audiofolder` feature
2. Push the dataset to the Hub with `push_to_hub()`
3. Download the Hub dataset and inspect the `audio.path` feature

### Expected behavior

`audio.path` points to the file associated with the audio data

### Environment info

- `datasets` version: 2.6.2.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
Reduce friction in tabular dataset workflow by eliminating having splits when dataset is loaded
State: open
Labels: name: enhancement

### Feature request

Sorry for cryptic name but I'd like to explain using code itself. When I want to load a specific dataset from a repository (for instance, this: https://huggingface.co/datasets/inria-soda/tabular-benchmark)

```python
from datasets import load_dataset
dataset = load_dataset("inria-soda/tabular-benchmark", data_files=["reg_cat/house_sales.csv"], streaming=True)
print(next(iter(dataset["train"])))
```

`datasets` library is essentially designed for people who'd like to use benchmark datasets on various modalities to fine-tune their models, and these benchmark datasets usually have pre-defined train and test splits. However, for tabular workflows, having train and test splits usually ends up model overfitting to validation split so usually the users would like to do validation techniques like `StratifiedKFoldCrossValidation` or when they tune for hyperparameters they do `GridSearchCrossValidation` so often the behavior is to create their own splits. Even [in this paper](https://hal.archives-ouvertes.fr/hal-03723551) a benchmark is introduced but the split is done by authors.
It's a bit confusing for average tabular user to try and load a dataset and see `"train"` so it would be nice if we would not load dataset into a split called `train `by default.

```diff
from datasets import load_dataset
dataset = load_dataset("inria-soda/tabular-benchmark", data_files=["reg_cat/house_sales.csv"], streaming=True)
-print(next(iter(dataset["train"])))
+print(next(iter(dataset)))
``` 

### Motivation

I explained it above 😅

### Your contribution

I think this is quite a big change that seems small (e.g. how to determine datasets that will not be load to train split?), it's best if we discuss first!
add: segmentation guide.
State: closed
Labels: name: documentation

Closes #5181 

I have opened a PR on Hub (https://huggingface.co/datasets/huggingface/documentation-images/discussions/5) to include the images in our central Hub repository. Once the PR is merged I will edit the image links. 

I have also prepared a [Colab Notebook](https://colab.research.google.com/drive/1BMDCfOTBnyshoME5RSxn5iQy-TWeFbOA?usp=sharing) in case anyone wants to play. 

- [x] Replace the image links
Incorrect error message when Dataset.from_sql fails and sqlalchemy not installed
State: closed

### Describe the bug

When calling `Dataset.from_sql` (in my case, with sqlite3), it fails with a message ```ValueError: Please pass `features` or at least one example when writing data``` when I don't have `sqlalchemy` installed. 


### Steps to reproduce the bug

Make a new sqlite db with `sqlite3` and `pandas` from a remote [URL](https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv).

```python
import sqlite3
import pandas as pd
from datasets import Dataset

conn = sqlite3.connect('us_covid_data.db')
df = pd.read_csv('https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv')
df.to_sql('states', conn, if_exists='replace')
```

Then if you try to query this DB like this:

```python
ds = Dataset.from_sql('''SELECT * from states WHERE state=="New York";''', "sqlite:///us_covid_data.db")
```

You run into the error I described above:

```ValueError: Please pass `features` or at least one example when writing data```

However, if you try to pass features, as the error suggests, then you get an error that tells you the underlying problem...

```python
from datasets import Dataset, Features, Value

features = Features({
    'date': Value('date32'),
    'label': Value('string'),
    'fips': Value('int32'),
    'cases': Value('int32'),
    'deaths': Value('int32')
})
ds = Dataset.from_sql(
    '''SELECT * from states WHERE state=="New York";''',
    "sqlite:///us_covid_data.db",
    features=features
)
```

Which results in the actual underlying error: `ImportError: Using URI string without sqlalchemy installed.`

### Expected behavior

Instead of `ValueError` about needing to pass features, we should provide the actual underlying error about not having SQLAlchemy installed when it isn't found in the environment. 

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.10
- PyArrow version: 10.0.0
- Pandas version: 1.2.5
Allow passing a subset of output features to Dataset.map
State: open
Labels: name: enhancement

### Feature request

Currently, map does one of two things to the features (if I'm not mistaken):

* when you do not pass features, types are assumed to be equal to the input if they can be cast, and inferred otherwise
* when you pass a full specification of features, output features are set to this

However, sometimes you want to just pass some of the output types, particularly when the first of these modes makes an incorrect type. This currently crashes.

### Motivation

To give a little background: this problem appears in converting labels to ids, where the labels happen to be floats rather than strings

Consider the following use of map to convert from float to int
```python
data = Dataset.from_dict({'y':[1.0,2.0,3.0]})
mapped = data.map(lambda r: {'y': int(r['y'])})
mapped['y'] # is floats, not ints
```

The result is a float again, since after the mapping operation it forces the old datatypes back on the data. 

Passing `features=Features({"y": Value(dtype="int64")})` to map works in principle, but then extending it a little to e.g.
```python
def format_data(r):
    return {**tokenizer(r["text"]), "y": int(r["y"])}

data = Dataset.from_dict({"y": [1.0, 2.0, 3.0], "text": ["one", "two", "three"]})
mapped = data.map(
    format_data,
    features=Features({'y': Value(dtype="int64")}),
    remove_columns=["text"],
)
```

Results in a crash in dataset internals, as it expects either all or no output features to be specified.
Of course one can pass a full feature specification, but this becomes tokenizer specific and very awkward.


### Your contribution

I've looked at `write_batch` and particularly `col_type = features[col] if features else None`, but checking for `col in features` here makes it fail elsewhere, but the structure makes it hard to understand how and why. I do not think I would have the time myself to get to the bottom of this anytime soon.
Loading an external dataset in a format similar to conll2003
State: closed

I'm trying to load a custom dataset in a Dataset object, it's similar to conll2003 but with 2 columns only (word entity), I used the following script:

features = datasets.Features(
            {"tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=["B-PER", .... etc.]))}
        )


from datasets import Dataset

INPUT_COLUMNS = "tokens ner_tags".split(" ")

def read_conll(file):
    #all_labels = []
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
          if line:
            if line.startswith("-DOCSTART-") and example["tokens"] != []:
                print(idx, example)
                yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            elif line == "\n" or (line.startswith("-DOCSTART-") and example["tokens"] == []):
              continue
            else:
                row_cols = line.split(" ")
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "/content/new_train.txt"}, features = features)   

The following error happened:
[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <genexpr>(.0)
    285     for key in unique_values(itertools.chain(*dicts)):  # set merge all keys
    286         # Will raise KeyError if the dict don't have the same keys
--> 287         yield key, tuple(d[key] for d in dicts)
    288 
    

TypeError: tuple indices must be integers or slices, not str

What does this mean and what should I modify?
Add notebook / other resource links to the task-specific data loading guides
State: closed
Labels: name: enhancement

Does it make sense to include links to notebooks / scripts that show how to use a dataset for training / fine-tuning a model? 

For example, here in [https://huggingface.co/docs/datasets/image_classification] we could include a mention of https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb.

Applies to https://huggingface.co/docs/datasets/object_detection as well.

Cc: @osanseviero @nateraw
Add a guide for semantic segmentation
State: closed
Labels: name: documentation

Currently, we have these guides for object detection and image classification:

* https://huggingface.co/docs/datasets/object_detection
* https://huggingface.co/docs/datasets/image_classification

I am proposing adding a similar guide for semantic segmentation. 

I am happy to contribute a PR for it. 

Cc: @osanseviero @nateraw
An example or recommendations for creating large image datasets?
State: open

I know that Apache Beam and `datasets` have [some connector utilities](https://huggingface.co/docs/datasets/beam). But it's a little unclear what we mean by "But if you want to run your own Beam pipeline with Dataflow, here is how:". What does that pipeline do? 

As a user, I was wondering if we have this support for creating large image datasets. If so, we should mention that [here](https://huggingface.co/docs/datasets/image_dataset). 

Cc @lhoestq
Loading an external NER dataset
State: closed

I need to use huggingface datasets to load a custom dataset similar to conll2003 but with more entities and each the files contain only two columns: word and ner tag.

I tried this code snnipet that I found here as an answer to a similar issue:

from datasets import Dataset

INPUT_COLUMNS = "ID Text NER".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

train = Dataset.from_generator(read_conll, gen_kwargs={"file": "some_path"})   

But the following error happened:

ValueError: Please pass `features` or at least one example when writing data
Raise ffmpeg warnings only once
State: closed

Our warnings looks nice now.

`librosa` warning that was raised at each decoding: 
```
/usr/local/lib/python3.7/dist-packages/librosa/core/audio.py:165: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn("PySoundFile failed. Trying audioread instead.")
```
is suppressed with `filterwarnings("ignore")` in a context manager. That means the first warning is also ignored (setting `filterwarnings("once")` didn't work!), so I added info that audioread is used for decoding to our message. Hope it's enough.


Tests failed at first because they used to check if the warning was raised at (each) decoding in `librosa` case but now we throw only one warning (at first decoding). I removed this check for warnings, do you think it's fine?
Inconsistency behavior between handling local file protocol and other FS protocols
State: open

### Describe the bug

These lines us used during load_from_disk:
```
if is_remote_filesystem(fs):
    dest_dataset_dict_path = extract_path_from_uri(dataset_dict_path)
else:
    fs = fsspec.filesystem("file")
    dest_dataset_dict_path = dataset_dict_path
```
If a local FS is given, then it will the URL as the path name. If a remote Fs is given, then it will use the path of the URL. This is an inconsistent behavior when handling a file: when using remote FS, you must write a URL, but for local FS, even if you passed LocalFileSystem as `fs` you still can't use a `file://` URL. It will be recognized as a directory named `file:`.

### Steps to reproduce the bug

```
import fsspec.core
url = "hdfs:///somewhere/MNIST"
# url = "file:///somewhere/MNIST"
fs, path = fsspec.core.url_to_fs(url)

fs.ls(path) # this will always work
load_from_disk(path, fs) # only works for local FS
load_from_disk(url, fs) # only works for remote FS

```

### Expected behavior

one of `url` or `path` should always work

I think we extract path from given URL by using `fsspec.core.url_to_fs` instead of using `is_remote_filesystem` and `extract_path_from_uri` will fix this, since:
```
fsspec.core.url_to_fs("/somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("file:///somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("hdfs:///somewhere/MNIST") ->  HDFS, '/somewhere/MNIST'
```
and
```
fsspec.core.url_to_fs("file:///somewhere/MNIST") == fsspec.core.url_to_fs("/somewhere/MNIST")
```
In theory, this wouldn't break anything, since giving local path and remote uri still works. It will only affect local URI (make it works too)

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-5.4.205.1**HIDDEN**
- Python version: 3.7.10
- PyArrow version: 8.0.0
- Pandas version: 1.2.4
[Caching] Deterministic hashing of torch tensors
State: closed
Labels: name: enhancement

Currently this fails
```python
import torch
from datasets.fingerprint import Hasher

t = torch.tensor([1.])

def func(x):
    return t + x

hash1 = Hasher.hash(func)
t = torch.tensor([1.])
hash2 = Hasher.hash(func)
assert hash1 == hash2
```

Also as noticed in https://discuss.huggingface.co/t/dataset-cant-cache-models-outputs/24945, using a model in a `map` function doesn't work well with caching. Indeed the `bert-base-uncased` model has a different hash every time you reload it. Supporting torch tensors may also help in this case.

This can be fixed by registering a custom pickling functions for torch tensors - as we did for other objects such as CodeType, FunctionType and Regex in `py_utils.py`
Fix CI require beam
State: closed

This PR:
- Fixes the CI `require_beam`: before it was requiring PyTorch instead
  ```python
  def require_beam(test_case):
      if not config.TORCH_AVAILABLE:
          test_case = unittest.skip("test requires PyTorch")(test_case)
      return test_case
  ```
- Fixes a missing `require_beam` in `test_beam_based_builder_download_and_prepare_as_parquet`
- Refactors `require_beam` to use `pytest` (`skipif`) instead
Add ffmpeg4 installation instructions in warnings
State: closed

Adds instructions on how to install `ffmpeg=4` on Linux (relevant for Colab users).

Looks pretty ugly because I didn't find a way to check `ffmpeg` version from python (without `subprocess.call()`; `ctypes.util.find_library` doesn't work`), so the warning is raised on each decoding. Any suggestions on how to make it look nice are welcome!

This is how it looks on Colab:
![image](https://user-images.githubusercontent.com/16348744/198052412-d48018d1-4416-4aa5-9114-f7f9b4af031f.png)
Support dill 0.3.6
State: closed

This PR:
- ~~Unpins dill to allow installing dill>=0.3.6~~
- ~~Removes the fix on dill for >=0.3.6 because they implemented a deterministic mode (to be confirmed by @anivegesana)~~
- Pins dill<0.3.7 to allow latest dill 0.3.6
- Implements a fix for dill `save_function` for dill 0.3.6
- Additionally had to implement a fix for dill `save_code` and `_save_regex` for dill 0.3.6
- Fixes the CI so that the latest dill version is tested (besides the minimum 0.3.1.1 required by apache-beam 2.42.0)

Fix #5162.
Memory explosion when trying to access 4d tensors in datasets cast to torch or np
State: open

### Describe the bug

When trying to access an item by index, in a datasets.Dataset cast to torch/np using `set_format` or `with_format`, we get a memory explosion if the item contains 4d (or above) tensors. 

### Steps to reproduce the bug

MWE:

```python
from datasets import load_dataset
import numpy as np

def create_4d_tensor(item):  
    i = item["num_nodes"]
    item["x_big"] = np.random.rand(i, 2*i, int(i/2), 1) + 1 # we create a big 4d tensor
    return item

if __name__ == "__main__":
    dataset = load_dataset(path=f"graphs-datasets/PROTEINS")

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset = dataset.map( 
        create_4d_tensor, 
        batched=False, 
        writer_batch_size=100, 
    )

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset.set_format("torch")
    
    print(dataset["train"].format)
    # This gets killed :(
    print(dataset["train"][0].keys())
```

The problem likely comes from `format_table` [here](https://cs.github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/src/datasets/arrow_dataset.py#L2328)

### Expected behavior

No memory explosion when trying to access dataset items after cast.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6
State: closed

### Describe the bug

When using `pip-compile` (part of `pip-tools`) to generate a pinned requirements file that includes `datasets`, a version conflict of `dill` appears.

It is caused by a transitive dependency conflict between `datasets` and `multiprocess`.

### Steps to reproduce the bug

```bash
$ echo "datasets" > requirements.in
$ pip install pip-tools
$ pip-compile requirements.in 
Could not find a version that matches dill<0.3.6,>=0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
Tried: 0.2, 0.2, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.2.5, 0.2.5, 0.2.6, 0.2.7, 0.2.7.1, 0.2.8, 0.2.8.1, 0.2.8.2, 0.2.9, 0.3.0, 0.3.1, 0.3.1.1, 0.3.2, 0.3.3, 0.3.3, 0.3.4, 0.3.4, 0.3.5, 0.3.5, 0.3.5.1, 0.3.5.1, 0.3.6, 0.3.6
Skipped pre-versions: 0.1a1, 0.2a1, 0.2a1, 0.2b1, 0.2b1
There are incompatible versions in the resolved dependencies:
  dill<0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
  dill>=0.3.6 (from multiprocess==0.70.14->datasets==2.6.1->-r requirements.in (line 1))
```

### Expected behavior

A correctly generated file `requirements.txt` with pinned dependencies

### Environment info

Tested with versions `2.6.1, 2.6.0, 2.5.2` on Python 3.8 and 3.10 on Ubuntu 20.04LTS and Python 3.10 on MacOS 12.6 (M1).
Dataset can’t cache model’s outputs
State: closed

### Describe the bug

Hi,

I try to cache some outputs of teacher model( Knowledge Distillation ) by using map function of Dataset library, while every time I run my code, I still recompute all the sequences. I tested Bert Model like this, I got different hash every single run, so any idea to deal with this?

### Steps to reproduce the bug

1. run below code
2. get different hash
```
from transformers import BertModel
from transformers import AutoTokenizer
import torch
token = ['hello']
model = BertModel.from_pretrained("bert-base-uncased").eval()
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
def abcd():
    with torch.no_grad():
        
        out = model(**tok(token,return_tensors='pt'))[0]
        # out = tok(token)
    return out
from datasets.fingerprint import Hasher
my_func = abcd
print(Hasher.hash(my_func))

print(abcd())
```

### Expected behavior

I wanna cache all the model output 

### Environment info

datasets:2.5.0
Automatically add filename for image/audio folder
State: open
Labels: name: enhancement

### Feature request

When creating a custom audio of image dataset, it would be great to automatically have access to the filename. It should be both:
a) Automatically displayed in the viewer
b) Automatically added as a column to the dataset when doing `load_dataset`

In `diffusers` our test rely quite heavily on images and audio files now and it's a bit tedious at the moment to download specific images from a datasets repo. 

E.g. we have a dataset of images for tests in `diffusers`: https://huggingface.co/datasets/hf-internal-testing/diffusers-images
where it would be extremely nice to have direct access to the filename both visually on the datasets page (@severo ) as well as via the `load_datasets` function. We currently have some akward functionality to download images by path name: https://github.com/huggingface/diffusers/blob/2fb8fafa4b761f6fc144cf75a6f6f0ea6af3a1c1/src/diffusers/utils/testing_utils.py#L131 
It would be much nicer to just go over `load_dataset(...)` 

### Motivation

Intuitively the filename is something people understand directly. E.g if you upload a folder of images online, it's nice if you recognize the image as well as the filename next to it directly and that you're able to use it right away. 

The label on the other hand is less intuitive to understand as you haven't added it yourself.

### Your contribution

Not sure if I have the time to add it myself anytime soon, but it would help us a lot for `diffusers`.
Fix language and license tag names in all Hub datasets
State: closed
Labels: name: dataset contribution

While working on this:
- #5137

we realized there are still many datasets with deprecated "languages" and "licenses" tag names (instead of "language" and "license").

This is a blocking issue: no subsequent PR can be opened to modify their metadata: a ValueError will be thrown.

We should fix the "language" and "license" tag names in all Hub datasets.

TODO:
- [x] Fix language and license tag names in 402 Hub datasets

CC: @julien-c
Consistent caching between python and jupyter
State: closed
Labels: name: enhancement

### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.


### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen
Unable to download dataset using Azure Data Lake Gen 2
State: closed

### Describe the bug

When using the DatasetBuilder method with the credentials for the cloud storage Azure Data Lake (adl) Gen2, the following error is showed:
```
Traceback (most recent call last):
  File "download_hf_dataset.py", line 143, in <module>
    main()
  File "download_hf_dataset.py", line 102, in main
    builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/datasets/builder.py", line 671, in download_and_prepare
    fs_token_paths = fsspec.get_fs_token_paths(output_dir, storage_options=storage_options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/core.py", line 639, in get_fs_token_paths
    fs = cls(**options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'account_name'
```

If I don't pass the storage_options argument (leave it as None), it requires the credentials used in ADL Gen 1:
`TypeError: __init__() missing 3 required positional arguments: 'tenant_id', 'client_id', and 'client_secret'`

Thus, it is not possible to download a dataset from the cloud using Azure Data Lake (adl) Gen2.

### Steps to reproduce the bug

Assuming that you have an account on Azure and at Storage Account that can be used for reproduce:

1. Create a dict with the format to connect to Azure Data Lake Gen 2
```
storage_options = {"account_name": ACCOUNT_NAME, "account_key": ACCOUNT_KEY)  # gen 2 filesystem
```
2. Create a dataset builder for any HF hosted dataset
```
builder = load_dataset_builder(dataset_name)
```
3. Try to download the dataset passing the storage_options as an argument
```
save_dir = 'adl://my_save_dir'
builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
```

### Expected behavior

Not seeing the error mentioned above and being able to download the dataset to the provided path on ADL

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
default Image/AudioFolder infers labels when there is no metadata files even if there is only one dir
State: closed
Labels: name: bug

### Describe the bug

By default FolderBasedBuilder infers labels if there is not metadata files, even if it's meaningless (for example, they are in a single directory or in the root folder, see this repo as an example: https://huggingface.co/datasets/patrickvonplaten/audios

As this is a corner case for quick exploration of images or audios on the Hub. 

### Steps to reproduce the bug

If you have directory like this:

```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
or

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
```
doing `ds = load_dataset(repo)` would create `label` feature:
```python
print(ds["train"][0])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 0}
```

Also, if you have the following structure:

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
   image4.jpg
   image5.jpg
   image6.jpg
```
it will infer two labels:

```python
print(ds["train"][0])
print(ds["train"][-1])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 1}
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x415 at 0x7FB5326555B0>, 'label': 0}
```

### Expected behavior

We should have only one base feature (Image/Audio) in such cases.

### Environment info

all versions of `datasets`
Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?)
State: open
Labels: name: enhancement

Now one can push only different splits within one default config of a dataset.

Would be nice to allow something like:
```
ds.push_to_hub(repo_name, config=config_name)
```

I'm not sure, but this will probably require changes in `data_files.py` patterns. If so, it would also allow to create different configs for packaged modules datasets.
Problems after upgrading to 2.6.1
State: open

### Describe the bug

Loading a dataset_dict from disk with `load_from_disk` is now creating a `KeyError "length"` that was not occurring in v2.5.2.

Context: 
- Each individual dataset in the dict is created with `Dataset.from_pandas` 
- The dataset_dict is create from a dict of `Dataset`s, e.g., `DatasetDict({"train": train_ds, "validation": val_ds})
- The pandas dataframe, besides text columns, has a column with a dictionary inside and potentially different keys in each row. Correctly the `Dataset.from_pandas` function adds `key: None` to all dictionaries in each row so that the schema can be correctly inferred.

### Steps to reproduce the bug

Steps to reproduce:

- Upgrade to datasets==2.6.1
- Create a dataset from pandas dataframe with `Dataset.from_pandas`
- Create a dataset_dict from a dict of `Dataset`s, e.g., `DatasetDict({"train": train_ds, "validation": val_ds})
- Save to disk with the `save` function

### Expected behavior

Same as in v2.5.2, that is load from disk without errors

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.4.209-129.367.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
Cannot find the rvl_cdip dataset
State: closed

Hi,
I am trying to use load_dataset to load the official "rvl_cdip" dataset but getting an error.

dataset = load_dataset("rvl_cdip")

Couldn't find 'rvl_cdip' on the Hugging Face Hub either: FileNotFoundError: Couldn't find the file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/rvl_cdip/rvl_cdip.py

Regards,
Allow ignoring kwargs inside fn_kwargs during dataset.map's fingerprinting
State: open
Labels: name: enhancement

### Feature request

`dataset.map` accepts a `fn_kwargs` that is passed to `fn`. Currently, the whole `fn_kwargs` is used by `fingerprint_transform` to calculate the new fingerprint.

I'd like to be able to inform `fingerprint_transform` which `fn_kwargs` shoud/shouldn't be taken into account during hashing.

Of course, users should be aware to properly use this new feature, just like the internal usages of `fingerprint_transform` [does](https://github.com/huggingface/datasets/blob/2699593b33ee63d17aad2a2bfddedd38a8df57b8/src/datasets/arrow_dataset.py#L2700).

### Motivation

This is originally motivated by https://github.com/huggingface/transformers/pull/18351#issuecomment-1263588680. 

Nonetheless, consider a more general processing function that accepts a kwarg that does not influence it's output:
```python
def fn(example, verbose=False):
    ...
```

Then `dataset.map(fn, verbose=True)` would not benefit from dataset caching.

I'm not sure if other methods in the `Dataset` API could benefit from this feature.

### Your contribution

Based on `fingerprint_transform `'s `wrapper` function [here](https://github.com/huggingface/datasets/blob/c59cc34fcd2a369d27b77cc678017f5976a926a9/src/datasets/fingerprint.py#L443), it seems to me that it should be possible to make `.map`/`._map_single` accept something like `fn_use_fingerprint_kwargs`/`fn_ignore_fingerprint_kwargs` (probably another arg name). This would then be used by `fingerprint_transform.wrapper` to better/more flexibly hash the transformation.

I could contribute with a PR if this feature and approach look good to you.
Dataset order is not deterministic with ZIP archives and `iter_files`
State: closed

### Describe the bug

For the `beans` dataset (did not try on other), the order of samples is not the same on different machines. Tested on my local laptop, github actions machine, and ec2 instance. The three yield a different order.

### Steps to reproduce the bug

In a clean docker container or conda environment with datasets==2.6.1, run

```python
from datasets import load_dataset
from pprint import pprint

data = load_dataset("beans", split="validation")

pprint(data["image_file_path"])
```

### Expected behavior

The order of the images is the same on all machines.

### Environment info

On the EC2 instance:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.14.291-218.527.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
- Numpy version: not checked
```

On my local laptop:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Numpy version: 1.23.1
```

On github actions:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.2.5
- Python version: 3.8.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
- Numpy version: 1.23.4
```
Inconsistent documentation on map remove_columns
State: closed
Labels: name: documentation, name: duplicate, name: good first issue, name: hacktoberfest

### Describe the bug

The page [process](https://huggingface.co/docs/datasets/process) says this about the parameter `remove_columns` of the function `map`:
When you remove a column, it is only removed after the example has been provided to the mapped function.

So it seems that the `remove_columns` parameter removes after the mapped functions.

However, another page, [the documentation of the function map](https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.Dataset.map.remove_columns) says:
Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding columns with names in remove_columns, these columns will be kept.

So one page says "after the mapped function" and another says "before the mapped function."
Is there something wrong?


### Steps to reproduce the bug

Not about code.

### Expected behavior

consistent about the descriptions of the behavior of the parameter `remove_columns` in the function `map`.

### Environment info

datasets V2.6.0
DownloadManager Git LFS support
State: closed
Labels: name: enhancement

### Feature request

Maybe I'm mistaken but the `DownloadManager` does not support extracting git lfs files out of the box right?
Using `dl_manager.download()` or `dl_manager.download_and_extract()` still returns lfs files afaict. 

Is there a good way to write a dataset loading script for a repo with lfs files?

### Motivation

/

### Your contribution

/
Align task tags in dataset metadata
State: closed
Labels: name: dataset contribution

## Describe
Once we have agreed on a common naming for task tags for all open source projects, we should align on them.

## Steps
- [x] Align task tags in canonical datasets
  - [x] task_categories: 4 datasets
  - [x] task_ids (by @lhoestq)
- [x] Open PRs in community datasets
  - [x] task_categories: 451 datasets
  - [x] task_ids: 556 datasets
Update docs once dataset scripts transferred to the Hub
State: closed

Todo:
- [x] Update docs:
  - [x] Datasets on GitHub (legacy)
  - [x] Load: offline
  - [x] About dataset load:
    - [x] Maintaining integrity
    - [x] Security
- [x] Update docstrings:
  - [x] Inspect:
    - [x] get_dataset_config_info
    - [x] get_dataset_split_names
  - [x] Load:
    - [x] dataset_module_factory
    - [x] load_dataset_builder
    - [x] load_dataset
- [x] Remove `ADD_NEW_DATASET.md`
- [x] Update `.github/ISSUE_TEMPLATE/config.yml`

Fix #5135.
Update docs once dataset scripts transferred to the Hub
State: closed
Labels: name: documentation

## Describe the bug
As discussed in:
- https://github.com/huggingface/hub-docs/pull/423#pullrequestreview-1146083701

we should update our docs once dataset scripts have been transferred to the Hub (and removed from GitHub):
- #4974

Concretely:
- [x] Datasets on GitHub (legacy): https://huggingface.co/docs/datasets/main/en/share#datasets-on-github-legacy
- [x] ADD_NEW_DATASET: https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md
- ...

This PR complements the work of:
- #5067

This PR is a follow-up of PRs:
- #3777

CC: @julien-c