KRadim/github-issues-minimal huggingface dataset:
[docs] troubleshooting guide
State: closed

Hi all! This is a PR adding a troubleshooting guide for Datasets docs. 
I went through the library's GitHub Issues and Forum questions and identified a few issues that are common enough that I think it would be valuable to include them in the troubleshooting guide. These are: 
- creating a dataset from a folder and not following the required format
- authentication issues when using `push_to_hub`
- `Too Many Requests` with `push_to_hub`
- Pickling issues when using Dataset.from_generator()

There's also a section on asking for help. Please let me know if there are other common issues or advice that we can include here.
Fix conda release by adding pyarrow-hotfix dependency
State: closed

Fix conda release by adding pyarrow-hotfix dependency.

Note that conda release failed in latest 2.14.7 release: https://github.com/huggingface/datasets/actions/runs/6874667214/job/18696761723
```
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/test_tmp/run_test.py", line 2, in <module>
    import datasets
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/__init__.py", line 22, in <module>
    from .arrow_dataset import Dataset
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 67, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import Features, Image, Value
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/features/__init__.py", line 18, in <module>
    from .features import Array2D, Array3D, Array4D, Array5D, ClassLabel, Features, Sequence, Value
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/features/features.py", line 34, in <module>
    import pyarrow_hotfix  # noqa: F401  # to fix vulnerability on pyarrow<14.0.1
    ^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pyarrow_hotfix'
```
Allow to choose the `writer_batch_size` when using `save_to_disk`
State: open
Labels: name: enhancement

### Feature request

Add an argument in `save_to_disk` regarding batch size, which would be passed to `shard` and other methods.

### Motivation

The `Dataset.save_to_disk` method currently calls `shard` without passing a `writer_batch_size` argument, thus implicitly using the default value (1000). This can result in RAM saturation when using a lot of processes on long text sequences or other modalities, or for specific IO configs.

### Your contribution

I would be glad to submit a PR, as long as it does not imply extensive tests refactoring.
Set `usedforsecurity=False` in hashlib methods (FIPS compliance)
State: closed

Related to https://github.com/huggingface/transformers/issues/27034 and https://github.com/huggingface/huggingface_hub/pull/1782.

**TL;DR:** `hashlib` is not a secure library for cryptography-related stuff. We are only using `hashlib` for non-security-related purposes in `datasets` so it's fine. From Python 3.9 we set can `usedforsecurity=False` in any `hashlib` method which is mandatory for companies that forbid the use of `hashlib` for security purposes. This PR fixes that.

**Note:** before merging this we need to release a new tokenizers version that would allow the newest `huggingface_hub` version (see https://github.com/huggingface/tokenizers/pull/1385). Otherwise it might create friction to users that want to install `datasets` + `tokenizers` at the same time.
using DownloadManager to download from local filesystem and disable_progress_bar, there will be an exception 
State: closed

### Describe the bug

i'm using datasets.download.download_manager.DownloadManager to download files like "file:///a/b/c.txt", and i disable_progress_bar() to disable bar. there will be an exception as follows:

`AttributeError: 'function' object has no attribute 'close'
Exception ignored in: <function TqdmCallback.__del__ at 0x7fa8683d84c0>
Traceback (most recent call last):
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/fsspec/callbacks.py", line 233, in __del__
    self.tqdm.close()`

i check your source code in datasets/utils/file_utils.py:348 you define TqdmCallback derive from fsspec.callbacks.TqdmCallback

but in the newest fsspec code [https://github.com/fsspec/filesystem_spec/blob/master/fsspec/callbacks.py](url) , line 146, in this case, _DEFAULT_CALLBACK will take effect, but in line 234, it calls "close()" function which _DEFAULT_CALLBACK don't have such thing.

so i think the class "TqdmCallback" in datasets/utils/file_utils.py may override "__del__" function or report this bug to fsspec.



### Steps to reproduce the bug

as i said

### Expected behavior

no exception

### Environment info

datasets: 2.14.4
python: 3.9
platform: x86_64
`IterableDataset` lost but not keep columns when map function adding columns with names in `remove_columns`
State: open

### Describe the bug

IterableDataset lost but not keep columns when map function adding columns with names in remove_columns,
Dataset not.

May be related to the code below:
https://github.com/huggingface/datasets/blob/06c3ffb8d068b6307b247164b10f7c7311cefed4/src/datasets/iterable_dataset.py#L750-L756

### Steps to reproduce the bug

```python
dataset: IterableDataset = load_dataset("Anthropic/hh-rlhf", streaming=True, split="train")
column_names = list(next(iter(dataset)).keys())  #  ['chosen', 'rejected']
# map_fn will return dict {"chosen": xxx, "rejected": xxx, "prompt": xxx, "history": xxxx}
dataset = dataset.map(map_fn, batched=True, remove_columns=column_names)
next(iter(dataset))
# output
# {'prompt': 'xxx, 'history': xxx}   
```

```python
# when load_dataset with streaming=False, the column_names are kept: 
dataset: Dataset = load_dataset("Anthropic/hh-rlhf", streaming=False, split="train")
column_names = list(next(iter(dataset)).keys())  #  ['chosen', 'rejected']
# map_fn will return dict {"chosen": xxx, "rejected": xxx, "prompt": xxx, "history": xxxx}
dataset = dataset.map(map_fn, batched=True, remove_columns=column_names)
next(iter(dataset))
# output
# {'prompt': 'xxx, 'history': xxx, "chosen": xxx, "rejected": xxx}   
```


### Expected behavior

IterableDataset keep columns when map function adding columns with names in remove_columns

### Environment info

datasets==2.14.6
CI Build PR Documentation is broken: ImportError: cannot import name 'TypeAliasType' from 'typing_extensions'
State: closed

Our CI Build PR Documentation is broken. See: https://github.com/huggingface/datasets/actions/runs/6799554060/job/18486828777?pr=6390
```
ImportError: cannot import name 'TypeAliasType' from 'typing_extensions'
```
ConfigNamesError on a simple CSV file
State: closed
Labels: name: bug

See https://huggingface.co/datasets/Nguyendo1999/mmath/discussions/1

```
Error code:   ConfigNamesError
Exception:    TypeError
Message:      __init__() missing 1 required positional argument: 'dtype'
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/dataset/config_names.py", line 65, in compute_config_names_response
                  for config in sorted(get_dataset_config_names(path=dataset, token=hf_token))
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 351, in get_dataset_config_names
                  dataset_module = dataset_module_factory(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1512, in dataset_module_factory
                  raise e1 from None
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1489, in dataset_module_factory
                  return HubDatasetModuleFactoryWithoutScript(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1039, in get_module
                  dataset_infos = DatasetInfosDict.from_dataset_card_data(dataset_card_data)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/info.py", line 468, in from_dataset_card_data
                  dataset_info = DatasetInfo._from_yaml_dict(dataset_card_data["dataset_info"])
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/info.py", line 399, in _from_yaml_dict
                  yaml_data["features"] = Features._from_yaml_list(yaml_data["features"])
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1838, in _from_yaml_list
                  return cls.from_dict(from_yaml_inner(yaml_data))
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1690, in from_dict
                  obj = generate_from_dict(dic)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1345, in generate_from_dict
                  return {key: generate_from_dict(value) for key, value in obj.items()}
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1345, in <dictcomp>
                  return {key: generate_from_dict(value) for key, value in obj.items()}
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1353, in generate_from_dict
                  return class_type(**{k: v for k, v in obj.items() if k in field_names})
              TypeError: __init__() missing 1 required positional argument: 'dtype'
```

This is the CSV file: https://huggingface.co/datasets/Nguyendo1999/mmath/blob/dbcdd7c2c6fc447f852ec136a7532292802bb46f/math_train.csv
Cannot import datasets on google colab (python 3.10.12)
State: closed

### Describe the bug

I'm trying A full colab demo notebook of zero-shot-distillation from https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation but i got this type of error when importing  datasets on my google colab (python version is 3.10.12)

![image](https://github.com/huggingface/datasets/assets/15389235/6f7758a2-681d-4436-87d0-5e557838e368)

I found the same problem that have been solved in [#3326 ] but it seem still error on the google colab. I can't try on my local using jupyter notebook because of my laptop resource doesn't fulfill the requirements. 

Please can anyone help me solve this problem. Thank you 😅 

### Steps to reproduce the bug

Error:

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-8-b6e092f83978>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from datasets import load_dataset
      2 
      3 # Print all the available datasets
      4 from huggingface_hub import list_datasets
      5 print([dataset.id for dataset in list_datasets()])

6 frames
[/usr/lib/python3.10/functools.py](https://localhost:8080/#) in update_wrapper(wrapper, wrapped, assigned, updated)
     59     # Issue #17482: set __wrapped__ last so we don't inadvertently copy it
     60     # from the wrapped function when updating __dict__
---> 61     wrapper.__wrapped__ = wrapped
     62     # Return the wrapper so this can be used as a decorator via partial()
     63     return wrapper

AttributeError: readonly attribute
```

### Expected behavior

Run success on Google Colab (free)

### Environment info

Windows 11 x64, Google Colab free
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array
State: open

### Describe the bug

Hi, I am preprocessing a large custom dataset with numpy arrays. I am running into this TypeError during writing in a dataset.map() function. I've tried decreasing writer batch size, but this error persists. This error does not occur for smaller datasets. 



Thank you!



### Steps to reproduce the bug

Traceback (most recent call last):
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 1354, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3493, in _map_single
    writer.write_batch(batch)
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_writer.py", line 555, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 243, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_writer.py", line 184, in __arrow_array__
    out = numpy_to_pyarrow_listarray(data)
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/features/features.py", line 1394, in numpy_to_pyarrow_listarray
    values = pa.ListArray.from_arrays(offsets, values)
  File "pyarrow/array.pxi", line 2004, in pyarrow.lib.ListArray.from_arrays
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

### Expected behavior

Type should not be a ChunkedArray

### Environment info

datasets v2.14.5
arrow v1.2.3
pyarrow v12.0.1
Raise a different exception for inexisting dataset vs files without known extension
State: closed

See https://github.com/huggingface/datasets-server/issues/2082#issuecomment-1805716557

We have the same error for:
- https://huggingface.co/datasets/severo/a_dataset_that_does_not_exist: a dataset that does not exist
- https://huggingface.co/datasets/severo/test_files_without_extension: a dataset with files without a known extension

```
>>> import datasets
>>> datasets.get_dataset_config_names('severo/a_dataset_that_does_not_exist')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 351, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1508, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /home/slesage/hf/datasets-server/services/worker/severo/a_dataset_that_does_not_exist/a_dataset_that_does_not_exist.py or any data file in the same directory. Couldn't find 'severo/a_dataset_that_does_not_exist' on the Hugging Face Hub either: FileNotFoundError: Dataset 'severo/a_dataset_that_does_not_exist' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
>>> datasets.get_dataset_config_names('severo/test_files_without_extension')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 351, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1508, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /home/slesage/hf/datasets-server/services/worker/severo/test_files_without_extension/test_files_without_extension.py or any data file in the same directory. Couldn't find 'severo/test_files_without_extension' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in severo/test_files_without_extension.
```

To differentiate, we must parse the error message (only the end is different). We should have a different exception for these two errors.
Add ability to set lock type
State: closed
Labels: name: enhancement

### Feature request

Allow setting file lock type, maybe from an environment variable

Currently, it only depends on whether fnctl is available:

https://github.com/huggingface/datasets/blob/12ebe695b4748c5a26e08b44ed51955f74f5801d/src/datasets/utils/filelock.py#L463-L470C16

### Motivation

In my environment, flock isn't supported on a network attached drive

### Your contribution

I'll be happy to submit a pr.
TorchFormatter images (H, W, C) instead of (C, H, W) format
State: closed

### Describe the bug

Using .set_format("torch") leads to images having shape (H, W, C), the same as in numpy. 
However, pytorch normally uses (C, H, W) format.

Maybe I'm missing something but this makes the format a lot less useful as I then have to permute it anyways.
If not using the format it is possible to directly use torchvision transforms but any non-transformed value will not be a tensor.

Is there a reason for this choice?

### Steps to reproduce the bug

```python
from datasets import Dataset, Features, Audio, Image
images = ["path/to/image.png"] * 10
features = Features({"image": Image()})
ds = Dataset.from_dict({"image": images}, features=features) 
ds = ds.with_format("torch")
ds[0]["image"].shape
```
```python
torch.Size([512, 512, 4])
```

### Expected behavior

```python
from datasets import Dataset, Features, Audio, Image
images = ["path/to/image.png"] * 10
features = Features({"image": Image()})
ds = Dataset.from_dict({"image": images}, features=features) 
ds = ds.with_format("torch")
ds[0]["image"].shape
```
```python
torch.Size([4, 512, 512])
```

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-6.5.9-100.fc37.x86_64-x86_64-with-glibc2.31
- Python version: 3.11.6
- Huggingface_hub version: 0.18.0
- PyArrow version: 14.0.1
- Pandas version: 2.1.2
Filter occasionally hangs
State: closed

### Describe the bug

A call to `.filter` occasionally hangs (after the filter is complete, according to tqdm)

There is a trace produced
```
Exception ignored in: <function Dataset.__del__ at 0x7efb48130c10>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/datasets/arrow_dataset.py", line 1366, in __del__
    if hasattr(self, "_indices"):
  File "/usr/lib/python3/dist-packages/composer/core/engine.py", line 123, in sigterm_handler
    sys.exit(128 + signal)
SystemExit: 143
```

but I'm not sure if the trace is actually from `datasets`, or from surrounding code that is trying to clean up after datasets gets stuck.

Unfortunately I can't reproduce this issue anywhere close to reliably. It happens infrequently when using `num_procs > 1`. Anecdotally I started seeing it when using larger datasets (~10M samples).

### Steps to reproduce the bug

N/A see description

### Expected behavior

map/filter calls always complete sucessfully

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.2
Webdataset dataset builder
State: closed

Allow `load_dataset` to support the Webdataset format.

It allows users to download/stream data from local files or from the Hugging Face Hub.

Moreover it will enable the Dataset Viewer for Webdataset datasets on HF.

## Implementation details

- I added a new Webdataset builder
- dataset with TAR files are now read using the Webdataset builder
- Basic decoding from `webdataset` is used by default, except unsafe ones like pickle
- HF authentication support is done with `xopen`

## TODOS

- [x] tests
- [x] docs
handle future deprecation argument
State: closed

getting this error:
```
/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/datasets/table.py:1387: FutureWarning: promote has been superseded by mode='default'.
  return cls._concat_blocks(pa_tables_to_concat_vertically, axis=0)
```

Since datasets supports arrow greater than 8.0.0, we need to handle both cases.

[Arrow v14 docs](https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html)
[Arrow v13 docs](https://arrow.apache.org/docs/13.0/python/generated/pyarrow.concat_tables.html)
How to create 3d medical imgae dataset?
State: open
Labels: name: enhancement

### Feature request

I am newer to huggingface, after i look up `datasets` docs, I can't find how to create the dataset contains 3d medical image (ends with '.mhd', '.dcm', '.nii')

### Motivation

help us to upload 3d medical dataset to huggingface!

### Your contribution

I'll submit a PR if I find a way to add this feature
How to load existing downloaded dataset ?
State: closed
Labels: name: enhancement

Hi @mariosasko @lhoestq  @katielink 

Thanks for your contribution and hard work.

### Feature request

First, I download a dataset as normal by:
```
from datasets import load_dataset

dataset = load_dataset('username/data_name', cache_dir='data')
```

The dataset format in `data` directory will be:
```
-data
  |-data_name
    |-test-00000-of-00001-bf4c733542e35fcb.parquet
    |-train-00000-of-00001-2a1df75c6bce91ab.parquet
```


Then I use SCP to clone this dataset into another machine, and then try:
```
from datasets import load_dataset

dataset = load_dataset('data/data_name')  # load from local path
```

This leads to re-generating training and validation split for each time, and the disk quota will be duplicated occupation.

How can I just load the dataset without generating and saving these splits again?

### Motivation

I do not want to download the same dataset in two machines, scp is much faster and better than HuggingFace API. I hope we can directly load the downloaded datasets (.parquest)

### Your contribution

Please refer to the feature
Formatting overhead
State: closed

### Describe the bug

Hi! I very recently noticed that my training time is dominated by batch formatting. Using Lightning's profilers, I located the bottleneck within `datasets.formatting.formatting` and then narrowed it down with `line-profiler`. It turns out that almost all of the overhead is due to creating new instances of `self.python_arrow_extractor`. I admit I'm confused why that could be the case - as far as I can tell there's no complex `__init__` logic to execute.

![image](https://github.com/huggingface/datasets/assets/320321/5e022e0b-0d21-43d0-8e6f-9e641142e96b)

### Steps to reproduce the bug

1. Set up a dataset `ds` with potentially several (4+) columns (not sure if this is necessary, but it did at one point of the investigation make overhead worse)
2. Process it using a custom transform, `ds = ds.with_transform(transform_func)`
3. Decorate this function https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py#L512 with `@profile` from https://pypi.org/project/line-profiler/
4. Profile with `$ kernprof -l script_to_profile.py`

### Expected behavior

Batch formatting should have acceptable overhead.

### Environment info

```
datasets=2.14.6
pyarrow=14.0.0
```
Get an error when i try to concatenate the squad dataset with my own dataset
State: closed

### Describe the bug

Hello,

I'm new here and I need to concatenate the squad dataset with my own dataset i created. I find the following error when i try to do it: Traceback (most recent call last):

  Cell In[9], line 1
    concatenated_dataset = concatenate_datasets([train_dataset, dataset1])

  File ~\anaconda3\Lib\site-packages\datasets\combine.py:213 in concatenate_datasets
    return _concatenate_map_style_datasets(dsets, info=info, split=split, axis=axis)

  File ~\anaconda3\Lib\site-packages\datasets\arrow_dataset.py:6002 in _concatenate_map_style_datasets
    _check_if_features_can_be_aligned([dset.features for dset in dsets])

  File ~\anaconda3\Lib\site-packages\datasets\features\features.py:2122 in _check_if_features_can_be_aligned
    raise ValueError(

ValueError: The features can't be aligned because the key answers of features {'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answers': Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)} has unexpected type - Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None) (expected either {'answer_start': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'text': Value(dtype='string', id=None)} or Value("null").

### Steps to reproduce the bug

```python

from huggingface_hub import notebook_login
from datasets import load_dataset



notebook_login("mymailadresse", "mypassword")
squad = load_dataset("squad", split="train[:5000]")
squad = squad.train_test_split(test_size=0.2)
dataset1 = squad["train"]




import json

mybase = [
    {
        "id": "1",
        "context": "She lives in Nantes",
        "question": "Where does she live?",
        "answers": {
            "text": "Nantes",
            "answer_start": [13],
        }
    }
]




# Save the data to a JSON file
json_file_path = r"C:\Users\mypath\thefile.json"
with open(json_file_path, "w", encoding= "utf-8") as json_file:
    json.dump(mybase, json_file, indent=4)




# Load the JSON file as a dataset
custom_dataset = load_dataset("json", data_files=json_file_path)


# Access the train split
train_dataset = custom_dataset["train"]


from datasets import concatenate_datasets


# Concatenate the datasets
concatenated_dataset = concatenate_datasets([train_dataset, dataset1])
```

### Expected behavior

I would expect the two datasets to be concatenated without error. The len(dataset1) is equal to 4000 and the len(train_dataset) is equal to 1 so I would exepect concatenated_dataset to be created and having lenght 4001.

### Environment info

Python 3.11.4 and using windows

Thank you for your help
imagenet-1k downloads over and over
State: closed

### Describe the bug

What could be causing this?
```
$ python3
Python 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> load_dataset("imagenet-1k")
Downloading builder script: 100%|██████████| 4.72k/4.72k [00:00<00:00, 7.51MB/s]
Downloading readme: 100%|███████████████████| 85.4k/85.4k [00:00<00:00, 510kB/s]
Downloading extra modules: 100%|████████████| 46.4k/46.4k [00:00<00:00, 300kB/s]
Downloading data: 100%|████████████████████| 29.1G/29.1G [19:36<00:00, 24.8MB/s]
Downloading data: 100%|████████████████████| 29.3G/29.3G [08:38<00:00, 56.5MB/s]
Downloading data: 100%|████████████████████| 29.0G/29.0G [09:26<00:00, 51.2MB/s]
Downloading data: 100%|████████████████████| 29.2G/29.2G [09:38<00:00, 50.6MB/s]
Downloading data: 100%|███████████████████▉| 29.2G/29.2G [09:37<00:00, 44.1MB/s^Downloading data:   0%|                     | 106M/29.1G [00:05<23:49, 20.3MB/s]
```

### Steps to reproduce the bug

See above commands/code

### Expected behavior

imagenet-1k is downloaded

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-6.2.0-34-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- PyArrow version: 14.0.0
- Pandas version: 1.5.2
Add CheXpert dataset for vision
State: open
Labels: name: enhancement, name: dataset request

### Feature request

### Name 

**CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison**

### Paper

https://arxiv.org/abs/1901.07031

### Data

https://stanfordaimi.azurewebsites.net/datasets/8cbd9ed4-2eb9-4565-affc-111cf4f7ebe2

### Motivation

CheXpert is one of the fundamental models in medical image classification and can serve as a viable pre-training dataset for radiology classification or low-scale ablation / exploratory studies.

This could also serve as a good pre-training dataset for Kaggle competitions.

### Your contribution

Would love to make a PR and pre-process / get this into 🤗
Add my dataset
State: closed

## medical data

**Description:**
This dataset, named "medical data," is a collection of text data from various sources, carefully curated and cleaned for use in natural language processing (NLP) tasks. It consists of a diverse range of text, including articles, books, and online content, covering topics from science to literature.

**Citation:**
If applicable, please include a citation for this dataset to give credit to the original sources or contributors.

**Key Features:**
- Language: The text is primarily in English, but it may include content in other languages as well.
- Use Cases: This dataset is suitable for text classification, language modeling, sentiment analysis, and other NLP tasks.

**Usage:**
To access this dataset, use the `load_your_dataset` function provided in the `your_dataset.py` script within this repository. You can specify the dataset split you need, such as "train," "test," or "validation," to get the data for your specific task.

**Contributors:**
- [Keyur Chaudhari]

**Contact:**
If you have any questions or need assistance regarding this dataset, please feel free to contact [keyurchaudhari536@gmail.com].

Please note that this dataset is shared under a specific license, which can be found in the [LICENSE](link to your dataset's license) file. Make sure to review and adhere to the terms of the license when using this dataset for your projects.
Fix for continuation behaviour on broken dataset archives due to starving download connections via HTTP-GET
State: open

This PR proposes a (slightly hacky) fix for an Issue that can occur when downloading large dataset parts over unstable connections.
The underlying issue is also being discussed in https://github.com/huggingface/datasets/issues/5594.

Issue Symptoms & Behaviour:
- Download of a large archive file during dataset download via HTTP-GET fails.
- An silent net exception (which I was unable to identify) is thrown within the `tqdm` download progress.
- Due to missing exception catch code, the above process just continues processing, assuming `http_get` completed successfully.
- Pending Archive file gets renamed to remove the `.incomplete` extension, despite not all data has been downloaded. 
- Also, for reasons I did not investigate, there seems to be no real integrity check for the downloaded files; or it does not detect this problem. This is especially problematic, since the downloader script won't retry downloading this archive after CRC-Checking, even if it is being manually restarted / executed again after running into errors on extraction.

Fix proposal: Adding a retry mechanic for HTTP-GET downloads, which adds the following behaviour:
- Download Progress Thread checks for download size validity in case the HTTP connection starves mid download. If the check fails, a RuntimeError is thrown
- Cache Downloader code with retry mechanic monitors for an exception thrown by the download progress thread, and retries download with updated `resume_size`.
- Cache Downloader will not mark incomplete files which have thrown an exception during download, and exceeded retries, as complete.
Caching problem when deleting a dataset 
State: closed

### Describe the bug

Pushing a dataset with n + m features to a repo which was deleted, but contained n features, will fail.

### Steps to reproduce the bug

1. Create a dataset with n features per row
2. `dataset.push_to_hub(YOUR_PATH, SPLIT, token=TOKEN)`
3. Go on the hub, delete the repo at `YOUR_PATH`
4. Update your local dataset to have n + m features per row
5. `dataset.push_to_hub(YOUR_PATH, SPLIT, token=TOKEN)` will fail because of a mismatch in features number


### Expected behavior

Step 5 should work or display a message to indicate the cache has not been cleared

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
CI is broken: TypeError: Couldn't cast array
State: closed

See: https://github.com/huggingface/datasets/actions/runs/6730567226/job/18293518039
```
FAILED tests/test_table.py::test_cast_sliced_fixed_size_array_to_features - TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[3]
to
Sequence(feature=Value(dtype='int64', id=None), length=3, id=None)
```
`Dataset.from_generator` should not try to download from HF GCS
State: closed

### Describe the bug

When using [`Dataset.from_generator`](https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/arrow_dataset.py#L1072) with `streaming=False`, the internal logic will call [`download_and_prepare`](https://github.com/huggingface/datasets/blob/main/src/datasets/io/generator.py#L47) which will attempt to download from HF GCS which is redundant, because user has already provided the generator from which the data should be drawn. 

If someone attempts to call `Dataset.from_generator` from an environment that doesn't have external internet access (for example internal production machine) and doesn't set `HF_DATASETS_OFFLINE=1`, this will result in process being stuck at building connection.

### Steps to reproduce the bug

```python
import datasets
def gen():
    for _ in range(100):
        yield {"text": "dummy text"}

dataset = datasets.Dataset.from_generator(gen)
```

A minimum example executed on any environment that doesn't have access to HF GCS can result in the error

### Expected behavior

`try_from_hf_gcs` should be set to False here https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/io/generator.py#L51

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.12
- Huggingface_hub version: 0.17.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
TensorDataset format does not work with Trainer from transformers
State: closed

### Describe the bug

The model was built to do fine tunning on BERT model for relation extraction.

trainer.train() returns an error message ```TypeError: vars() argument must have __dict__ attribute``` when it has `train_dataset` generated from `torch.utils.data.TensorDataset`

However, in the document, the required data format is `torch.utils.data.TensorDataset`. 
![image](https://github.com/huggingface/datasets/assets/49014051/36fa34ac-3127-4c64-9580-9ab736136d83)

Transformers trainer is supposed to accept the train_dataset in the format of torch.utils.data.TensorDataset, but it returns error message *"TypeError: vars() argument must have __dict__ attribute"*

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-5df728c929a2> in <cell line: 1>()
----> 1 trainer.train()
      2 trainer.evaluate(test_dataset)

9 frames
/usr/local/lib/python3.10/dist-packages/transformers/data/data_collator.py in <listcomp>(.0)
    107 
    108     if not isinstance(features[0], Mapping):
--> 109         features = [vars(f) for f in features]
    110     first = features[0]
    111     batch = {}

TypeError: vars() argument must have __dict__ attribute
```

### Steps to reproduce the bug

Create train_dataset using `torch.utils.data.TensorDataset`, for instance,

```train_dataset = torch.utils.data.TensorDataset(train_input_ids, train_attention_masks, train_labels)```

Feed this `train_dataset` to your trainer and run trainer.train

```
trainer = Trainer(model,
                  training_args,
                  train_dataset=train_dataset,
                  eval_dataset=dev_dataset,
                  compute_metrics=compute_metrics,
                  )
```

### Expected behavior

Trainer should start training

### Environment info

It is running on Google Colab

- `datasets` version: 2.14.6
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
Multi process map did not load cache file correctly
State: closed

### Describe the bug

When I  was training model on Multiple GPUs by DDP, the dataset is tokenized multiple times after main process. 
![1698820541284](https://github.com/huggingface/datasets/assets/14285786/0b2fe054-54d8-4e00-96e6-6ca5b69e662b)

![1698820501568](https://github.com/huggingface/datasets/assets/14285786/dd62bf6f-a58f-41bf-9848-ea4fb3b62b9b)

Code is modified from  [run_clm.py](https://github.com/huggingface/transformers/blob/7d8ff3629b2725ec43ace99c1a6e87ac1978d433/examples/pytorch/language-modeling/run_clm.py#L484)


### Steps to reproduce the bug

```
block_size = data_args.block_size
IGNORE_INDEX = -100
Ignore_Input = False

def tokenize_function(examples):
    sources = []
    targets = []
    for instruction, inputs, output in zip(examples['instruction'], examples['input'], examples['output']):

        source = instruction + inputs 

        target = f"{output}{tokenizer.eos_token}"

        sources.append(source)
        targets.append(target)

    tokenized_sources = tokenizer(sources, return_attention_mask=False)

    tokenized_targets = tokenizer(targets, return_attention_mask=False,
                                  add_special_tokens=False
                                  )

    all_input_ids = []
    all_labels = []
    for s, t in zip(tokenized_sources['input_ids'], tokenized_targets['input_ids']):
        if len(s) > block_size and Ignore_Input == False:
            # print(s)
            continue
        input_ids = torch.LongTensor(s + t)[:block_size]
        if Ignore_Input:
            labels = torch.LongTensor([IGNORE_INDEX] * len(s) + t)[:block_size]
        else:
            labels = input_ids
        assert len(input_ids) == len(labels)
        all_input_ids.append(input_ids)
        all_labels.append(labels)

    results = {
        'input_ids': all_input_ids,
        'labels': all_labels,

    }
    return results



with training_args.main_process_first(desc="dataset map tokenization ", local=False):
    # print('local_rank',training_args.local_rank)
    if not data_args.streaming:
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on dataset ",
        )
    else:
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            remove_columns=column_names,
            desc="Running tokenizer on dataset "
        )
```

### Expected behavior

This code should only tokenize the dataset in the main process, and the other processes load the dataset after waiting

### Environment info

transformers == 4.34.1
datasets == 2.14.5
with_format() function returns bytes instead of PIL images even when image column is not part of "columns"
State: closed

### Describe the bug

When using the with_format() function on a dataset containing images, even if the image column is not part of the columns provided in the function, its type will be changed to bytes.
Here is a minimal reproduction of the bug:
https://colab.research.google.com/drive/1hyaOspgyhB41oiR1-tXE3k_gJCdJUQCf?usp=sharing

### Steps to reproduce the bug

1. Load the image dataset
2. apply with_format(columns=["text"])
3. Check the type of images in the "image" column before and after applying with_format

### Expected behavior

The type should stay the same, but it does not

### Environment info

datasets==2.14.6
Parquet size grows exponential for categorical data
State: closed

### Describe the bug

It seems that when saving a data frame with a categorical column inside the size can grow exponentially.

This seems to happen because when we save the categorical data to parquet, we are saving the data + all the categories existing in the original data. This happens even when the categories are not present in the original data.


### Steps to reproduce the bug

To reproduce the bug, it is enough to run this script:

```
import pandas as pd
import os

if __name__ == "__main__":
    for n in [10, 1e2, 1e3, 1e4, 1e5]:
        for n_col in [1, 10, 100, 1000, 10000]:
            input = pd.DataFrame([{"{i}": f"{i}_cat" for col in range(n_col)} for i in range(int(n))])
            input.iloc[0:100].to_parquet("a.parquet")
            for col in input.columns:
                input[col] = input[col].astype("category")
            input.iloc[0:100].to_parquet("b.parquet")
            a_size_mb = os.stat("a.parquet").st_size / (1024 * 1024)
            b_size_mb = os.stat("b.parquet").st_size / (1024 * 1024)
            print(f"{n} {n_col} {a_size_mb} {b_size_mb} {100*b_size_mb/a_size_mb:.2f}")
```

That produces this output:

<img width="464" alt="Screenshot 2023-10-31 at 11 25 25" src="https://github.com/huggingface/datasets/assets/82567957/2b8a9284-7f9e-4c10-a006-0a27236ebd15">



### Expected behavior

In my opinion either:

1. The two file should have (almost) the same size
2. There should be warning telling the user that such difference in size is possible

### Environment info

Python 3.8.18
pandas==2.0.3
numpy==1.24.4
Add support for `Sequence(Audio/Image)` feature in `push_to_hub`
State: closed
Labels: name: enhancement

### Feature request

Allow for `Sequence` of `Image` (or `Audio`) to be embedded inside the shards.

### Motivation

Currently, thanks to #3685, when `embed_external_files` is set to True (which is the default) in `push_to_hub`, features of type `Image` and `Audio` are embedded inside the arrow/parquet shards, instead of only storing paths to the files.

I've noticed that this behavior does not extend to `Sequence` of `Image`, when working with a [dataset of timelapse images](https://huggingface.co/datasets/1aurent/Human-Embryo-Timelapse).

### Your contribution

I'll submit a PR if I find a way to add this feature
Stuck in "Resolving data files..."
State: open

### Describe the bug

I have an image dataset with 300k images, the size of image is 768 * 768.

When I run `dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')` in second time, it takes 50 minutes to finish "Resolving data files" part, what's going on in this part?

From my understand, after Arrow files been created in the first run, the second run should not take time longer than one or two minutes.

### Steps to reproduce the bug

# Run following code two times
dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')

### Expected behavior

Fast dataset building

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
Mounting datasets cache fails due to absolute paths.
State: closed

### Describe the bug

Creating a datasets cache and mounting this into, for example, a docker container, renders the data unreadable due to absolute paths written into the cache.

### Steps to reproduce the bug

1. Create a datasets cache by downloading some data
2. Mount the dataset folder into a docker container or remote system.
3. (Edit) Set `HF_HOME` or `HF_DATASET_CACHE` to point to the mounted cache.
4. Attempt to access the data from within the docker container.
5. An error is thrown saying no file exists at \<absolute path to original cache location\>

### Expected behavior

The data is loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3
Allow passing a multiprocessing context to functions that support `num_proc`
State: open
Labels: name: enhancement

### Feature request

Allow specifying [a multiprocessing context](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) to functions that support `num_proc` or use multiprocessing pools. For example, the following could be done:

```python
dataset = dataset.map(_func, num_proc=2, mp_context=multiprocess.get_context("spawn"))
```

Or at least the multiprocessing start method ("fork", "spawn", "fork_server" or `None`):

```python
dataset = dataset.map(_func, num_proc=2, mp_start_method="spawn")
```

Another option could be passing the `pool` as an argument.

### Motivation

By default, `multiprocess` (the `multiprocessing`-fork library that this repo uses) uses the "fork" start method for multiprocessing pools (for the default context). It could be changed by using `set_start_method`. However, this conditions the multiprocessing start method from all processing in a Python program that uses the default context, because [you can't call that function more than once](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods:~:text=set_start_method()%20should%20not%20be%20used%20more%20than%20once%20in%20the%20program.). My proposal is to allow using a different multiprocessing context, not to condition the whole Python program.

One reason to change the start method is that "fork" (the default) makes child processes likely deadlock if thread pools were created before (and also this is not supported by POSIX). For example, this happens when using PyTorch because OpenMP threads are used for CPU intra-op parallelism, which is enabled by default (e.g., for context see [`torch.set_num_threads`](https://pytorch.org/docs/stable/generated/torch.set_num_threads.html)). This can also be fixed by setting `torch.set_num_threads(1)` (or similarly by other methods) but this conditions the whole Python program as it can only be set once to guarantee its behavior (similarly to). There are noticeable performance differences when setting this number to 1 even when using GPU(s). Using, e.g., a "spawn" start method would solve this issue.

For more context, see:

* https://discuss.huggingface.co/t/dataset-map-stuck-with-torch-set-num-threads-set-to-2-or-larger/37984
* https://discuss.huggingface.co/t/using-num-proc-1-in-dataset-map-hangs/44310

### Your contribution

I'd be happy to review a PR that makes such a change. And if you really don't have the bandwidth for it, I'd consider creating one.