noamaanMulla-03/datasets-issues huggingface dataset:
Only two cores are getting used in sagemaker with pytorch 3.10 kernel

### Describe the bug

When using the newer pytorch 3.10 kernel, only 2 cores are being used by huggingface filter and map functions.  The Pytorch 3.9 kernel would use as many cores as specified in the num_proc field.

We have solved this in our own code by placing the following snippet in the code that is called inside subprocesses:

```os.sched_setaffinity(0, {i for i in range(1000)})```

The problem, as near as we can tell, us that once upon a time, cpu affinity was set using a bitmask ("0xfffff" and the like), and affinity recently changed to a list of processors rather than to using the mask.  As such, only processors 1 and 17 are shown to be working in htop.
![Selection_072](https://github.com/huggingface/datasets/assets/107141022/04c5a824-5321-4531-afca-7bc84dff36b4)


When running functions via `map`, the above resetting of affinity works to spread across the cores.  When using `filter`, however, only two cores are active.

### Steps to reproduce the bug

Repro steps:

1.  Create an aws sagemaker instance
2. use the pytorch 3_10 kernel
3. Load a dataset
4. run a filter operation
5. watch as only 2 cores are used when num_proc > 2
6. run a map operation
7. watch as only 2 cores are used when num_proc > 2
8. run a map operation with processor affinity reset inside the function called via map
9. Watch as all cores run



### Expected behavior

All specified cores are used via the num_proc argument.

### Environment info

AWS sagemaker with the following init script run in the terminal after instance creation:

conda init bash
bash
conda activate pytorch_p310
pip install Wand PyPDF pytesseract datasets seqeval pdfplumber transformers pymupdf sentencepiece timm donut-python accelerate optimum xgboost
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
sudo yum -y install htop
sudo yum -y update
sudo yum -y install wget libstdc++ autoconf automake libtool autoconf-archive pkg-config gcc gcc-c++ make libjpeg-devel libpng-devel libtiff-devel zlib-devel


I think it's more likely that this issue is related to PyTorch than Datasets, as PyTorch (on import) registers functions to execute when forking a process. Maybe this is the culprit: https://github.com/pytorch/pytorch/issues/99625

From reading that ticket, it may be down in mkl?  Is it worth hotfixing in the meantime, with the express intention of turning it off?  I know that's a horribly crufty solution, but it's also deeply frustrating to be limited to 2 cores for operations as simple as filtration.

This is too specific and unrelated to `datasets`, so this shouldn't be fixed here.

@mariosasko @mmr-crexi  I had the exact same problem on my kubernetes cluster. the datasets subprocess only user 1 and 17 core
Use arrow ipc file format

According to the [documentation](https://arrow.apache.org/docs/format/Columnar.html?highlight=arrow1#ipc-file-format), it's identical to the streaming format except that it contains the memory offsets of each sample:

> We define a “file format” supporting random access that is build with the stream format. The file starts and ends with a magic string ARROW1 (plus padding). What follows in the file is identical to the stream format. At the end of the file, we write a footer containing a redundant copy of the schema (which is a part of the streaming format) plus memory offsets and sizes for each of the data blocks in the file. This enables random access any record batch in the file. See File.fbs for the precise details of the file footer.

Since it stores more metadata regarding the positions of the examples in the file, it should enable better example retrieval performances. However from the discussion in https://github.com/huggingface/datasets/issues/1803 it looks like it's not the case unfortunately. Maybe in the future this will allow speed gains.

I think it's still a good idea to start using it anyway for these reasons:
- in the future we may have speed gains
- it contains the arrow streaming format data
- it's compatible with the pyarrow Dataset implementation (it allows to load remote dataframes for example) if we want to use it in the future
- it's also the format used by arrow feather if we want to use it in the future
- it's roughly the same size as the streaming format
- it's easy to have backward compatibility with the streaming format




Should we close this PR?

Yes, this one was mostly related to https://github.com/huggingface/datasets/issues/4542 but now I think the TF support is not needed at the moment.

What about enabling the Arrow IPC format through an environment variable/config? Would be very helpful to better interop. with other libraries (e.g. [polars](https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.scan_ipc.html)) + supporting reads through pyarrow datasets.
Parquet size grows exponential for categorical data

### Describe the bug

It seems that when saving a data frame with a categorical column inside the size can grow exponentially.

This seems to happen because when we save the categorical data to parquet, we are saving the data + all the categories existing in the original data. This happens even when the categories are not present in the original data.


### Steps to reproduce the bug

To reproduce the bug, it is enough to run this script:

```
import pandas as pd
import os

if __name__ == "__main__":
    for n in [10, 1e2, 1e3, 1e4, 1e5]:
        for n_col in [1, 10, 100, 1000, 10000]:
            input = pd.DataFrame([{"{i}": f"{i}_cat" for col in range(n_col)} for i in range(int(n))])
            input.iloc[0:100].to_parquet("a.parquet")
            for col in input.columns:
                input[col] = input[col].astype("category")
            input.iloc[0:100].to_parquet("b.parquet")
            a_size_mb = os.stat("a.parquet").st_size / (1024 * 1024)
            b_size_mb = os.stat("b.parquet").st_size / (1024 * 1024)
            print(f"{n} {n_col} {a_size_mb} {b_size_mb} {100*b_size_mb/a_size_mb:.2f}")
```

That produces this output:

<img width="464" alt="Screenshot 2023-10-31 at 11 25 25" src="https://github.com/huggingface/datasets/assets/82567957/2b8a9284-7f9e-4c10-a006-0a27236ebd15">



### Expected behavior

In my opinion either:

1. The two file should have (almost) the same size
2. There should be warning telling the user that such difference in size is possible

### Environment info

Python 3.8.18
pandas==2.0.3
numpy==1.24.4


Wrong repo.
CI is broken: TypeError: Couldn't cast array

See: https://github.com/huggingface/datasets/actions/runs/6730567226/job/18293518039
```
FAILED tests/test_table.py::test_cast_sliced_fixed_size_array_to_features - TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[3]
to
Sequence(feature=Value(dtype='int64', id=None), length=3, id=None)
```
with_format() function returns bytes instead of PIL images even when image column is not part of "columns"

### Describe the bug

When using the with_format() function on a dataset containing images, even if the image column is not part of the columns provided in the function, its type will be changed to bytes.
Here is a minimal reproduction of the bug:
https://colab.research.google.com/drive/1hyaOspgyhB41oiR1-tXE3k_gJCdJUQCf?usp=sharing

### Steps to reproduce the bug

1. Load the image dataset
2. apply with_format(columns=["text"])
3. Check the type of images in the "image" column before and after applying with_format

### Expected behavior

The type should stay the same, but it does not

### Environment info

datasets==2.14.6



Thanks for reporting! I've opened a PR with a fix.
`Dataset.from_generator` should not try to download from HF GCS

### Describe the bug

When using [`Dataset.from_generator`](https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/arrow_dataset.py#L1072) with `streaming=False`, the internal logic will call [`download_and_prepare`](https://github.com/huggingface/datasets/blob/main/src/datasets/io/generator.py#L47) which will attempt to download from HF GCS which is redundant, because user has already provided the generator from which the data should be drawn. 

If someone attempts to call `Dataset.from_generator` from an environment that doesn't have external internet access (for example internal production machine) and doesn't set `HF_DATASETS_OFFLINE=1`, this will result in process being stuck at building connection.

### Steps to reproduce the bug

```python
import datasets
def gen():
    for _ in range(100):
        yield {"text": "dummy text"}

dataset = datasets.Dataset.from_generator(gen)
```

A minimum example executed on any environment that doesn't have access to HF GCS can result in the error

### Expected behavior

`try_from_hf_gcs` should be set to False here https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/io/generator.py#L51

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.12
- Huggingface_hub version: 0.17.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3


Indeed, setting `try_from_gcs` to `False` makes sense for `from_generator`.

We plan to deprecate and remove `try_from_hf_gcs`  soon, as we can use Hub for file hosting now, but this is a good temporary fix.
Fix for continuation behaviour on broken dataset archives due to starving download connections via HTTP-GET

This PR proposes a (slightly hacky) fix for an Issue that can occur when downloading large dataset parts over unstable connections.
The underlying issue is also being discussed in https://github.com/huggingface/datasets/issues/5594.

Issue Symptoms & Behaviour:
- Download of a large archive file during dataset download via HTTP-GET fails.
- An silent net exception (which I was unable to identify) is thrown within the `tqdm` download progress.
- Due to missing exception catch code, the above process just continues processing, assuming `http_get` completed successfully.
- Pending Archive file gets renamed to remove the `.incomplete` extension, despite not all data has been downloaded. 
- Also, for reasons I did not investigate, there seems to be no real integrity check for the downloaded files; or it does not detect this problem. This is especially problematic, since the downloader script won't retry downloading this archive after CRC-Checking, even if it is being manually restarted / executed again after running into errors on extraction.

Fix proposal: Adding a retry mechanic for HTTP-GET downloads, which adds the following behaviour:
- Download Progress Thread checks for download size validity in case the HTTP connection starves mid download. If the check fails, a RuntimeError is thrown
- Cache Downloader code with retry mechanic monitors for an exception thrown by the download progress thread, and retries download with updated `resume_size`.
- Cache Downloader will not mark incomplete files which have thrown an exception during download, and exceeded retries, as complete.
Formatting overhead

### Describe the bug

Hi! I very recently noticed that my training time is dominated by batch formatting. Using Lightning's profilers, I located the bottleneck within `datasets.formatting.formatting` and then narrowed it down with `line-profiler`. It turns out that almost all of the overhead is due to creating new instances of `self.python_arrow_extractor`. I admit I'm confused why that could be the case - as far as I can tell there's no complex `__init__` logic to execute.

![image](https://github.com/huggingface/datasets/assets/320321/5e022e0b-0d21-43d0-8e6f-9e641142e96b)

### Steps to reproduce the bug

1. Set up a dataset `ds` with potentially several (4+) columns (not sure if this is necessary, but it did at one point of the investigation make overhead worse)
2. Process it using a custom transform, `ds = ds.with_transform(transform_func)`
3. Decorate this function https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py#L512 with `@profile` from https://pypi.org/project/line-profiler/
4. Profile with `$ kernprof -l script_to_profile.py`

### Expected behavior

Batch formatting should have acceptable overhead.

### Environment info

```
datasets=2.14.6
pyarrow=14.0.0
```


Ah I think the `line-profiler` log is off-by-one and it is in fact the `extract_batch` method that's taking forever. Will investigate further.

I tracked it down to a quirk of my setup. Apologies.
How to create 3d medical imgae dataset?

### Feature request

I am newer to huggingface, after i look up `datasets` docs, I can't find how to create the dataset contains 3d medical image (ends with '.mhd', '.dcm', '.nii')

### Motivation

help us to upload 3d medical dataset to huggingface!

### Your contribution

I'll submit a PR if I find a way to add this feature

Labels: name: enhancement
Add my dataset

## medical data

**Description:**
This dataset, named "medical data," is a collection of text data from various sources, carefully curated and cleaned for use in natural language processing (NLP) tasks. It consists of a diverse range of text, including articles, books, and online content, covering topics from science to literature.

**Citation:**
If applicable, please include a citation for this dataset to give credit to the original sources or contributors.

**Key Features:**
- Language: The text is primarily in English, but it may include content in other languages as well.
- Use Cases: This dataset is suitable for text classification, language modeling, sentiment analysis, and other NLP tasks.

**Usage:**
To access this dataset, use the `load_your_dataset` function provided in the `your_dataset.py` script within this repository. You can specify the dataset split you need, such as "train," "test," or "validation," to get the data for your specific task.

**Contributors:**
- [Keyur Chaudhari]

**Contact:**
If you have any questions or need assistance regarding this dataset, please feel free to contact [keyurchaudhari536@gmail.com].

Please note that this dataset is shared under a specific license, which can be found in the [LICENSE](link to your dataset's license) file. Make sure to review and adhere to the terms of the license when using this dataset for your projects.



Hi! We do not host datasets in this repo. Instead, you should use `dataset.push_to_hub` to upload the dataset to the HF Hub.

@mariosasko could you provide me proper guide to push data on HF hub 

You can find this info here: https://huggingface.co/docs/datasets/upload_dataset. Also, check https://huggingface.co/docs/datasets/loading for how to load a local dataset (before pushing it to the Hub).
Cannot import datasets on google colab (python 3.10.12)

### Describe the bug

I'm trying A full colab demo notebook of zero-shot-distillation from https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation but i got this type of error when importing  datasets on my google colab (python version is 3.10.12)

![image](https://github.com/huggingface/datasets/assets/15389235/6f7758a2-681d-4436-87d0-5e557838e368)

I found the same problem that have been solved in [#3326 ] but it seem still error on the google colab. I can't try on my local using jupyter notebook because of my laptop resource doesn't fulfill the requirements. 

Please can anyone help me solve this problem. Thank you 😅 

### Steps to reproduce the bug

Error:

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-8-b6e092f83978>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from datasets import load_dataset
      2 
      3 # Print all the available datasets
      4 from huggingface_hub import list_datasets
      5 print([dataset.id for dataset in list_datasets()])

6 frames
[/usr/lib/python3.10/functools.py](https://localhost:8080/#) in update_wrapper(wrapper, wrapped, assigned, updated)
     59     # Issue #17482: set __wrapped__ last so we don't inadvertently copy it
     60     # from the wrapped function when updating __dict__
---> 61     wrapper.__wrapped__ = wrapped
     62     # Return the wrapper so this can be used as a decorator via partial()
     63     return wrapper

AttributeError: readonly attribute
```

### Expected behavior

Run success on Google Colab (free)

### Environment info

Windows 11 x64, Google Colab free


You are most likely using an outdated version of `datasets` in the notebook, which can be verified with the `!datasets-cli env` command. You can run `!pip install -U datasets` to update the installation.

okay, it works! thank you so much! 😄
`IterableDataset` lost but not keep columns when map function adding columns with names in `remove_columns`

### Describe the bug

IterableDataset lost but not keep columns when map function adding columns with names in remove_columns,
Dataset not.

May be related to the code below:
https://github.com/huggingface/datasets/blob/06c3ffb8d068b6307b247164b10f7c7311cefed4/src/datasets/iterable_dataset.py#L750-L756

### Steps to reproduce the bug

```python
dataset: IterableDataset = load_dataset("Anthropic/hh-rlhf", streaming=True, split="train")
column_names = list(next(iter(dataset)).keys())  #  ['chosen', 'rejected']
# map_fn will return dict {"chosen": xxx, "rejected": xxx, "prompt": xxx, "history": xxxx}
dataset = dataset.map(map_fn, batched=True, remove_columns=column_names)
next(iter(dataset))
# output
# {'prompt': 'xxx, 'history': xxx}   
```

```python
# when load_dataset with streaming=False, the column_names are kept: 
dataset: Dataset = load_dataset("Anthropic/hh-rlhf", streaming=False, split="train")
column_names = list(next(iter(dataset)).keys())  #  ['chosen', 'rejected']
# map_fn will return dict {"chosen": xxx, "rejected": xxx, "prompt": xxx, "history": xxxx}
dataset = dataset.map(map_fn, batched=True, remove_columns=column_names)
next(iter(dataset))
# output
# {'prompt': 'xxx, 'history': xxx, "chosen": xxx, "rejected": xxx}   
```


### Expected behavior

IterableDataset keep columns when map function adding columns with names in remove_columns

### Environment info

datasets==2.14.6
Allow to choose the `writer_batch_size` when using `save_to_disk`

### Feature request

Add an argument in `save_to_disk` regarding batch size, which would be passed to `shard` and other methods.

### Motivation

The `Dataset.save_to_disk` method currently calls `shard` without passing a `writer_batch_size` argument, thus implicitly using the default value (1000). This can result in RAM saturation when using a lot of processes on long text sequences or other modalities, or for specific IO configs.

### Your contribution

I would be glad to submit a PR, as long as it does not imply extensive tests refactoring.

Labels: name: enhancement

We have a config variable that controls the batch size in `save_to_disk`:
```python
import datasets
datasets.config.DEFAULT_MAX_BATCH_SIZE = <smaller_batch_size>
...
ds.save_to_disk(...)
```

Thank you for your answer!

From what I am reading in `https://github.com/huggingface/datasets/blob/2.14.5/src/datasets/arrow_dataset.py`, every function involved (`select`, `shard`, ...) has a default hardcoded batch size of 1000, as such:
```python
def select(
        self,
        indices: Iterable,
        keep_in_memory: bool = False,
        indices_cache_file_name: Optional[str] = None,
        writer_batch_size: Optional[int] = 1000,
        new_fingerprint: Optional[str] = None,
    ) -> "Dataset":
...
```
Then, `ArrowWriter` is instantiated with the specified `writer_batch_size`. In `ArrowWriter`, `writer_batch_size` is set to `datasets.config.DEFAULT_MAX_BATCH_SIZE` if it is `None`(https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_writer.py#L345C14-L345C31). However, in our case, it is already set to 1000 by "parent" methods, so it won't happen.

Nevertheless, due to this: 
```python
def _save_to_disk_single(job_id: int, shard: "Dataset", fpath: str, storage_options: Optional[dict]):
        batch_size = config.DEFAULT_MAX_BATCH_SIZE
...
```
it seems to work. I will use it as such, but it should maybe be added to documentation? And maybe improved in next versions?
How to load existing downloaded dataset ?

Hi @mariosasko @lhoestq  @katielink 

Thanks for your contribution and hard work.

### Feature request

First, I download a dataset as normal by:
```
from datasets import load_dataset

dataset = load_dataset('username/data_name', cache_dir='data')
```

The dataset format in `data` directory will be:
```
-data
  |-data_name
    |-test-00000-of-00001-bf4c733542e35fcb.parquet
    |-train-00000-of-00001-2a1df75c6bce91ab.parquet
```


Then I use SCP to clone this dataset into another machine, and then try:
```
from datasets import load_dataset

dataset = load_dataset('data/data_name')  # load from local path
```

This leads to re-generating training and validation split for each time, and the disk quota will be duplicated occupation.

How can I just load the dataset without generating and saving these splits again?

### Motivation

I do not want to download the same dataset in two machines, scp is much faster and better than HuggingFace API. I hope we can directly load the downloaded datasets (.parquest)

### Your contribution

Please refer to the feature

Labels: name: enhancement

Feel free to use `dataset.save_to_disk(...)`, then scp the directory containing the saved dataset and reload it on your other machine using `dataset = load_from_disk(...)`
custom load dataset to hub

### System Info

kaggle notebook

i transformed dataset:
```
dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
```
to 
formatted_dataset: 
```
Dataset({
    features: ['message_tree_id', 'message_tree_text'],
    num_rows: 33143
})
```
but would like to know how to upload to hub

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

shared above

### Expected behavior

load dataset to hub


This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

> This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

Thanks @sgugger , I guess I will wait for them to address the issue . Looking forward to hearing from them 

You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

> You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot ! 

> > You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.
> 
> how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot !

I solved it by upgrading `Datasets` version to 2.15.0