dalssy/data_del huggingface dataset:
GitHub issue: .Jsonl metadata not detected
Author: nighting0le01 (NONE)
State: open | Created: 2024-04-04 06:31:53

### Describe the bug

Hi I have the following directory structure:
|--dataset
| |-- images
| |-- metadata1000.csv
| |-- metadata1000.jsonl
| |-- padded_images

Example of metadata1000.jsonl file
{"caption": "a drawing depicts a full shot of a black t-shirt with a triangular pattern on the front there is a white label on the left side of the triangle", "image": "images/212734.png", "gaussian_padded_image": "padded_images/p_212734.png"}
{"caption": "an eye-level full shot of a large elephant and a baby elephant standing in a watering hole on the left side is a small elephant with its head turned to the right of dry land, trees, and bushes", "image": "images/212735.png", "gaussian_padded_image": "padded_images/p_212735.png"}
.
.
.

I'm trying to use dataset = load_dataset("imagefolder", data_dir='/dataset/', split='train') to load the the dataset, however it is not able to load according to the fields in the metadata1000.jsonl .
please assist to load the data properly

also getting 

```
  File "/workspace/train_trans_vae.py", line 1089, in <module>
    print(get_metadata_patterns('/dataset/'))
  File "/opt/conda/lib/python3.10/site-packages/datasets/data_files.py", line 499, in get_metadata_patterns
    raise FileNotFoundError(f"The directory at {base_path} doesn't contain any metadata file") from None
FileNotFoundError: The directory at /dataset/ doesn't contain any metadata file
```

when trying 

```
    from datasets.data_files import get_metadata_patterns
    print(get_metadata_patterns('/dataset/'))
```




### Steps to reproduce the bug

dataset Version: 2.18.0
make a similar jsonl and similar directory format

### Expected behavior

creates a dataset object with the column names, caption,image,gaussian_padded_image

### Environment info

dataset Version: 2.18.0
GitHub issue: IndexError: Invalid key: 0 is out of bounds for size 0
Author: kk2491 (NONE)
State: open | Created: 2024-04-03 17:06:30

### Describe the bug

I am trying to fine-tune llama2-7b model in GCP. The notebook I am using for this can be found [here](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_llama2_peft_finetuning.ipynb).  

When I use the dataset given in the example, the training gets successfully completed (example dataset can be found [here](https://huggingface.co/datasets/timdettmers/openassistant-guanaco)).  
However when I use my own dataset which is in the same format as the example dataset, I get the below error (my dataset can be found [here](https://huggingface.co/datasets/kk2491/finetune_dataset_002)).   

![image](https://github.com/huggingface/datasets/assets/38481564/47fa2de3-95e0-478b-a35f-58cbaf90427a)  

I see the files are being read correctly from the logs:  

![image](https://github.com/huggingface/datasets/assets/38481564/b0b6316c-2cc7-476c-9674-ca2222c8f4e3)



### Steps to reproduce the bug

1. Clone the [vertex-ai-samples](https://github.com/GoogleCloudPlatform/vertex-ai-samples) repository. 
2. Run the [llama2-7b peft fine-tuning](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_llama2_peft_finetuning.ipynb).
3. Change the dataset `kk2491/finetune_dataset_002`   

### Expected behavior

The training should complete successfully, and model gets deployed to an endpoint. 



### Environment info

Python version : Python 3.10.12
Dataset : https://huggingface.co/datasets/kk2491/finetune_dataset_002
GitHub issue:  Generating split is very slow when Image format is PNG
Author: Tramac (NONE)
State: open | Created: 2024-04-03 07:47:31

### Describe the bug

When I create a dataset, it gets stuck while generating cached data. 
The image format is PNG, and it will not get stuck when the image format is jpeg.

![image](https://github.com/huggingface/datasets/assets/22740819/3b888fd8-e6d6-488f-b828-95a8f206a152)

After debugging, I know that it is because of the `pa.array` operation in [arrow_writer](https://github.com/huggingface/datasets/blob/2.13.0/src/datasets/arrow_writer.py#L553), but i don't why.

### Steps to reproduce the bug

```
from datasets import Dataset

def generator(lines):
    for line in lines:
        img = Image.open(open(line["url"], "rb"))
        # print(img.format)  # "PNG"
        yield {
            "image": img,
        }

lines = open(dataset_path, "r")
dataset = Dataset.from_generator(
    generator,
    gen_kwargs={"lines": lines}
)
```

### Expected behavior

Generating split done.

### Environment info

datasets    2.13.0
GitHub issue: Dataset on Hub re-downloads every time?
Author: manestay (NONE)
State: closed | Created: 2024-04-02 17:23:22

### Describe the bug

Hi, I have a dataset on the hub [here](https://huggingface.co/datasets/manestay/borderlines). It has 1k+ downloads, which I sure is mostly just me and my colleagues working with it. It should have far fewer, since I'm using the same machine with a properly set up HF_HOME variable. However, whenever I run the below function `load_borderlines_hf`, it downloads the entire dataset from the hub and then does the other logic:
https://github.com/manestay/borderlines/blob/4e161f444661e2ebfe643f3fe149d9258d63a57d/run_gpt/lib.py#L80

Let me know what I'm doing wrong here, or if it's a bug with the `datasets` library itself. On the hub I have my data stored in CSVs, but several columns are lists, so that's why I have the code to map splitting on `;`. I looked into dataset loading scripts, but it seemed difficult to set up. I have verified that other `datasets` and `models` on my system are using the cache properly (e.g. I have a 13B parameter model and large datasets, but those are cached and don't redownload).

__EDIT: __ as pointed out in the discussion below, it may be the `map()` calls that aren't being cached properly. Supposing the `load_dataset()` retrieve from the cache, then it should be the case that the `map()` calls also retrieve from the cached output. But the `map()` commands re-execute sometimes.

### Steps to reproduce the bug

1. Copy and paste the function from [here](https://github.com/manestay/borderlines/blob/4e161f444661e2ebfe643f3fe149d9258d63a57d/run_gpt/lib.py#L80) (lines 80-100)
2. Run it in Python `load_borderlines_hf(None)`
3. It completes successfully, downloading from HF hub, then doing the mapping logic etc.
4. If you run it again after some time, it will re-download, ignoring the cache

### Expected behavior

Re-running the code, which calls `datasets.load_dataset('manestay/borderlines', 'territories')`, should use the cached version

### Environment info


- `datasets` version: 2.16.1
- Platform: Linux-5.14.21-150500.55.7-default-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0
GitHub issue: `remove_columns`/`rename_columns` doc fixes
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2024-04-02 15:41:28

Use more consistent wording in `remove_columns` to explain why it's faster than `map` and update `remove_columns`/`rename_columns` docstrings to fix in-place calls.


Reported in https://github.com/huggingface/datasets/issues/6700
GitHub issue: (Willing to PR) Datasets with custom python objects
Author: fzyzcjy (CONTRIBUTOR)
State: open | Created: 2024-04-01 13:18:47
Labels: name: enhancement

### Feature request

Hi thanks for the library! I would like to have a huggingface Dataset, and one of its column is custom (non-serializable) Python objects. For example, a minimal code:

```
class MyClass:
    pass

dataset = datasets.Dataset.from_list([
    dict(a=MyClass(), b='hello'),
])
```

It gives error:

```
ArrowInvalid: Could not convert <__main__.MyClass object at 0x7a852830d050> with type MyClass: did not recognize Python value type when inferring an Arrow data type
```

I guess it is because Dataset forces to convert everything into arrow format. However, is there any ways to make the scenario work? Thanks!

### Motivation

(see above)

### Your contribution

Yes, I am happy to PR!

Cross-posted: https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050?u=fzyzcjy

EDIT: possibly related https://github.com/huggingface/datasets/issues/5766
GitHub issue: Compatibility issue between s3fs, fsspec, and datasets
Author: njbrake (NONE)
State: closed | Created: 2024-03-29 19:57:24

### Describe the bug

Here is the full error stack when installing:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires fsspec[http]<=2024.2.0,>=2023.1.0, but you have fsspec 2024.3.1 which is incompatible.
Successfully installed aiobotocore-2.12.1 aioitertools-0.11.0 botocore-1.34.51 fsspec-2024.3.1 jmespath-1.0.1 s3fs-2024.3.1 urllib3-2.0.7 wrapt-1.16.0
```
When I install with pip, pip allows this error to exist while still installing s3fs, but this error breaks poetry, since poetry will refuse to install s3fs because of the dependency conflict.

Maybe I'm missing something so maybe it's not a bug but some mistake on my end? Any input would be helpful. Thanks!

### Steps to reproduce the bug

1. conda create -n tmp python=3.10 -y
2. conda activate tmp
3. pip install datasets
4. pip install s3fs

### Expected behavior

I would expect there to be no error.

### Environment info

MacOS (ARM), Python3.10, conda 23.11.0.
GitHub issue: load_dataset can't work with symbolic links
Author: VladimirVincan (NONE)
State: open | Created: 2024-03-29 17:49:28
Labels: name: enhancement

### Feature request

Enable the `load_dataset` function to load local datasets with symbolic links. 

E.g, this dataset can be loaded:
├── example_dataset/
│   ├── data/
│   │   ├── train/
│   │   │   ├── file0
│   │   │   ├── file1
│   │   ├── dev/
│   │   │   ├── file2
│   │   │   ├── file3
│   ├── metadata.csv

while this dataset can't:
├── example_dataset_symlink/
│   ├── data/
│   │   ├── train/
│   │   │   ├── sym0 -> file0
│   │   │   ├── sym1 -> file1
│   │   ├── dev/
│   │   │   ├── sym2 -> file2
│   │   │   ├── sym3 -> file3
│   ├── metadata.csv

I have created an example dataset in order to reproduce the problem:

1. Unzip `example_dataset.zip`.
2. Run `no_symlink.sh`. Training should start without issues. 
3. Run `symlink.sh`. You will see that all four examples will be in train split, instead of having two examples in train and two examples in dev. The script won't load the correct audio files.

[example_dataset.zip](https://github.com/huggingface/datasets/files/14807053/example_dataset.zip)

### Motivation

I have a very large dataset locally. Instead of initiating training on the entire dataset, I need to start training on smaller subsets of the data. Due to the purpose of the experiments I am running, I will need to create many smaller datasets with overlapping data. Instead of copying the all the files for each subset, I would prefer copying symbolic links of the data. This way, the memory usage would not significantly increase beyond the initial dataset size.

Advantages of this approach:

- It would leave a smaller memory footprint on the hard drive
- Creating smaller datasets would be much faster

### Your contribution

I would gladly contribute, if this is something useful to the community. It seems like a simple change of code, something like `file_path = os.path.realpath(file_path)` should be added before loading the files. If anyone has insights on how to incorporate this functionality, I would greatly appreciate your knowledge and input.
GitHub issue: Fix issue with case sensitivity when loading dataset from local cache
Author: Sumsky21 (NONE)
State: open | Created: 2024-03-28 14:52:35

When a dataset with upper-cases in its name is first loaded using `load_dataset()`, the local cache directory is created with all lowercase letters. 

However, upon subsequent loads, the current version attempts to locate the cache directory using the dataset's original name, which includes uppercase letters. This discrepancy can lead to confusion and, particularly in offline mode, results in errors. 

### Reproduce
```bash
~$ python            
Python 3.9.19 (main, Mar 21 2024, 17:11:28) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("locuslab/TOFU", "full")
>>> quit()

~$ export HF_DATASETS_OFFLINE=1 
~$ python                      
Python 3.9.19 (main, Mar 21 2024, 17:11:28) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("locuslab/TOFU", "full")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 1871, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'locuslab/TOFU': Offline mode is enabled.
>>> 
```

I fix this issue by lowering the dataset name (`.lower()`) when generating cache_dir.
GitHub issue: Remove deprecated code
Author: Wauplin (CONTRIBUTOR)
State: closed | Created: 2024-03-28 09:57:57

What does this PR do?
1. remove `list_files_info` in favor of `list_repo_tree`. As of `0.23`, `list_files_info` will be removed for good. `datasets` had a utility to support both pre-0.20 and post-0.20 versions. Since `hfh` version is already pinned to `>=0.21.2`, I removed the legacy part.
2. `preupload_lfs_files` had also a different behavior between `<0.20` and `>=0.20`. I remove it since huggingface_hub is now pinned to `>=0.21.2`
3. `hf_hub_url` is overwritten to default to the dataset repo_type. I do think it is misleading to keep the same method naming for it. I renamed it to `get_dataset_url` for clarity. Let me know if you prefer to see this change reverted.
GitHub issue: Load codeparrot/apps raising UnicodeDecodeError in datasets-2.18.0
Author: yucc-leon (NONE)
State: open | Created: 2024-03-28 03:44:26

### Describe the bug

This happens with datasets-2.18.0; I downgraded the version to 2.14.6 fixing this temporarily.
```
Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 1879, in dataset_module_factory
    raise e1 from None
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 1831, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```

### Steps to reproduce the bug

1. Using Python3.10/3.11
2. Install datasets-2.18.0
3. test with 
```
from datasets import load_dataset
dataset = load_dataset("codeparrot/apps")
```


### Expected behavior

Normally it should manage to download and load the dataset without such error.

### Environment info

Ubuntu, Python3.10/3.11
GitHub issue: Persistent multi-process Pool
Author: fostiropoulos (NONE)
State: open | Created: 2024-03-26 17:35:25
Labels: name: enhancement

### Feature request

Running .map and filter functions with `num_procs` consecutively instantiates several multiprocessing pools iteratively. 

As instantiating a Pool is very resource intensive it can be a bottleneck to performing iteratively filtering. 

My ideas:
1. There should be an option to declare `persistent_workers` similar to pytorch DataLoader. Downside would be that would be complex to determine the correct resource allocation and deallocation of the pool. i.e. the dataset can outlive the utility of the pool. 
2. Provide a pool as an argument. Downside would be the expertise required by the user. Upside, is that there is better resource management.

### Motivation

Is really slow to iteratively perform map and filter operations on a dataset.

### Your contribution

If approved I could integrate it. I would need to know what method would be most suitable to implement from the two options above.
GitHub issue: Passing `sample_by` to `load_dataset` when loading text data does not work
Author: ntoxeg (NONE)
State: closed | Created: 2024-03-26 14:55:33

### Describe the bug

I have a dataset that consists of a bunch of text files, each representing an example. There is an undocumented `sample_by` argument for the `TextConfig` class that is used by `Text` to decide whether to split files into lines, paragraphs or take them whole. Passing `sample_by=“document”` to `load_dataset` results in files getting split into lines regardless. I have edited `src/datasets/packaged_modules/text/text.py` for myself to switch the default and it works fine.

As a side note, the `if-else` for `sample_by` will silently load an empty dataset if someone makes a typo in the argument, which is not ideal.

### Steps to reproduce the bug

1. Prepare data as a bunch of files in a directory.
2. Load that data via `load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)`.
3. Inspect the resultant dataset — every item should have the form of `{“text”: <a line from a file>}`.

### Expected behavior

`load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)` should result in a dataset with items of the form `{“text”: <one document>}`.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-1046-nvidia-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
GitHub issue: Test disabling transformers containers in docs CI
Author: Wauplin (CONTRIBUTOR)
State: open | Created: 2024-03-25 17:16:11

Related to https://github.com/huggingface/doc-builder/pull/487 and [internal slack thread](https://huggingface.slack.com/archives/C04F8N7FQNL/p1711384899462349?thread_ts=1711041424.720769&cid=C04F8N7FQNL). There is now a `custom_container` option when building docs in CI. When set to `""` (instead of `"huggingface/transformers-doc-builder"` by default), we don't run the CI inside a container, therefore saving ~2min of download time. The plan is to test disabling the transformers container on a few "big" repo and if everything works correctly, we will stop making it the default container. More details on https://github.com/huggingface/doc-builder/pull/487.

cc @mishig25
GitHub issue: Support SQLite files?
Author: severo (COLLABORATOR)
State: closed | Created: 2024-03-25 11:48:05
Labels: name: enhancement

### Feature request

Support loading a dataset from a SQLite file

https://huggingface.co/datasets/severo/test_iris_sqlite/tree/main

### Motivation

SQLite is a popular file format.

### Your contribution

See discussion on slack: https://huggingface.slack.com/archives/C04L6P8KNQ5/p1702481859117909 (internal)

In particular: a SQLite file can contain multiple tables, which might be matched to multiple configs. Maybe the detail of splits and configs should be defined in the README YAML, or use the same format as for ZIP files: `Iris.sqlite::Iris`. 

See dataset here: https://huggingface.co/datasets/severo/test_iris_sqlite

Note: should we also support DuckDB files?
GitHub issue: Small typo on the documentation
Author: fostiropoulos (NONE)
State: closed | Created: 2024-03-24 21:47:52
Labels: name: good first issue

### Describe the bug

There is a small typo on https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938

It should be `caching is enabled`. 

### Steps to reproduce the bug

Please visit 

https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938


### Expected behavior

`caching is enabled`

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-5.15.0-101-generic-x86_64-with-glibc2.35
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.10.0
GitHub issue: Fix cache path to snakecase for `CachedDatasetModuleFactory` and `Cache`
Author: izhx (CONTRIBUTOR)
State: closed | Created: 2024-03-24 06:59:15

Fix https://github.com/huggingface/datasets/issues/6750#issuecomment-2016678729

I didn't find a guideline on how to run the tests, so i just run the following steps to make sure that this bug is fixed.
1. `python test.py`,
2. then `HF_DATASETS_OFFLINE=1 python test.py`

The `test.py` is
```
  import datasets

  datasets.utils.logging.set_verbosity_info()

  ds = datasets.load_dataset('izhx/STS17-debug')
  print(ds)

  ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')
  print(ds)
```
GitHub issue: Precision being changed from float16 to float32 unexpectedly
Author: gcervantes8 (NONE)
State: open | Created: 2024-03-23 20:53:56

### Describe the bug

I'm loading a HuggingFace Dataset for images. 

I'm running a preprocessing (map operation) step that runs a few operations, one of them being conversion to float16.  The Dataset features also say that the 'img' is of type float16.  Whenever I take an image from that HuggingFace Dataset instance, the type turns out to be float32.  



### Steps to reproduce the bug

```python
import torchvision.transforms.v2 as transforms
from datasets import load_dataset

dataset = load_dataset('cifar10', split='test')
dataset = dataset.with_format("torch")

data_transform = transforms.Compose([transforms.Resize((32, 32)), 
                                     transforms.ToDtype(torch.float16, scale=True),
                                     transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
                                     ])
def _preprocess(examples):
    # Permutes from (BS x H x W x C) to (BS x C x H x W)
    images = torch.permute(examples['img'], (0, 3, 2, 1))
    examples['img'] = data_transform(images)
    return examples

dataset = dataset.map(_preprocess, batched=True, batch_size=8)
```

Now at this point the dataset.features are showing float16 which is great because that's what I want. 

```python
print(data_loader.features['img'])

Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float16', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None)
```

But when I try to sample an image from this dataloader; I'm getting a float32 image, when I'm expecting float16:
```python
print(next(iter(data_loader))['img'].dtype)

torch.float32
```

### Expected behavior

I'm expecting the images loaded after the transformation to stay in float16.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.9
- `huggingface_hub` version: 0.21.4
- PyArrow version: 14.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0
GitHub issue: Strange slicing behavior
Author: Luciennnnnnn (NONE)
State: open | Created: 2024-03-22 01:49:13

### Describe the bug

I have loaded a dataset, and then slice first 300 samples using `:` ops, however, the resulting dataset is not expected, as the output below:

```bash
len(dataset)=1050324
len(dataset[:300])=2
len(dataset[0:300])=2
len(dataset.select(range(300)))=300
```

### Steps to reproduce the bug

load a dataset then:
```bash
    dataset = load_from_disk(args.train_data_dir)

    print(f"{len(dataset)=}", flush=True)
    print(f"{len(dataset[:300])=}", flush=True)
    print(f"{len(dataset[0:300])=}", flush=True)
    print(f"{len(dataset.select(range(300)))=}", flush=True)
```

### Expected behavior

```bash
len(dataset)=1050324
len(dataset[:300])=300
len(dataset[0:300])=300
len(dataset.select(range(300)))=300
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- `huggingface_hub` version: 0.20.2
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0
GitHub issue: ExpectedMoreSplits error when loading C4 dataset
Author: billwang485 (NONE)
State: closed | Created: 2024-03-21 02:53:04

### Describe the bug

I encounter bug when running the example command line 

```python
    python main.py \
    --model decapoda-research/llama-7b-hf \
    --prune_method wanda \
    --sparsity_ratio 0.5 \
    --sparsity_type unstructured \
    --save out/llama_7b/unstructured/wanda/ 
```

The bug occurred at these lines of code (when loading c4 dataset) 

```python
traindata = load_dataset('allenai/c4', 'allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train')
valdata = load_dataset('allenai/c4', 'allenai--c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation')
```

The error message states:
```    
raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))                                                                           
datasets.utils.info_utils.ExpectedMoreSplits: {'validation'}
```



### Steps to reproduce the bug

1. I encounter bug when running the example command line 


### Expected behavior

The error message states:
```    
raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))                                                                           
datasets.utils.info_utils.ExpectedMoreSplits: {'validation'}
```



### Environment info

I'm using cuda 12.4, so I use ```pip install pytorch``` instead of conda provided in install.md

Also, I've tried another environment using the same commands in install.md, but the same bug occured
GitHub issue: Scraping the whole of github including private repos is bad; kindly stop
Author: ghost (NONE)
State: closed | Created: 2024-03-20 20:54:06
Labels: name: enhancement

### Feature request

https://github.com/bigcode-project/opt-out-v2 - opt out is not consent. kindly quit this ridiculous nonsense.

### Motivation

[EDITED: insults not tolerated]

### Your contribution

[EDITED: insults not tolerated]
GitHub issue: Option to disable file locking
Author: VRehnberg (NONE)
State: open | Created: 2024-03-20 15:59:45
Labels: name: enhancement

### Feature request

Commands such as `load_dataset` creates file locks with `filelock.FileLock`. It would be good if there was a way to disable this.

### Motivation

File locking doesn't work on all file-systems (in my case NFS mounted Weka). If the `cache_dir` only had small files then it would be possible to point to local disk and the problem would be solved. However, as cache_dir is both where the small info files are written and the processed datasets are put this isn't a feasible solution.

Considering https://github.com/huggingface/datasets/issues/6395 I still do think this is something that belongs in HuggingFace. The possibility to control packages separately is valuable. It might be that a user has their dataset on a file-system that doesn't support file-locking while they are using file locking on local disk to control some other type of access.

### Your contribution

My suggested solution:
```
diff --git a/src/datasets/utils/_filelock.py b/src/datasets/utils/_filelock.py
index 19620e6e..58f41a02 100644
--- a/src/datasets/utils/_filelock.py
+++ b/src/datasets/utils/_filelock.py
@@ -18,11 +18,15 @@
 import os
 
 from filelock import FileLock as FileLock_
-from filelock import UnixFileLock
+from filelock import SoftFileLock, UnixFileLock
 from filelock import __version__ as _filelock_version
 from packaging import version
 
 
+if os.getenv('HF_USE_SOFTFILELOCK', 'false').lower() in ('true', '1'):
+    FileLock_ = SoftFileLock
+
+
 class FileLock(FileLock_):
     """
     A `filelock.FileLock` initializer that handles long paths.
```
GitHub issue: Fix offline mode with single config
Author: lhoestq (MEMBER)
State: closed | Created: 2024-03-19 10:48:32

Reported in https://github.com/huggingface/datasets/issues/4760

The cache was not able to reload a dataset with a single config form the cache if the config name is not specificed

For example

```python
from datasets import load_dataset, config

config.HF_DATASETS_OFFLINE = True

load_dataset("openai_humaneval")
```

This was due to a regression in https://github.com/huggingface/datasets/pull/6632
GitHub issue: Support for loading geotiff files as a part of the ImageFolder
Author: sunny1401 (NONE)
State: closed | Created: 2024-03-18 20:00:39
Labels: name: enhancement

### Feature request

Request for adding rasterio support to load geotiff as a part of ImageFolder, instead of using PIL

### Motivation

As of now, there are many datasets in HuggingFace Hub which are predominantly focussed towards RemoteSensing or are from RemoteSensing. The current ImageFolder (if I have understood correctly) uses PIL. This is not really optimized because mostly these datasets have images with many channels and additional metadata. Using PIL makes one loose it unless we provide a custom script. Hence, maybe an API could be added to have this in common?

### Your contribution

If the issue is accepted - i can contribute the code, because I would like to have it automated and generalised.
GitHub issue: Dict feature is non-nullable while nested dict feature is
Author: polinaeterna (CONTRIBUTOR)
State: closed | Created: 2024-03-18 14:31:47
Labels: name: bug

When i try to create a `Dataset` object with None values inside a dict column, like this: 
```python
from datasets import Dataset, Features, Value

Dataset.from_dict(
    {
        "dict": [{"a": 0, "b": 0}, None],
    }, features=Features(
        {"dict": {"a": Value("int16"), "b": Value("int16")}}
    )
)
```
i get `ValueError: Got None but expected a dictionary instead`.


At the same time, having None in _nested_ dict feature works, for example, this doesn't throw any errors:

```python
from datasets import Dataset, Features, Value, Sequence

dataset = Dataset.from_dict(
    {
        "list_dict": [[{"a": 0, "b": 0}], None],
        "sequence_dict": [[{"a": 0, "b": 0}], None],
    }, features=Features({
        "list_dict": [{"a": Value("int16"), "b": Value("int16")}],
        "sequence_dict": Sequence({"a": Value("int16"), "b": Value("int16")}),
    })
)
```
Other types of features also seem to be nullable (but I haven't checked all of them).

Version of `datasets` is the latest atm (2.18.0)


Is this an expected behavior or a bug?
GitHub issue: Invalid pattern: '**' can only be an entire path component
Author: JPonsa (NONE)
State: closed | Created: 2024-03-16 19:28:46

### Describe the bug

ValueError: Invalid pattern: '**' can only be an entire path component
when loading any dataset


### Steps to reproduce the bug

import datasets
ds = datasets.load_dataset("TokenBender/code_instructions_122k_alpaca_style")

### Expected behavior

loading the dataset successfully 

### Environment info

- `datasets` version: 2.18.0
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.12.2
GitHub issue: Mosaic Streaming (MDS) Support
Author: siddk (NONE)
State: open | Created: 2024-03-16 18:42:04
Labels: name: enhancement

### Feature request

I'm a huge fan of the current HF Datasets `webdataset` integration (especially the built-in streaming support). However, I'd love to upload some robotics and multimodal datasets I've processed for use with [Mosaic Streaming](https://docs.mosaicml.com/projects/streaming/en/stable/), specifically their [MDS Format](https://docs.mosaicml.com/projects/streaming/en/stable/fundamentals/dataset_format.html#mds). 

Because the shard files have similar semantics to WebDataset, I'm hoping that adding such support won't be too much trouble?

### Motivation

One of the downsides with WebDataset is a lack of out-of-the-box determinism (especially for large-scale training and reproducibility), easy job resumption, and the ability to quickly debug / visualize individual examples. 

Mosaic Streaming provides a [great interface for this out of the box](https://docs.mosaicml.com/projects/streaming/en/stable/#key-features), so I'd love to see it supported in HF Datasets.

### Your contribution

Happy to help test things / provide example data. Can potentially submit a PR if maintainers could point me to the necessary WebDataset logic / steps for adding a new streaming format!
GitHub issue: Tokenization slows towards end of dataset
Author: ethansmith2000 (NONE)
State: open | Created: 2024-03-15 03:27:36

### Describe the bug

Mapped tokenization slows down substantially towards end of dataset.

train set started off very slow, caught up to 20k then tapered off til the end.

what's particularly strange is that the tokenization crashed a few times before due to errors with invalid tokens somewhere or corrupted downloads, and the speed ups/downs consistently happened the same times

```bash
Running tokenizer on dataset (num_proc=48):   0%|          | 847000/881416735 [12:18<252:45:45, 967.72 examples/s]
Running tokenizer on dataset (num_proc=48):   0%|          | 848000/881416735 [12:19<224:16:10, 1090.66 examples/s]

Running tokenizer on dataset (num_proc=48):  10%|▉         | 84964000/881416735 [3:48:00<11:21:34, 19476.01 examples/s]
Running tokenizer on dataset (num_proc=48):  10%|▉         | 84967000/881416735 [3:48:00<12:04:01, 18333.79 examples/s]

Running tokenizer on dataset (num_proc=48):  61%|██████    | 538631977/881416735 [13:46:40<27:50:04, 3420.84 examples/s]
Running tokenizer on dataset (num_proc=48):  61%|██████    | 538632977/881416735 [13:46:40<23:48:20, 3999.77 examples/s]

Running tokenizer on dataset (num_proc=48): 100%|█████████▉| 881365886/881416735 [38:30:19<04:34, 185.10 examples/s]
Running tokenizer on dataset (num_proc=48): 100%|█████████▉| 881366886/881416735 [38:30:25<04:36, 180.57 examples/s]
```

and validation set as well

```bash
Running tokenizer on dataset (num_proc=48):  90%|████████▉ | 41544000/46390354 [28:44<02:37, 30798.76 examples/s]
Running tokenizer on dataset (num_proc=48):  90%|████████▉ | 41550000/46390354 [28:44<02:08, 37698.08 examples/s]

Running tokenizer on dataset (num_proc=48):  96%|█████████▋| 44747422/46390354 [2:15:48<12:22:44, 36.87 examples/s]
Running tokenizer on dataset (num_proc=48):  96%|█████████▋| 44747422/46390354 [2:16:00<12:22:44, 36.87 examples/s]

```



### Steps to reproduce the bug

    using the following kwargs
    
```python
with accelerator.main_process_first():
        lm_datasets = tokenized_datasets.map(
            group_texts,
            batched=True,
            num_proc=48
            load_from_cache_file=True,
            desc=f"Grouping texts in chunks of {block_size}",
        )
```

running through slurm script 
```bash
#SBATCH --partition=gpu-nvidia-a100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus-per-task=8
#SBATCH --cpus-per-task=96
```

using this dataset https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T





### Expected behavior

Constant speed throughout

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.10
- Python version: 3.8.18
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0
GitHub issue: EmptyDatasetError when loading dataset downloaded with HuggingFace cli
Author: StwayneXG (NONE)
State: open | Created: 2024-03-14 16:41:27

### Describe the bug

I am using a cluster that does not have access to the internet when given a job. I tried downloading the dataset using the huggingface-cli command and then loading it with load_dataset but I get an error:

```raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None```

The dataset I'm using is "lmsys/chatbot_arena_conversations". The folder structure is

- README.md
- data
  - train-00000-of-00001-cced8514c7ed782a.parquet



### Steps to reproduce the bug

1. Download dataset using HuggingFace CLI: ```huggingface-cli download lmsys/chatbot_arena_conversations --local-dir ./lmsys/chatbot_arena_conversations```
2. In Python
```
from datasets import load_dataset
load_dataset("lmsys/chatbot_arena_conversations")
```

### Expected behavior

Should return a Dataset Dict in the form of
```
DatasetDict({
    train: Dataset({
        features: [...],
        num_rows: 33,000
    })
})
```
### Environment info

Python 3.11.5
Datasets 2.18.0
Transformers 4.38.2
Pytorch 2.2.0
Pyarrow 15.0.1
Rocky Linux release 8.9 (Green Obsidian)
GitHub issue: Unexpected behavior when using load_dataset with streaming=True in a for loop
Author: uApiv (NONE)
State: closed | Created: 2024-03-12 23:26:43

### Describe the bug

### My Code
```
from datasets import load_dataset
res=[]
for i in [0,1]:
    di=load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
        ).map(lambda x: {"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

### Unexpected Behavior
Data in `res[0]` has `source=1`. However the expected value is 0.

### FYI
I further switch `streaming` to `False`. And the output value is as expected (0). So there may exist bugs in setting `streaming=True` in a for loop.

### Environment
Python 3.8.0
datasets==2.18.0
transformers==4.28.1

### Steps to reproduce the bug

1. Create a Json file with any content.
2. Run the provided code.
3. Switch `streaming` to `False` and run again to see the expected behavior.

### Expected behavior

The expected behavior is the data are mapped with its corresponding value in the for loop.

### Environment info

Python 3.8.0

datasets==2.18.0
transformers==4.28.1

Ubuntu 20.04
GitHub issue: Deprecate Pandas builder
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2024-03-12 15:12:13

The Pandas packaged builder is undocumented and relies on `pickle` to read the data, making it **unsafe**. Moreover, I haven't seen a single instance of this builder being used (not even using the GH/Hub search), so we should deprecate it.
GitHub issue: Request for a comparison of huggingface datasets compared with other data format especially webdataset
Author: Luciennnnnnn (NONE)
State: open | Created: 2024-03-08 08:23:01
Labels: name: enhancement

### Feature request

Request for a comparison of huggingface datasets compared with other data format especially webdataset

### Motivation

I see huggingface datasets uses Apache Arrow as its backend, it seems to be great, but I'm curious about how it is good compared with other dataset format, like webdataset, what's the pros/cons of them.

### Your contribution

More information
GitHub issue: Dataset with loading script does not work in renamed repos
Author: BramVanroy (CONTRIBUTOR)
State: open | Created: 2024-03-07 17:38:38

### Describe the bug

My data repository was first called `BramVanroy/hplt-mono-v1-2` but I then renamed to use underscores instead of dashes. However, it seems that `datasets` retrieves the old repo name when it checks whether the repo contains data loading scripts in this line.

https://github.com/huggingface/datasets/blob/6fb6c834f008996c994b0a86c3808d0a33d44525/src/datasets/load.py#L1845

When I print `filename` it returns `hplt-mono-v1-2.py` but the files in the repo are of course `['.gitattributes', 'README.md', 'hplt_mono_v1_2.py']`. So the `filename` is the original reponame instead of the renamed one.

I am not sure if this is a caching issue or not or how I can resolve it.

### Steps to reproduce the bug

```
from datasets import load_dataset


ds = load_dataset(
    "BramVanroy/hplt-mono-v1-2",
    "ky",
    trust_remote_code=True
)
```

### Expected behavior

That the most recent repo name is used when `filename` is generated.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0
GitHub issue: Hi,do you know how to load the dataset from local file now?
Author: Gera001 (NONE)
State: open | Created: 2024-03-07 13:58:40

              Hi, if I want to load the dataset from local file, then how to specify the configuration name?

_Originally posted by @WHU-gentle in https://github.com/huggingface/datasets/issues/2976#issuecomment-1333455222_
GitHub issue: Is there any way to solve hanging of IterableDataset using split by node + filtering during inference
Author: ssharpe42 (NONE)
State: open | Created: 2024-03-05 15:55:13

### Describe the bug

I am using an iterable dataset in a multi-node setup, trying to do training/inference while filtering the data on the fly. I usually do not use `split_dataset_by_node` but it is very slow using the IterableDatasetShard in `accelerate` and `transformers`. When I filter after applying `split_dataset_by_node`, it results in shards that are not equal sizes due to unequal samples filtered from each one. 

The distributed process hangs when trying to accomplish this. Is there any way to resolve this or is it impossible to implement?



### Steps to reproduce the bug

Here is a toy example of what I am trying to do that reproduces the behavior

```
# torchrun --nproc-per-node 2 file.py


import os

import pandas as pd
import torch
from accelerate import Accelerator
from datasets import Features, Value, load_dataset
from datasets.distributed import split_dataset_by_node
from torch.utils.data import DataLoader

accelerator = Accelerator(device_placement=True, dispatch_batches=False)
if accelerator.is_main_process:
    if not os.path.exists("scratch_data"):
        os.mkdir("scratch_data")

    n_shards = 4
    for i in range(n_shards):
        df = pd.DataFrame({"id": list(range(10 * i, 10 * (i + 1)))})
        df.to_parquet(f"scratch_data/shard_{i}.parquet")


world_size = accelerator.num_processes
local_rank = accelerator.process_index


def collate_fn(examples):
    input_ids = []
    for example in examples:
        input_ids.append(example["id"])
    return torch.LongTensor(input_ids)


dataset = load_dataset(
    "parquet", data_dir="scratch_data", split="train", streaming=True
)
dataset = (
    split_dataset_by_node(dataset, rank=local_rank, world_size=world_size)
    .filter(lambda x: x["id"] < 35)
    .shuffle(seed=42, buffer_size=100)
)

batch_size = 2
train_dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    collate_fn=collate_fn,
    num_workers=2
)  

for x in train_dataloader:
    x = x.to(accelerator.device)
    print({"rank": local_rank, "id": x})
   
    y = accelerator.gather_for_metrics(x)
    if accelerator.is_main_process:
        print("gathered", y)


```

### Expected behavior

Is there any way to continue training/inference on the GPUs that have remaining data left without waiting for the others? Is it impossible to filter when 

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.10.209-198.812.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.6.0
GitHub issue: Non-deterministic `Dataset.builder_name` value
Author: harupy (NONE)
State: closed | Created: 2024-03-05 09:23:21

### Describe the bug

I'm not sure if this is a bug, but `print(ds.builder_name)` in the following code sometimes prints out `rotten_tomatoes` instead of `parquet`:

```python
import datasets

for _ in range(100):
    ds = datasets.load_dataset("rotten_tomatoes", split="train")
    print(ds.builder_name)  # prints out "rotten_tomatoes" sometimes instead of "parquet"
```

Output:

```
...
parquet
parquet
parquet
rotten_tomatoes
parquet
parquet
parquet
...
```

Here's a reproduction using GitHub Actions:
https://github.com/mlflow/mlflow/actions/runs/8153247984/job/22284263613?pr=11329#step:12:241

One of our tests is flaky because `builder_name` is not deterministic.

### Steps to reproduce the bug

1. Run the code above.

### Expected behavior

Always prints out `parquet`?

### Environment info

```
Copy-and-paste the text below in your GitHub issue.
- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-1015-azure-x86_64-with-glibc2.34
- Python version: 3.8.18
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0
```
GitHub issue: Fix sliced ConcatenationTable pickling with mixed schemas vertically
Author: lhoestq (MEMBER)
State: closed | Created: 2024-03-04 21:02:07

A sliced + pickled ConcatenationTable could end up with a different schema than the original schema, if the slice only contains blocks with only a subset of the columns.

This can lead to issues when saving datasets from a concatenation of datasets with mixed schemas

Reported in https://discuss.huggingface.co/t/datasetdict-save-to-disk-with-num-proc-1-seems-to-hang-with-error/75595
GitHub issue: Bump huggingface-hub lower version to 0.21.2
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-03-04 13:00:52

This should fix the version compatibility issue when using `huggingface_hub` < 0.21.2 and latest fsspec (>=2023.12.0).

See my comment: https://github.com/huggingface/datasets/pull/6687#issuecomment-1976493336
>> EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`
>
>Please note that people using `huggingface_hub` < 0.21.2 and latest `fsspec` will have issues when using `datasets`:
>- https://github.com/huggingface/lighteval/actions/runs/8139147047/job/22241658122?pr=86
>- https://github.com/huggingface/lighteval/pull/84

CC: @clefourrier