josteece/github-issues huggingface dataset:
Column order is nondeterministic when loading from JSON
Opened by albertvillanova (closed)

As reported by @meg-huggingface, the order of the JSON object keys is not preserved while loading a dataset from a JSON file with a list of objects.

For example, when loading a JSON files with a list of objects, each with the following ordered keys:
- [ID, Language, Topic], 

the resulting dataset may have columns:
- [ID, Topic, Language], or
- [Topic, Language, ID], or
- [Topic, ID, Language],...

This issue is caused by the use of a Python set (which does not preserve the order):
https://github.com/huggingface/datasets/blob/60d21efbc01e15d0b596ac1072750cbecd91548a/src/datasets/packaged_modules/json/json.py#L168
introduced in
- #5772

Comments:
Add MedImg for streaming
Opened by lhallee (open)

### Feature request

Host the MedImg dataset (similar to Imagenet but for biomedical images). 

### Motivation

There is a clear need for biomedical image foundation models and large scale biomedical datasets that are easily streamable. This would be an excellent tool for the biomedical community.

### Your contribution

MedImg can be found [here](https://www.cuilab.cn/medimg/#).

Comments:
@mariosasko, @lhoestq, @albertvillanova
Hello! Can anyone help? or can you guys suggest who can help with this?

Hi ! Feel free to download the dataset and create a `Dataset` object with it.

Then your'll be able to use `push_to_hub()` to upload the dataset to HF in Parquet format and make it streamable :)

> Hi ! Feel free to download the dataset and create a `Dataset` object with it.
> 
> Then your'll be able to use `push_to_hub()` to upload the dataset to HF in Parquet format and make it streamable :)

The dataset is several TB in total, which I do not have the resources to handle.

Hi @lhoestq and @albertvillanova , just following up about this.

for big datasets you can push_to_hub one part at a time (e.g. as different splits) and merge the parts (just a simple modification in the YAML part of the README)

Sure, that makes sense. However, isn't there a size limit to what typical users can push?

Yes there is a limit, simply let us know by email at datasets [at] huggingface.co - this way we can give you a storage grant also help making sure the dataset is all good for people to use it easily

> Yes there is a limit, simply let us know by email at datasets [at] huggingface.co - this way we can give you a storage grant also help making sure the dataset is all good for people to use it easily

Got it, that would be great.
Support the deserialization of json lines files comprised of lists
Opened by umarbutler (open)

### Feature request

I manage a somewhat large and popular Hugging Face dataset known as the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus). I recently updated my corpus to be stored in a json lines file where each line is an array and each element represents a value at a particular column. Previously, my corpus was stored as a json lines file where each line was a dictionary and the keys were the fields.

Essentially, a line in my json lines file used to look like this:
```json
{"version_id":"","type":"","jurisdiction":"","source":"","citation":"","url":"","when_scraped":"","text":""}
```

And now it looks like this:
```json
["","","","","","","",""]
```

This saves 65 bytes per document and allows me very quickly serialise and deserialise documents via `msgspec`.

After making this change, I found that `datasets` was incapable of deserialising my Corpus without a custom loading script, even if I ensured that the `dataset_info` field in my dataset card contained the desired names of my features.

I would like to request that functionality be added to support this format which is more memory-efficent and faster than using dictionaries.

### Motivation

The [documentation](https://huggingface.co/docs/datasets/en/dataset_script) for creating dataset loading scripts asserts that:
> In the next major release, the new safety features of 🤗 Datasets will disable running dataset loading scripts by default, and you will have to pass trust_remote_code=True to load datasets that require running a dataset script.

I would rather not require my users to pass `trust_remote_code=True` which means that I will need built-in support for this format.

### Your contribution

I would be happy to submit a PR for this if this is something you would incorporate into `datasets` and if I can be pointed to where the code would need to go.

Comments:
Update: I ended up deciding to go back to use lines of dictionaries instead of arrays, not because of this issue as my users would be capable of downloading my corpus without `datasets`, but the speed and storage savings are not currently worth breaking my API and harming the backwards compatibility of each new revision.

With that said, for a static dataset that is not regularly updated like mine, and particularly for extremely large datasets with millions or billions of rows, using arrays could have a meaningful impact, and so there is probably still value in supporting this structure, provided the effort is not too much.
Extraction protocol for arrow files is not defined
Opened by radulescupetru (closed)

### Describe the bug

Passing files with `.arrow` extension into data_files argument, at least when `streaming=True` is very slow.

### Steps to reproduce the bug

 Basically it goes through the `_get_extraction_protocol` method located [here](https://github.com/huggingface/datasets/blob/main/src/datasets/utils/file_utils.py#L820)
The method then looks at some base known extensions where `arrow` is not defined so it proceeds to determine the compression with the magic number method which is slow when dealing with a lot of files which are stored in s3 and by looking at this predefined list, I don't see `arrow` in there either so in the end it return None:

```
MAGIC_NUMBER_TO_COMPRESSION_PROTOCOL = {
    bytes.fromhex("504B0304"): "zip",
    bytes.fromhex("504B0506"): "zip",  # empty archive
    bytes.fromhex("504B0708"): "zip",  # spanned archive
    bytes.fromhex("425A68"): "bz2",
    bytes.fromhex("1F8B"): "gzip",
    bytes.fromhex("FD377A585A00"): "xz",
    bytes.fromhex("04224D18"): "lz4",
    bytes.fromhex("28B52FFD"): "zstd",
}
```

### Expected behavior

My expectation is that `arrow` would be in the known lists so it would return None without going through the magic number method.

### Environment info

datasets 2.19.0

Comments:
Fixed in https://github.com/huggingface/datasets/pull/7083
HTTPError 403 raised by CLI convert_to_parquet when creating script branch on 3rd party repos
Opened by albertvillanova (closed)

CLI convert_to_parquet cannot create "script" branch on 3rd party repos.

It can only create it on repos where the user executing the script has write access.

Otherwise, a 403 Forbidden HTTPError is raised:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/ORG/DATASET/branch/script

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/convert_to_parquet.py", line 92, in run
    create_branch(dataset_id, branch="script", repo_type="dataset", token=token, exist_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 367, in hf_raise_for_status
    raise HfHubHTTPError(message, response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError:  (Request ID: Root=1-6645ee0d-4db1ed8a1fbe04956be15897;139a6e23-df7d-4f62-b5ba-adb6d8e6e696)

403 Forbidden: Forbidden: cannot write to script.
Cannot access content at: https://huggingface.co/api/datasets/ORG/DATASET/branch/script.
If you are trying to create or update content,make sure you have a token with the `write` role.
```

Comments:
[WebDataset] KeyError with user-defined `Features` when a field is missing in an example
Opened by lhoestq (closed)

reported at https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-dalle3-high-quality-captions/discussions/1

```
File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 109, in _generate_examples
    example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
```

Comments:
@lhoestq How difficult of fix is this?

It shouldn't be difficult, I think it's just a matter of adding the missing fields from `self.config.features` in `example` here: before it iterates on image_field_names and audio_field_names. A missing field should have a value set to None

https://github.com/huggingface/datasets/blob/768cb35ede5a6c35fa7545aa3671f3e321c96440/src/datasets/packaged_modules/webdataset/webdataset.py#L113-L116

@lhoestq So like this then?

```    
def _generate_examples(self, tar_paths, tar_iterators):
        image_field_names = [
            field_name for field_name, feature in self.info.features.items() if isinstance(feature, datasets.Image)
        ]
        audio_field_names = [
            field_name for field_name, feature in self.info.features.items() if isinstance(feature, datasets.Audio)
        ]
	
        all_field_names = list(self.config.features.keys())
    
        for tar_idx, (tar_path, tar_iterator) in enumerate(zip(tar_paths, tar_iterators)):
            for example_idx, example in enumerate(self._get_pipeline_from_tar(tar_path, tar_iterator)):
                for field_name in all_field_names:
                    if field_name not in example:
                        if field_name in self.config.features:
                            example[field_name] = self.config.features[field_name]
                        else:
                            example[field_name] = None
    
                # Process image and audio fields
                for field_name in image_field_names + audio_field_names:
                    if example[field_name] is not None:
                        example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
    
                yield f"{tar_idx}_{example_idx}", example
```

Or should we avoid trying add the missing values and just set them to None?

```
            for field_name in all_field_names:
                if field_name not in example:
                    example[field_name] = None
```

Yup this is the solution !

```python
            for field_name in all_field_names:
                if field_name not in example:
                    example[field_name] = None
```

@lhoestq Awesome, thanks! I made a PR with the fixes
List of dictionary features get standardized
Opened by sohamparikh (open)

### Describe the bug

Hi, i’m trying to create a HF dataset from a list using Dataset.from_list.

Each sample in the list is a dict with the same keys (which will be my features). The values for each feature are a list of dictionaries, and each such dictionary has a different set of keys. However, the datasets library standardizes all dictionaries under a feature and adds all possible keys (with None value) from all the dictionaries under that feature.

How can I keep the same set of keys as in the original list for each dictionary under a feature?

### Steps to reproduce the bug

```
from datasets import Dataset

# Define a function to generate a sample with "tools" feature
def generate_sample():
    # Generate random sample data
    sample_data = {
        "text": "Sample text",
        "feature_1": []
    }
    
    # Add feature_1 with random keys for this sample
    feature_1 = [{"key1": "value1"}, {"key2": "value2"}]  # Example feature_1 with random keys
    sample_data["feature_1"].extend(feature_1)
    
    return sample_data

# Generate multiple samples
num_samples = 10
samples = [generate_sample() for _ in range(num_samples)]

# Create a Hugging Face Dataset
dataset = Dataset.from_list(samples)
dataset[0]
```

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1', 'key2': None}, {'key1': None, 'key2': 'value2'}]}```

### Expected behavior

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1'}, {'key2': 'value2'}]}```

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.15.0-1040-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0

Comments:
I think this may be a limitation of the arrow format

Dupe of #5950
datasets template guide :: issue in documentation YAML
Opened by bghira (closed)

### Describe the bug

There is a YAML error at the top of the page, and I don't think it's supposed to be there 

### Steps to reproduce the bug

1. Browse to [this tutorial document](https://github.com/huggingface/datasets/blob/main/templates/README_guide.md)
2. Observe a big red error at the top
3. The rest of the document remains functional

### Expected behavior

I think the YAML block should be displayed or ignored.

### Environment info

N/A

Comments:
Hello, @bghira.

Thanks for reporting. Please note that the text originating the error is not supposed to be valid YAML: it contains the instructions to generate the actual YAML content, that should replace the instructions comment.

On the other hand, I agree that it is not nice to have that YAML error message at the top of the page: 
![Screenshot from 2024-05-14 06-58-02](https://github.com/huggingface/datasets/assets/8515462/28409eb4-99e7-4b24-8eaa-21a65a8f23b2)

I am proposing a change to make the YAML error disappear.

thanks albert! i looked at it for a while to figure it out. i think the `raw` view option is the correct way to look at it?
Unable to load JSON saved using `to_json`
Opened by DarshanDeshpande (closed)

### Describe the bug

Datasets stored in the JSON format cannot be loaded using `json.load()`

### Steps to reproduce the bug

```
import json
from datasets import load_dataset

dataset = load_dataset("squad")
train_dataset, test_dataset = dataset["train"], dataset["validation"]
test_dataset.to_json("full_dataset.json")

# This works
loaded_test = load_dataset("json", data_files="full_dataset.json")

# This fails
loaded_test = json.load(open("full_dataset.json", "r"))
```

### Expected behavior

The JSON should be correctly formatted when writing so that it can be loaded using `json.load()`.

### Environment info

Colab: https://colab.research.google.com/drive/1st1iStFUVgu9ZPvnzSzL4vDeYWDwYpUm?usp=sharing

Comments:
Hi @DarshanDeshpande,

Please note that the default format of the method `Dataset.to_json` is [JSON-Lines](https://jsonlines.org/): it passes `orient="records", lines=True` to `pandas.DataFrame.to_json`. This format is specially useful for large datasets, since unlike regular JSON files, it does not require loading all the data into memory at once, but can be done iteratively by batches.

In order to read this file using the `json` library, you should parse line by line:
```python
with open("full_dataset.json", "r") as f:
    data = [json.loads(line) for line in f]
len(data)
```
Maybe we should explain this better in our docs.

Now we explain this better in out docs:
- #6895
add `with_transform` and/or `set_transform` to IterableDataset
Opened by not-lain (open)

### Feature request

when working with a really large dataset it would save us a lot of time (and compute resources) to use either with_transform or the set_transform from the Dataset class instead of waiting for the entire dataset to map

### Motivation

don't want to wait for a really long dataset to map, this would give IterableDataset an extra advantage over the Dataset class.
reducing time and resources

### Your contribution

I am a little busy with my job search lately, but would post about this feature in my social media.
Apologies again (dad going to kick me out soon), if I ever have some free time I will contribute to making this a reality, but that's going to be hard
      / (┬┬﹏┬┬)\

Comments:
Support WebDataset containing file basenames with dots
Opened by albertvillanova (closed)

Support WebDataset containing file basenames with dots.

Fix #6880.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6888). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I think webdataset splits the file name and extension using the first dot no ?

https://github.com/webdataset/webdataset/blob/945b251a872ec0d337be8f9ea17a9c5b0d017ff3/webdataset/tariterators.py#L226

links to this function that splits on first dot

```python

def base_plus_ext(path):
    """Split off all file extensions.

    Returns base, allext.

    Args:
        path: path with extensions

    Returns:
        path with all extensions removed
    """
    match = re.match(r"^((?:.*/|)[^.]+)[.]([^/]*)$", path)
    if not match:
        return None, None
    return match.group(1), match.group(2)
```

So maybe the original issue is actually due to one of the files containing a dot in its file name that is not for the extension

```python
>>> base_plus_ext("15_Cohen_1-s2.0-S0929664620300449-gr3_lrg-b.png")
('15_Cohen_1-s2', '0-S0929664620300449-gr3_lrg-b.png')
```

Thanks for your review, @lhoestq.

I was not aware that `webdataset` requires filenames without dots in their basenames.

I they can have dots for the extension (that becomes the column name) but not in the key used to group files into samples
FAISS load to None
Opened by brainer3220 (open)

### Describe the bug

I've use FAISS with Datasets and save to FAISS.

Then load to save FAISS then no error, then ds to None

```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Steps to reproduce the bug

# 1.
```python
ds_with_embeddings = ds.map(lambda example: {'embeddings': model(transforms(example['image']).unsqueeze(0)).squeeze()}, batch_size=64)

ds_with_embeddings.add_faiss_index(column='embeddings')

ds_with_embeddings.save_faiss_index('embeddings', 'index.faiss')
```

# 2.
```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Expected behavior

Add column in Datasets.

### Environment info

Google Colab, SageMaker Notebook

Comments:
Hello,

I'm not sure I understand. 
The return value of `ds.load_faiss_index` is None as expected.

I see that loading an Index on a dataset that doesn't have an `embedding` column doesn't raise an Issue. Is that the issue?

So `ds` doesn't have an `embedding` column, but we load an index that looks for it. But this will raise an issue only when calling `ds.search`.
load_dataset with data_dir and cache_dir set fail with not supported
Opened by fah (open)

### Describe the bug

with python 3.11 I execute:
```py
from transformers import Wav2Vec2Processor, Data2VecAudioModel
import torch
from torch import nn
from datasets import load_dataset, concatenate_datasets

# load demo audio and set processor
dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
```
This fails in the last line with 
```log
Found cached dataset librispeech_asr (file:///Users/as/Documents/Project/git/audio2vec/cache/librispeech_asr/clean-data_dir=data/2.1.0/cff5df6e7955c80a67f80e27e7e655de71c689e2d2364bece785b972acb37fe7)
Traceback (most recent call last):
  File "/Users/as/Documents/Project/git/audio2vec/src/music2vec-v1.py", line 7, in <module>
    dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/load.py", line 1810, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/builder.py", line 1113, in as_dataset
    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
```

### Steps to reproduce the bug

I setup an venv with requirements.txt
```txt
transformers==4.40.2
torch==2.2.2
datasets==2.16.0
fsspec==2023.9.2
```

pip freeze is:
```
aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.16.0
dill==0.3.7
filelock==3.14.0
frozenlist==1.4.1
fsspec==2023.9.2
huggingface-hub==0.23.0
idna==3.7
Jinja2==3.1.4
MarkupSafe==2.1.5
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.15
networkx==3.3
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.0.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
regex==2024.4.28
requests==2.31.0
safetensors==0.4.3
six==1.16.0
sympy==1.12
tokenizers==0.19.1
torch==2.2.2
tqdm==4.66.4
transformers==4.40.2
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4

```

I execute this on a M1 Mac.


### Expected behavior

I don't understand the error message. Why is "local" caching not supported. Would it possible to give some additional hint with the error message how to solve this issue?


### Environment info

source ....
python -u example.py

Comments:
CI is broken after jax-0.4.27 release: AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'
Opened by albertvillanova (closed)

After jax-0.4.27 release (https://github.com/google/jax/releases/tag/jax-v0.4.27), our CI is broken with the error:
```Python traceback
AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'. Did you mean: 'devices'?
```

See: https://github.com/huggingface/datasets/actions/runs/8997488610/job/24715736153
```Python traceback
___________________ FormatterTest.test_jax_formatter_device ____________________
[gw1] linux -- Python 3.10.14 /opt/hostedtoolcache/Python/3.10.14/x64/bin/python

self = <tests.test_formatting.FormatterTest testMethod=test_jax_formatter_device>

    @require_jax
    def test_jax_formatter_device(self):
        import jax
    
        from datasets.formatting import JaxFormatter
    
        pa_table = self._create_dummy_table()
        device = jax.devices()[0]
        formatter = JaxFormatter(device=str(device))
        row = formatter.format_row(pa_table)
>       assert row["a"].device() == device
E       AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'. Did you mean: 'devices'?

tests/test_formatting.py:630: AttributeError
```

Comments: