josteece/github-issues huggingface dataset:
Configurations in yaml not working
Opened by tsor13 (open)

### Dataset configurations cannot be created in YAML/README

Hello! I'm trying to follow the docs here in order to create structure in my dataset as added from here (#5331): https://github.com/huggingface/datasets/blob/8b8e6ee067eb74e7965ca2a6768f15f9398cb7c8/docs/source/repository_structure.mdx#L110-L118

I have the exact example in my config file for [my data repo](https://huggingface.co/datasets/tsor13/test):
```
configs:
- config_name: main_data
  data_files: "main_data.csv"
- config_name: additional_data
  data_files: "additional_data.csv"
```

Yet, I'm unable to load different configurations:
```
from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test', use_auth_token=True)
```
returns a single split, `['tsor13--test']`

Does anyone have any insights?

@polinaeterna thank you for adding this feature, it is super useful. Do you happen to have any ideas?

### Steps to reproduce the bug

from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test')

### Expected behavior

I would expect there to be two splits, `main_data` and `additional_data`. However, only `['tsor13--test']` test is returned.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.1

Comments:
Unfortunately, I cannot reproduce this behavior on my machine or Colab - the reproducer returns `['main_data', 'additional_data']` as expected.

Thank you for looking into this, Mario. Is this on [my repository](https://huggingface.co/datasets/tsor13/test), or on another one that you have reproduced? Would you mind pointing me to it if so?

Whoa, in colab I received the correct behavior using my dataset. It must have something to do with my local copy of `datasets` (which again just failed).

I've tried uninstalling/reinstnalling to no avail

hi @tsor13 , I haven't been able to reproduce your issue on `tsor13/test` dataset locally either. reinstalling doesn't help?
Error type: ArrowInvalid Details: Failed to parse string: '[254,254]' as a scalar of type int32
Opened by shishirCTC (open)

### Describe the bug

I am getting the following error while I am trying to upload the CSV sheet to train a model. My CSV sheet content is exactly same as shown in the example CSV file in the Auto Train page. Attaching screenshot of error for reference. I have also tried converting the index of the answer that are integer into string by placing inverted commas and also without inverted commas. 
Can anyone please help me out? 
FYI : I am using Chrome browser.

Error type: ArrowInvalid
Details: Failed to parse string: '[254,254]' as a scalar of type int32

![Screenshot 2023-08-19 165827](https://github.com/huggingface/datasets/assets/90616801/95fad96e-7dce-4bb5-9f83-9f1659a32891)


### Steps to reproduce the bug

Kindly let me know how to fix this?

### Expected behavior

Kindly let me know how to fix this?

### Environment info

Kindly let me know how to fix this?

Comments:
Answered on the forum [here](https://discuss.huggingface.co/t/error-type-arrowinvalid-details-failed-to-parse-string-254-254-as-a-scalar-of-type-int32/51323).

it seems that no answer in this link
Add `BoundingBox` feature
Opened by mariosasko (open)

... to make working with object detection datasets easier. Currently, `Sequence(int_or_float, length=4)` can be used to represent this feature optimally (in the storage backend), so I only see this feature being useful if we make it work with the viewer. Also, bounding boxes usually come in 4 different formats (explained [here](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/)), so we need to decide which one to support (or maybe all of them).

cc @NielsRogge @severo

Comments:
My proposal would look like this:

```python
from datasets import load_dataset
from datasets.features import Sequence, BboxFeature

# load images along with boxes
dataset = load_dataset("imagefolder", data_dir="/path/to/folder", split="train")

# map the boxes column to the appropriate format
dataset = dataset.map(lambda example: map_to_ymin_xmin_ymax_xmax(example))

# cast to feature
dataset = dataset.cast_column('boxes', Sequence(BboxFeature))
````
We could for instance make [BboxFeature](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/BBoxFeature) default to normalized coordinates in the ymin, xmin, ymax, xmax format. This is just one of the many different formats of bounding boxes, so we would for instance default to the format used by Tensorflow datasets.
Why not use self._epoch as seed to shuffle in distributed training with IterableDataset
Opened by npuichigo (closed)

### Describe the bug

Currently, distributed training with `IterableDataset` needs to pass fixed seed to shuffle to keep each node use the same seed to avoid overlapping.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1174-L1177

My question is why not directly use `self._epoch` which is set by `set_epoch` as seed? It's almost the same across nodes.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1790-L1801

If not using `self._epoch` as shuffling seed, what does this method do to prepare an epoch seeded generator?
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1206



### Steps to reproduce the bug

As mentioned above.

### Expected behavior

As mentioned above.

### Environment info

Not related

Comments:
@lhoestq 

`_effective_generator` returns a RNG that takes into account `self._epoch` and the current dataset's base shuffling RNG (which can be set by specifying `seed=` in `.shuffle() for example`).

To fix your error you can pass `seed=` to `.shuffle()`. And the shuffling will depend on both this seed and `self._epoch`

Thanks for the reply
FolderBase Dataset automatically resolves under current directory when data_dir is not specified
Opened by npuichigo (closed)

### Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:
```
load_dataset("audiofolder")
```
takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling https://github.com/huggingface/datasets/blob/cb8c5de5145c7e7eee65391cb7f4d92f0d565d62/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58-L59

### Steps to reproduce the bug

```
load_dataset("audiofolder")
```

### Expected behavior

Error report

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

Comments:
@lhoestq 

Makes sense, I guess this can be fixed in the load_dataset_builder method.
It concerns every packaged builder I think (see values in `_PACKAGED_DATASETS_MODULES`)

I think the behavior is related to these lines, which short circuited the error handling.
https://github.com/huggingface/datasets/blob/664a1cb72ea1e6ef7c47e671e2686ca4a35e8d63/src/datasets/load.py#L946-L952

So should data_dir be checked here or still delegating to actual `DatasetModule`? In that case, how to properly set `data_files` here.

This is location in PackagedDatasetModuleFactory.get_module seems the be the right place to check if at least data_dir or data_files are passed

@mariosasko can you please assign this issue to me,I want to work on this

#self-assign

@mariosasko is this issue still open? i would love to kickstart my journey to open source with this issue!
Regards
zutarich

@zutarich It is unless @debrupf2946 is working on it.

#self-assign

I am working and will open a pull request soon @Etelis 


@mariosasko can i take this up? 

#self-assign

Yes, feel free to work on this :)

i think its working as expected . Heres the log i get for the same line -

![image](https://github.com/huggingface/datasets/assets/63234112/8a857ec5-8dd0-4b01-b3a7-7c93444b9558)


#self-assign

Hey! I saw that PR #7314 fixed the issue when no matching data files are found, which is great. But I think the builder still defaults to the current directory when `data_dir` isn't specified, which could lead to unexpected behavior.

If no one’s actively working on that part, I’d be happy to pick it up and propose a fix. Let me know if that works!


Cool, feel free to ping me if you open a PR @aniyagnik :)

Hey! I noticed this is still open and unresolved. I’d like to take this up and submit a PR to prevent resolving to the current directory when data_dir is not specified!!!

Hi @aniyagnik @npuichigo @sahillihas @lhoestq @mariosasko,
I've opened a PR to address the issue discussed in #6152 by adding an early validation check for folder-based datasets. Would appreciate your review whenever convenient. Thanks!

https://github.com/huggingface/datasets/pull/7618
Faster sorting for single key items
Opened by jackapbutler (closed)

### Feature request

A faster way to sort a dataset which contains a large number of rows.

### Motivation

The current sorting implementations took significantly longer than expected when I was running on a dataset trying to sort by timestamps. 

**Code snippet:**
```python
ds = datasets.load_dataset( "json", **{"data_files": {"train": "path-to-jsonlines"}, "split": "train"}, num_proc=os.cpu_count(), keep_in_memory=True) 
sorted_ds = ds.sort("pubDate", keep_in_memory=True)
```

However, once I switched to a different method which
1. unpacked to a list of tuples
2. sorted tuples by key
3. run `.select` with the sorted list of indices
It was significantly faster (orders of magnitude, especially with M's of rows)

### Your contribution

I'd be happy to implement a crude single key sorting algorithm so that other users can benefit from this trick. Broadly, this would take a `Dataset` and perform;

```python
# ds is a Dataset object
# key_name is the sorting key

class Dataset:
   ...
   def _sort(key_name: str) -> Dataset:
      index_keys = [(i,x) for i,x in enumerate(self[key_name])]
      sorted_rows = sorted(row_pubdate, key=lambda x: x[1])
      sorted_indicies = [x[0] for x in sorted_rows]
      return self.select(sorted_indicies)
```

Comments:
`Dataset.sort` essentially does the same thing except it uses `pyarrow.compute.sort_indices` which doesn't involve copying the data into python objects (saving memory)

```python
sort_keys = [(col, "ascending") for col in column_names]
indices = pc.sort_indices(self.data, sort_keys=sort_keys)
return self.select(indices)
```

Ok interesting, I'll continue debugging to see what is going wrong on my end.
Allow dataset implement .take
Opened by brando90 (open)

### Feature request

I want to do:
```
dataset.take(512)
```

but it only works with streaming = True

### Motivation

uniform interface to data sets. Really surprising the above only works with streaming = True. 

### Your contribution

Should be trivial to copy paste the IterableDataset .take to use the local path in the data (when streaming = False)

Comments:
```
    dataset = IterableDataset(dataset) if type(dataset) != IterableDataset else dataset  # to force dataset.take(batch_size) to work in non-streaming mode
   ```


hf discuss: https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770

so: https://stackoverflow.com/questions/76902824/how-does-one-make-dataset-take512-work-with-streaming-false-with-hugging-fac

Feel free to work on this. In addition, `IterableDataset` supports `skip`, so we should also add this method to `Dataset`.
Dataset.from_parquet cannot load subset of columns
Opened by dwyatte (closed)

### Describe the bug

When using `Dataset.from_parquet(path_or_paths, columns=[...])` and a subset of columns, loading fails with a variant of the following

```
ValueError: Couldn't cast
a: int64
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [], "columns": [{"name":' + 273
to
{'a': Value(dtype='int64', id=None), 'b': Value(dtype='int64', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:
```

Looks to be triggered by https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/table.py#L2285-L2286

### Steps to reproduce the bug

```
import pandas as pd
from datasets import Dataset


pd.DataFrame([{"a": 1, "b": 2}]).to_parquet("test.pq")
Dataset.from_parquet("test.pq", columns=["a"])
```

### Expected behavior

A subset of columns should be loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.10.0-23-cloud-amd64-x86_64-with-glibc2.2.5
- Python version: 3.8.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

Comments:
Looks like this regression was introduced in `datasets==2.13.0` (`2.12.0` could load a subset of columns)

This does not appear to be fixed by https://github.com/huggingface/datasets/pull/6045 (bug still exists on `main`)
ValueError when running BeamBasedBuilder with GCS path in cache_dir
Opened by ktrk115 (closed)

### Describe the bug

When running the BeamBasedBuilder with a GCS path specified in the cache_dir, the following ValueError occurs:

```
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://my-bucket/huggingface_datasets/my_beam_dataset/default/0.0.0/my_beam_dataset-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']
```

Same error occurs after running `pip install apache-beam[gcp]` as instructed.

### Steps to reproduce the bug

Put `my_beam_dataset.py`:

```python
import datasets


class MyBeamDataset(datasets.BeamBasedBuilder):
    def _info(self):
        features = datasets.Features({"value": datasets.Value("int64")})
        return datasets.DatasetInfo(features=features)

    def _split_generators(self, dl_manager, pipeline):
        return [datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={})]

    def _build_pcollection(self, pipeline):
        import apache_beam as beam

        return pipeline | beam.Create([{"value": i} for i in range(10)])
```

Run:
```bash
datasets-cli run_beam my_beam_dataset.py --cache_dir=gs://my-bucket/huggingface_datasets/ --beam_pipeline_options="runner=DirectRunner"
```

### Expected behavior

Running the BeamBasedBuilder with a GCS cache path without any errors.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.3

Comments:
The cause of the error seems to be that `datasets` adds "gcs://" as a schema, while `beam` checks only "gs://".

datasets: https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/builder.py#L822

beam: [link](https://github.com/apache/beam/blob/25e1a64641b1c8a3c0a6c75c6e86031b87307f22/sdks/python/apache_beam/io/filesystems.py#L98-L101)
```
      systems = [
          fs for fs in FileSystem.get_all_subclasses()
          if fs.scheme() == path_scheme
      ]
```

Hi! We've deprecated the Beam API, as we don't have the bandwidth to support it properly...
custom load dataset to hub
Opened by andysingal (closed)

### System Info

kaggle notebook

i transformed dataset:
```
dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
```
to 
formatted_dataset: 
```
Dataset({
    features: ['message_tree_id', 'message_tree_text'],
    num_rows: 33143
})
```
but would like to know how to upload to hub

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

shared above

### Expected behavior

load dataset to hub

Comments:
This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

> This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

Thanks @sgugger , I guess I will wait for them to address the issue . Looking forward to hearing from them 

You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

> You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot ! 

> > You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.
> 
> how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot !

I solved it by upgrading `Datasets` version to 2.15.0
Misalignment between file format specified in configs metadata YAML and the inferred builder
Opened by albertvillanova (closed)

There is a misalignment between the format of the `data_files` specified in the configs metadata YAML (CSV):
```yaml
configs:
  - config_name: default
    data_files:
      - split: train
        path: data.csv
``` 
and the inferred builder (JSON). Note there are multiple JSON files in the repo, but they do not appear in the configs metadata YAML.

See: https://huggingface.co/datasets/freddyaboulton/chatinterface_with_image_csv/discussions/1

CC: @freddyaboulton @polinaeterna 

Comments:
Offline dataset viewer
Opened by yuvalkirstain (closed)

### Feature request

The dataset viewer feature is very nice. It enables to the user to easily view the dataset. However, when working for private companies we cannot always upload the dataset to the hub. Is there a way to create dataset viewer offline? I.e. to run a code that will open some kind of html or something that makes it easy to view the dataset.

### Motivation

I want to easily view my dataset even when it is hosted locally.

### Your contribution

N.A.

Comments:
Hi, thanks for the suggestion. It's not possible at the moment. The viewer is part of the Hub codebase and only works on public datasets. Also, it relies on [Datasets Server](https://github.com/huggingface/datasets-server/), which prepares the data and provides an API to access the rows, size, etc.

If you're interested in hosting your data as a private dataset on the Hub, you might want to look at https://github.com/huggingface/datasets-server/issues/39.

Hi, we are building an offline dataset viewer: https://github.com/Renumics/spotlight
It supports many HF datasets, but currently you have to use it via Pandas:
df=ds.to_pandas()
spotlight.show(df)

Would love to hear from you if that works for your use case. If not, feel free to open an issue on the repo: https://github.com/Renumics/spotlight/issues

@ssuwelack thank you! I will definitely try it out.

Related issues:
- https://github.com/huggingface/datasets-server/issues/213
- https://github.com/huggingface/datasets-server/issues/441
- https://github.com/huggingface/datasets/issues/6014

Closing for now, as developing and maintaining an offline viewer is not planned.

@yuvalkirstain the dataset viewer is now available on private datasets for [PRO users](https://huggingface.co/pricing#pro) and [Enterprise Hub orgs](https://huggingface.co/enterprise). Would it fit your needs?

Hi @ssuwelack I tried loading a HF dataset with your viewer but got this error https://github.com/Renumics/spotlight/issues/461 hope the team can help me on this. Thanks!
CI check_code_quality error: E721 Do not compare types, use `isinstance()`
Opened by albertvillanova (closed)

After latest release of `ruff` (https://pypi.org/project/ruff/0.0.284/), we get the following CI error:
```
src/datasets/utils/py_utils.py:689:12: E721 Do not compare types, use `isinstance()`
```

Comments:
Dataset is slower after calling `to_iterable_dataset`
Opened by npuichigo (open)

### Describe the bug

Can anyone explain why looping over a dataset becomes slower after calling `to_iterable_dataset` to convert to `IterableDataset`

### Steps to reproduce the bug

Any dataset after converting to `IterableDataset`

### Expected behavior

Maybe it should be faster on big dataset? I only test on small dataset

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

Comments:
@lhoestq 

It's roughly the same code between the two so we can expected roughly the same speed, could you share a benchmark ?
Reinforcement Learning and Robotics are not task categories in HF datasets metadata
Opened by StoneT2000 (closed)

### Describe the bug

In https://huggingface.co/models there are task categories for RL and robotics but none in https://huggingface.co/datasets

Our lab is currently moving our datasets over to hugging face and would like to be able to add those 2 tags

Moreover we see some older datasets that do have that tag, but we can't seem to add it ourselves.

### Steps to reproduce the bug

1. Create a new dataset on Hugging face
2. Try to type reinforcemement-learning or robotics into the tasks categories, it does not allow you to commit

### Expected behavior

Expected to be able to add RL and robotics as task categories as some previous datasets have these tags

### Environment info

N/A

Comments:
Upload README via `push_to_hub`
Opened by liyucheng09 (closed)

### Feature request

`push_to_hub` now allows users to upload datasets programmatically. However, based on the latest doc, we still need to open the dataset page to add readme file manually.

However, I do discover snippets to intialize a README for every `push_to_hub`:
```
dataset_card = (
    DatasetCard(
        "---\n"
        + str(dataset_card_data)
        + "\n---\n"
        + f'# Dataset Card for "{repo_id.split("/")[-1]}"\n\n[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)'
    )
    if dataset_card is None
    else dataset_card
)

HfApi(endpoint=config.HF_ENDPOINT).upload_file(
    path_or_fileobj=str(dataset_card).encode(),
    path_in_repo="README.md",
    repo_id=repo_id,
    token=token,
    repo_type="dataset",
    revision=branch,
)
```

So, if we can enable `push_to_hub` to upload a readme file by ourselves instead of using the auto generated ones, it can save ton of time, and will definitely alleviate the current "lack-of-dataset-card" situation.

### Motivation

as elabrated above.

### Your contribution

I might be able to make a pr.

Comments:
You can use `huggingface_hub`'s [Card API](https://huggingface.co/docs/huggingface_hub/package_reference/cards) to programmatically push a dataset card to the Hub.