helmo/github-issues huggingface dataset:
Why not use self._epoch as seed to shuffle in distributed training with IterableDataset
State: closed | Author: npuichigo (CONTRIBUTOR)

### Describe the bug

Currently, distributed training with `IterableDataset` needs to pass fixed seed to shuffle to keep each node use the same seed to avoid overlapping.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1174-L1177

My question is why not directly use `self._epoch` which is set by `set_epoch` as seed? It's almost the same across nodes.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1790-L1801

If not using `self._epoch` as shuffling seed, what does this method do to prepare an epoch seeded generator?
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1206



### Steps to reproduce the bug

As mentioned above.

### Expected behavior

As mentioned above.

### Environment info

Not related

Comments: @lhoestq 

`_effective_generator` returns a RNG that takes into account `self._epoch` and the current dataset's base shuffling RNG (which can be set by specifying `seed=` in `.shuffle() for example`).

To fix your error you can pass `seed=` to `.shuffle()`. And the shuffling will depend on both this seed and `self._epoch`

Thanks for the reply
FolderBase Dataset automatically resolves under current directory when data_dir is not specified
State: open | Author: npuichigo (CONTRIBUTOR)
Labels: name: good first issue

### Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:
```
load_dataset("audiofolder")
```
takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling https://github.com/huggingface/datasets/blob/cb8c5de5145c7e7eee65391cb7f4d92f0d565d62/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58-L59

### Steps to reproduce the bug

```
load_dataset("audiofolder")
```

### Expected behavior

Error report

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

Comments: @lhoestq 

Makes sense, I guess this can be fixed in the load_dataset_builder method.
It concerns every packaged builder I think (see values in `_PACKAGED_DATASETS_MODULES`)

I think the behavior is related to these lines, which short circuited the error handling.
https://github.com/huggingface/datasets/blob/664a1cb72ea1e6ef7c47e671e2686ca4a35e8d63/src/datasets/load.py#L946-L952

So should data_dir be checked here or still delegating to actual `DatasetModule`? In that case, how to properly set `data_files` here.

This is location in PackagedDatasetModuleFactory.get_module seems the be the right place to check if at least data_dir or data_files are passed

@mariosasko can you please assign this issue to me,I want to work on this

#self-assign

@mariosasko is this issue still open? i would love to kickstart my journey to open source with this issue!
Regards
zutarich

@zutarich It is unless @debrupf2946 is working on it.

#self-assign

I am working and will open a pull request soon @Etelis 


@mariosasko can i take this up? 

#self-assign

Yes, feel free to work on this :)

i think its working as expected . Heres the log i get for the same line -

![image](https://github.com/huggingface/datasets/assets/63234112/8a857ec5-8dd0-4b01-b3a7-7c93444b9558)


#self-assign

Hey! I saw that PR #7314 fixed the issue when no matching data files are found, which is great. But I think the builder still defaults to the current directory when `data_dir` isn't specified, which could lead to unexpected behavior.

If no one’s actively working on that part, I’d be happy to pick it up and propose a fix. Let me know if that works!


Cool, feel free to ping me if you open a PR @aniyagnik :)
Faster sorting for single key items
State: closed | Author: jackapbutler (NONE)
Labels: name: enhancement

### Feature request

A faster way to sort a dataset which contains a large number of rows.

### Motivation

The current sorting implementations took significantly longer than expected when I was running on a dataset trying to sort by timestamps. 

**Code snippet:**
```python
ds = datasets.load_dataset( "json", **{"data_files": {"train": "path-to-jsonlines"}, "split": "train"}, num_proc=os.cpu_count(), keep_in_memory=True) 
sorted_ds = ds.sort("pubDate", keep_in_memory=True)
```

However, once I switched to a different method which
1. unpacked to a list of tuples
2. sorted tuples by key
3. run `.select` with the sorted list of indices
It was significantly faster (orders of magnitude, especially with M's of rows)

### Your contribution

I'd be happy to implement a crude single key sorting algorithm so that other users can benefit from this trick. Broadly, this would take a `Dataset` and perform;

```python
# ds is a Dataset object
# key_name is the sorting key

class Dataset:
   ...
   def _sort(key_name: str) -> Dataset:
      index_keys = [(i,x) for i,x in enumerate(self[key_name])]
      sorted_rows = sorted(row_pubdate, key=lambda x: x[1])
      sorted_indicies = [x[0] for x in sorted_rows]
      return self.select(sorted_indicies)
```

Comments: `Dataset.sort` essentially does the same thing except it uses `pyarrow.compute.sort_indices` which doesn't involve copying the data into python objects (saving memory)

```python
sort_keys = [(col, "ascending") for col in column_names]
indices = pc.sort_indices(self.data, sort_keys=sort_keys)
return self.select(indices)
```

Ok interesting, I'll continue debugging to see what is going wrong on my end.
Allow dataset implement .take
State: open | Author: brando90 (NONE)
Labels: name: enhancement

### Feature request

I want to do:
```
dataset.take(512)
```

but it only works with streaming = True

### Motivation

uniform interface to data sets. Really surprising the above only works with streaming = True. 

### Your contribution

Should be trivial to copy paste the IterableDataset .take to use the local path in the data (when streaming = False)

Comments: ```
    dataset = IterableDataset(dataset) if type(dataset) != IterableDataset else dataset  # to force dataset.take(batch_size) to work in non-streaming mode
   ```


hf discuss: https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770

so: https://stackoverflow.com/questions/76902824/how-does-one-make-dataset-take512-work-with-streaming-false-with-hugging-fac

Feel free to work on this. In addition, `IterableDataset` supports `skip`, so we should also add this method to `Dataset`.
Dataset.from_parquet cannot load subset of columns
State: closed | Author: dwyatte (CONTRIBUTOR)

### Describe the bug

When using `Dataset.from_parquet(path_or_paths, columns=[...])` and a subset of columns, loading fails with a variant of the following

```
ValueError: Couldn't cast
a: int64
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [], "columns": [{"name":' + 273
to
{'a': Value(dtype='int64', id=None), 'b': Value(dtype='int64', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:
```

Looks to be triggered by https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/table.py#L2285-L2286

### Steps to reproduce the bug

```
import pandas as pd
from datasets import Dataset


pd.DataFrame([{"a": 1, "b": 2}]).to_parquet("test.pq")
Dataset.from_parquet("test.pq", columns=["a"])
```

### Expected behavior

A subset of columns should be loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.10.0-23-cloud-amd64-x86_64-with-glibc2.2.5
- Python version: 3.8.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

Comments: Looks like this regression was introduced in `datasets==2.13.0` (`2.12.0` could load a subset of columns)

This does not appear to be fixed by https://github.com/huggingface/datasets/pull/6045 (bug still exists on `main`)
ValueError when running BeamBasedBuilder with GCS path in cache_dir
State: closed | Author: ktrk115 (NONE)

### Describe the bug

When running the BeamBasedBuilder with a GCS path specified in the cache_dir, the following ValueError occurs:

```
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://my-bucket/huggingface_datasets/my_beam_dataset/default/0.0.0/my_beam_dataset-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']
```

Same error occurs after running `pip install apache-beam[gcp]` as instructed.

### Steps to reproduce the bug

Put `my_beam_dataset.py`:

```python
import datasets


class MyBeamDataset(datasets.BeamBasedBuilder):
    def _info(self):
        features = datasets.Features({"value": datasets.Value("int64")})
        return datasets.DatasetInfo(features=features)

    def _split_generators(self, dl_manager, pipeline):
        return [datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={})]

    def _build_pcollection(self, pipeline):
        import apache_beam as beam

        return pipeline | beam.Create([{"value": i} for i in range(10)])
```

Run:
```bash
datasets-cli run_beam my_beam_dataset.py --cache_dir=gs://my-bucket/huggingface_datasets/ --beam_pipeline_options="runner=DirectRunner"
```

### Expected behavior

Running the BeamBasedBuilder with a GCS cache path without any errors.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.3

Comments: The cause of the error seems to be that `datasets` adds "gcs://" as a schema, while `beam` checks only "gs://".

datasets: https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/builder.py#L822

beam: [link](https://github.com/apache/beam/blob/25e1a64641b1c8a3c0a6c75c6e86031b87307f22/sdks/python/apache_beam/io/filesystems.py#L98-L101)
```
      systems = [
          fs for fs in FileSystem.get_all_subclasses()
          if fs.scheme() == path_scheme
      ]
```

Hi! We've deprecated the Beam API, as we don't have the bandwidth to support it properly...
custom load dataset to hub
State: closed | Author: andysingal (NONE)

### System Info

kaggle notebook

i transformed dataset:
```
dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
```
to 
formatted_dataset: 
```
Dataset({
    features: ['message_tree_id', 'message_tree_text'],
    num_rows: 33143
})
```
but would like to know how to upload to hub

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

shared above

### Expected behavior

load dataset to hub

Comments: This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

> This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

Thanks @sgugger , I guess I will wait for them to address the issue . Looking forward to hearing from them 

You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

> You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot ! 

> > You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.
> 
> how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot !

I solved it by upgrading `Datasets` version to 2.15.0
Misalignment between file format specified in configs metadata YAML and the inferred builder
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

There is a misalignment between the format of the `data_files` specified in the configs metadata YAML (CSV):
```yaml
configs:
  - config_name: default
    data_files:
      - split: train
        path: data.csv
``` 
and the inferred builder (JSON). Note there are multiple JSON files in the repo, but they do not appear in the configs metadata YAML.

See: https://huggingface.co/datasets/freddyaboulton/chatinterface_with_image_csv/discussions/1

CC: @freddyaboulton @polinaeterna
Offline dataset viewer
State: closed | Author: yuvalkirstain (NONE)
Labels: name: enhancement

name: dataset-viewer

### Feature request

The dataset viewer feature is very nice. It enables to the user to easily view the dataset. However, when working for private companies we cannot always upload the dataset to the hub. Is there a way to create dataset viewer offline? I.e. to run a code that will open some kind of html or something that makes it easy to view the dataset.

### Motivation

I want to easily view my dataset even when it is hosted locally.

### Your contribution

N.A.

Comments: Hi, thanks for the suggestion. It's not possible at the moment. The viewer is part of the Hub codebase and only works on public datasets. Also, it relies on [Datasets Server](https://github.com/huggingface/datasets-server/), which prepares the data and provides an API to access the rows, size, etc.

If you're interested in hosting your data as a private dataset on the Hub, you might want to look at https://github.com/huggingface/datasets-server/issues/39.

Hi, we are building an offline dataset viewer: https://github.com/Renumics/spotlight
It supports many HF datasets, but currently you have to use it via Pandas:
df=ds.to_pandas()
spotlight.show(df)

Would love to hear from you if that works for your use case. If not, feel free to open an issue on the repo: https://github.com/Renumics/spotlight/issues

@ssuwelack thank you! I will definitely try it out.

Related issues:
- https://github.com/huggingface/datasets-server/issues/213
- https://github.com/huggingface/datasets-server/issues/441
- https://github.com/huggingface/datasets/issues/6014

Closing for now, as developing and maintaining an offline viewer is not planned.

@yuvalkirstain the dataset viewer is now available on private datasets for [PRO users](https://huggingface.co/pricing#pro) and [Enterprise Hub orgs](https://huggingface.co/enterprise). Would it fit your needs?

Hi @ssuwelack I tried loading a HF dataset with your viewer but got this error https://github.com/Renumics/spotlight/issues/461 hope the team can help me on this. Thanks!
CI check_code_quality error: E721 Do not compare types, use `isinstance()`
State: closed | Author: albertvillanova (MEMBER)
Labels: name: maintenance

After latest release of `ruff` (https://pypi.org/project/ruff/0.0.284/), we get the following CI error:
```
src/datasets/utils/py_utils.py:689:12: E721 Do not compare types, use `isinstance()`
```
Dataset is slower after calling `to_iterable_dataset`
State: open | Author: npuichigo (CONTRIBUTOR)

### Describe the bug

Can anyone explain why looping over a dataset becomes slower after calling `to_iterable_dataset` to convert to `IterableDataset`

### Steps to reproduce the bug

Any dataset after converting to `IterableDataset`

### Expected behavior

Maybe it should be faster on big dataset? I only test on small dataset

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

Comments: @lhoestq 

It's roughly the same code between the two so we can expected roughly the same speed, could you share a benchmark ?
Reinforcement Learning and Robotics are not task categories in HF datasets metadata
State: closed | Author: StoneT2000 (NONE)

### Describe the bug

In https://huggingface.co/models there are task categories for RL and robotics but none in https://huggingface.co/datasets

Our lab is currently moving our datasets over to hugging face and would like to be able to add those 2 tags

Moreover we see some older datasets that do have that tag, but we can't seem to add it ourselves.

### Steps to reproduce the bug

1. Create a new dataset on Hugging face
2. Try to type reinforcemement-learning or robotics into the tasks categories, it does not allow you to commit

### Expected behavior

Expected to be able to add RL and robotics as task categories as some previous datasets have these tags

### Environment info

N/A
Upload README via `push_to_hub`
State: closed | Author: liyucheng09 (NONE)
Labels: name: enhancement

### Feature request

`push_to_hub` now allows users to upload datasets programmatically. However, based on the latest doc, we still need to open the dataset page to add readme file manually.

However, I do discover snippets to intialize a README for every `push_to_hub`:
```
dataset_card = (
    DatasetCard(
        "---\n"
        + str(dataset_card_data)
        + "\n---\n"
        + f'# Dataset Card for "{repo_id.split("/")[-1]}"\n\n[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)'
    )
    if dataset_card is None
    else dataset_card
)

HfApi(endpoint=config.HF_ENDPOINT).upload_file(
    path_or_fileobj=str(dataset_card).encode(),
    path_in_repo="README.md",
    repo_id=repo_id,
    token=token,
    repo_type="dataset",
    revision=branch,
)
```

So, if we can enable `push_to_hub` to upload a readme file by ourselves instead of using the auto generated ones, it can save ton of time, and will definitely alleviate the current "lack-of-dataset-card" situation.

### Motivation

as elabrated above.

### Your contribution

I might be able to make a pr.

Comments: You can use `huggingface_hub`'s [Card API](https://huggingface.co/docs/huggingface_hub/package_reference/cards) to programmatically push a dataset card to the Hub.
Lookahead streaming support?
State: open | Author: PicoCreator (NONE)
Labels: name: enhancement

### Feature request

From what I understand, streaming dataset currently pulls the data, and process the data as it is requested.
This can introduce significant latency delays when data is loaded into the training process, needing to wait for each segment.

While the delays might be dataset specific (or even mapping instruction/tokenizer specific)

Is it possible to introduce a `streaming_lookahead` parameter, which is used for predictable workloads (even shuffled dataset with fixed seed). As we can predict in advance what the next few datasamples will be. And fetch them while the current set is being trained.

With enough CPU & bandwidth to keep up with the training process, and a sufficiently large lookahead, this will reduce the various latency involved while waiting for the dataset to be ready between batches.

### Motivation

Faster streaming performance, while training over extra large TB sized datasets

### Your contribution

I currently use HF dataset, with pytorch lightning trainer for RWKV project, and would be able to help test this feature if supported.

Comments: In which format is your dataset? We could expose the `pre_buffer` flag for Parquet to use PyArrow's background thread pool to speed up loading.
[Docs] The "Process" how-to guide lacks description of `select_columns` function
State: closed | Author: unifyh (CONTRIBUTOR)
Labels: name: enhancement

### Feature request

The [how to process dataset guide](https://huggingface.co/docs/datasets/main/en/process) currently does not mention the [`select_columns`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.select_columns) function. It would be nice to include it in the guide.

### Motivation

This function is a commonly requested feature (see this [forum thread](https://discuss.huggingface.co/t/how-to-create-a-new-dataset-from-another-dataset-and-select-specific-columns-and-the-data-along-with-the-column/15120) and #5468 #5474). However, it has not been included in the guide since its implementation by PR #5480.

Mentioning it in the guide would help future users discover this added feature.

### Your contribution

I could submit a PR to add a brief description of the function to said guide.

Comments: Great idea, feel free to open a PR! :)
[BUG] Dataset initialized from in-memory data does not create cache.
State: closed | Author: MattYoon (NONE)

### Describe the bug

`Dataset` initialized from in-memory data (dictionary in my case, haven't tested with other types) does not create cache when processed with the `map` method, unlike `Dataset` initialized by other methods such as `load_dataset`.

### Steps to reproduce the bug

```python
# below code was run the second time so the map function can be loaded from cache if exists

from datasets import load_dataset, Dataset

dataset = load_dataset("tatsu-lab/alpaca")['train']

dataset = dataset.map(lambda x: {'input': x['input'] + 'hi'}) # some random map
print(len(dataset.cache_files))
# 1

# copy the exact same data but initialize from a dictionary
memory_dataset = Dataset.from_dict({
    'instruction': dataset['instruction'],
    'input': dataset['input'],
    'output': dataset['output'],
    'text': dataset['text']})

memory_dataset = memory_dataset.map(lambda x: {'input': x['input'] + 'hi'}) # exact same map
print(len(memory_dataset.cache_files))
# Map: 100%|██████████| 52002[/52002]
# 0
```

### Expected behavior

The `map` function should create cache regardless of the method the `Dataset` was created.

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments: This is expected behavior. You must provide `cache_file_name` when performing `.map` on an in-memory dataset for the result to be cached.
load local json_file as dataset
State: closed | Author: CiaoHe (NONE)

### Describe the bug

I tried to load local json file as dataset but failed to parsing json file because some columns are 'float' type.

### Steps to reproduce the bug

1. load json file with certain columns are 'float' type. For example `data = load_data("json", data_files=JSON_PATH)`
2. Then, the error will be triggered like `ArrowInvalid: Could not convert '-0.2253' with type str: tried to convert to double

### Expected behavior

Should allow some columns are 'float' type, at least it should convert those columns to str type.
I tried to avoid the error by naively convert the float item to str:
```python
# if col type is not str, we need to convert it to str
mapping = {}
for col in keys:
    if isinstance(dataset[0][col], str):
        mapping[col] = [row.get(col) for row in dataset]
    else:
        mapping[col] = [str(row.get(col)) for row in dataset]
```

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

Comments: Hi! We use PyArrow to read JSON files, and PyArrow doesn't allow different value types in the same column. #5776 should address this.

In the meantime, you can combine `Dataset.from_generator` with the above code to cast the values to the same type. 

Thanks for your help!