ahmedfarazsyk/datasets-github-issues huggingface dataset:
FORCE_REDOWNLOAD raises "Directory not empty" exception on second run
Author: rcasero
State: open

### Describe the bug

Running `load_dataset(..., download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` twice raises a `Directory not empty` exception on the second run.

### Steps to reproduce the bug

I cannot test this on datasets v2.11.0 due to #5711, but this happens in v2.10.1.

1. Set up a script `my_dataset.py` to generate and load an offline dataset.
2. Load it with 
    ```python
            ds = datasets.load_dataset(path=/path/to/my_dataset.py,
                                       name='toy',
                                       data_dir=/path/to/my_dataset.py,
                                       cache_dir=cache_dir,
                                       download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD,
                                       )
    ```
    It loads fine
    ```
    Dataset my_dataset downloaded and prepared to /path/to/cache/toy-..e05e/1.0.0/...5b4c. Subsequent calls will reuse this data.
    ```
3. Try to load it again with the same snippet and the splits are generated, but at the end of the loading process it raises the error
    ```
    2023-04-11 12:10:19,965: DEBUG:	open file: /path/to/cache/toy-..e05e/1.0.0/...5b4c.incomplete/dataset_info.json
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
        builder_instance.download_and_prepare(
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 852, in download_and_prepare
        with incomplete_dir(self._output_dir) as tmp_output_dir:
      File "/path/to/conda/environment/lib/python3.10/contextlib.py", line 142, in __exit__
        next(self.gen)
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 826, in incomplete_dir
        shutil.rmtree(dirname)
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 730, in rmtree
        onerror(os.rmdir, path, sys.exc_info())
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 728, in rmtree
        os.rmdir(path)
    OSError: [Errno 39] Directory not empty: '/path/to/cache/toy-..e05e/1.0.0/...5b4c'
    ```


### Expected behavior

Regenerate the dataset from scratch and reload it.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-4.18.0-483.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.2


Comments: Hi ! I couldn't reproduce your issue :/

It seems that `shutil.rmtree` failed. It is supposed to work even if the directory is not empty, but you still end up with `OSError: [Errno 39] Directory not empty:`. Can you make sure another process is not using this directory at the same time ?

I have the same error with `datasets==2.14.5` and `pyarrow==13.0.0`. Python 3.10.13

I have same error. Any workaround?
Enwik8 should support the standard split
Author: lucaslingle
State: closed
Labels: name: enhancement

### Feature request

The HuggingFace Datasets library currently supports two BuilderConfigs for Enwik8. One config yields individual lines as examples, while the other config yields the entire dataset as a single example. Both support only a monolithic split: it is all grouped as "train". 

The HuggingFace Datasets library should include a BuilderConfig for Enwik8 with train, validation, and test sets derived from the first 90 million bytes, next 5 million bytes, and last 5 million bytes, respectively. This Enwik8 split is standard practice in LM papers, as elaborated and motivated below. 

### Motivation

Enwik8 is commonly split into 90M, 5M, 5M consecutive bytes. This is done in the Transformer-XL [codebase](https://github.com/kimiyoung/transformer-xl/blob/44781ed21dbaec88b280f74d9ae2877f52b492a5/getdata.sh#L34), and is additionally mentioned in the Sparse Transformers [paper](https://arxiv.org/abs/1904.10509) and the Compressive Transformers [paper](https://arxiv.org/abs/1911.05507). This split is pretty much universal among language modeling papers. 

One may obtain the splits by manual wrangling, using the data yielded by the ```enwik8-raw``` BuilderConfig. However, this undermines the seamless functionality of the library: one must slice the single raw example, extract it into three tensors, and wrap each in a separate dataset. 

This becomes even more of a nuisance if using the current Enwik8 HuggingFace dataset as a TfdsDataSource with [SeqIO](https://github.com/google/seqio), where a pipeline of preprocessors is typically included in a SeqIO Task definition, to be applied immediately after loading the data with TFDS. 

### Your contribution

Supporting this functionality in HuggingFace Datasets will only require an additional BuilderConfig for Enwik8 and a few additional lines of code. I will submit a PR. 

Comments: #self-assign

The Enwik8 pipeline is not present in this codebase, and is hosted elsewhere. I have opened a PR [there](https://huggingface.co/datasets/enwik8/discussions/4) instead.
The order of data split names is nondeterministic
Author: albertvillanova
State: closed
Labels: name: bug

After this CI error: https://github.com/huggingface/datasets/actions/runs/4639528358/jobs/8210492953?pr=5718
```
FAILED tests/test_data_files.py::test_get_data_files_patterns[data_file_per_split4] - AssertionError: assert ['random', 'train'] == ['train', 'random']
  At index 0 diff: 'random' != 'train'
  Full diff:
  - ['train', 'random']
  + ['random', 'train']
```
I have checked locally and found out that the data split order is nondeterministic.

This is caused by the use of `set` for sharded splits.
Fallback JSON Dataset loading does not load all values when features specified manually
Author: myluki2000
State: closed

### Describe the bug

The fallback JSON dataset loader located here:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L130-L153

does not load the values of features correctly when features are specified manually and not all features have a value in the first entry of the dataset. I'm pretty sure this is not supposed to be expected bahavior?

To fix this you'd have to change this line:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L140

To pass a schema to pyarrow which has the same structure as the features argument passed to the load_dataset() method.

### Steps to reproduce the bug

Consider a dataset JSON like this:

```
[
{
	"instruction": "Do stuff",
	"output": "Answer stuff"
},
{
	"instruction": "Do stuff2",
	"input": "Additional Input2",
	"output": "Answer stuff2"
}
]
```

Using this code to load the dataset:

```
from datasets import load_dataset, Features, Value

features = {
    "instruction": Value("string"),
    "input": Value("string"),
    "output": Value("string")
}
features = Features(features)

ds = load_dataset("json", data_files="./ds.json", features=features)

for row in ds["train"]:
    print(row)
```

we get a dataset that looks like this:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | None               | "Answer Stuff2" |

### Expected behavior

The input column should contain values other than None for dataset entries that have the "input" attribute set:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | "Additional Input2" | "Answer Stuff2" |

### Environment info

Python 3.10.10
Datasets 2.11.0
Windows 10

Comments: Thanks for reporting, @myluki2000.

I am working on a fix.
How to limit the number of examples in dataset, for testing?
Author: ndvbd
State: closed

### Describe the bug

I am using this command:
`data = load_dataset("json", data_files=data_path)`
However, I want to add a parameter, to limit the number of loaded examples to be 10, for development purposes, but can't find this simple parameter.

### Steps to reproduce the bug

In the description.

### Expected behavior

To be able to limit the number of examples

### Environment info

Nothing special

Comments: Hi! You can use the `nrows` parameter for this:
```python
data = load_dataset("json", data_files=data_path, nrows=10)
```

@mariosasko I get:

`TypeError: __init__() got an unexpected keyword argument 'nrows'`

I misread the format in which the dataset is stored -  the `nrows` parameter works for CSV, but not JSON.

This means the only option is first to create a DataFrame and then convert it to a Dataset object:
```python
import pandas as pd
from datasets import Dataset

df = pd.read_json(data_path, lines=True, nrows=10)
ds = Dataset.from_pandas(df)
```
Calling datasets.load_dataset("text" ...) results in a wrong split.
Author: cyrilzakka
State: open

### Describe the bug

When creating a text dataset, the training split should have the bulk of the examples by default. Currently, testing does.

### Steps to reproduce the bug

I have a folder with 18K text files in it. Each text file essentially consists in a document or article scraped from online. Calling the following codeL
```
folder_path = "/home/cyril/Downloads/llama_dataset"
data = datasets.load_dataset("text", data_dir=folder_path)
data.save_to_disk("/home/cyril/Downloads/data.hf")

data = datasets.load_from_disk("/home/cyril/Downloads/data.hf")
print(data)
```

Results in the following split:
```
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 2114
    })
    test: Dataset({
        features: ['text'],
        num_rows: 200882
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 152
    })
})
```

It seems to me like the train/test/validation splits are in the wrong order since test split >>>> train_split

### Expected behavior

Train split should have the bulk of the training examples.

### Environment info

datasets 2.11.0, python 3.10.6
Handle empty audio
Author: zyb8543d
State: closed

Some audio paths exist, but they are empty, and an error will be reported when reading the audio path.How to use the filter function to avoid the empty audio path?
when a audio is empty, when do resample , it will break:
`array, sampling_rate = sf.read(f) array = librosa.resample(array, orig_sr=sampling_rate, target_sr=self.sampling_rate)`

Comments: Hi! Can you share one of the problematic audio files with us?

I tried to reproduce the error with the following code: 
```python
import soundfile as sf
import numpy as np
from datasets import Audio

sf.write("empty.wav",  np.array([]), 16000)
Audio(sampling_rate=24000).decode_example({"path": "empty.wav", "bytes": None})
```
But without success.

Also, what version of `librosa` is installed in your env? (You can get this info with `python -c "import librosa; print(librosa.__version__)`)



I'm closing this issue as the reproducer hasn't been provided.
Return Numpy Array (fixed length) Mode, in __get_item__, Instead of List
Author: jungbaepark
State: closed
Labels: name: enhancement

### Feature request

There are old known issues, but they can be easily forgettable problems in multiprocessing with pytorch-dataloader:

Too high usage of RAM or shared-memory in pytorch when we set num workers > 1 and returning type of dataset or dataloader is "List" or "Dict".
https://github.com/pytorch/pytorch/issues/13246 

With huggingface datasets, unfortunately, the default return type is the list, so the problem is raised too often if we do not set anything for the issue.


However, this issue can be released when the returning output is fixed in length.
Therefore, I request the mode, returning outputs with fixed length (e.g. numpy array) rather than list.

The design would be good when we load datasets as
```python
load_dataset(..., with_return_as_fixed_tensor=True)
```

### Motivation

The general solution for this issue is already in the comments:  https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 

: Numpy or Pandas seems not to have problems, while both have the string type.
(I'm not sure that the sequence of huggingface datasets can solve this problem as well)


### Your contribution

 I'll read it ! thanks

Comments: Hi! 

You can use [`.set_format("np")`](https://huggingface.co/docs/datasets/process#format) to get NumPy arrays (or Pytorch tensors with `.set_format("torch")`) in `__getitem__`.

Also, have you been able to reproduce the linked PyTorch issue with a HF dataset?
Dataset sizes are in MiB instead of MB in dataset cards
Author: albertvillanova
State: closed
Labels: name: bug

name: dataset-viewer

As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations

Comments: Example of bulk edit: https://huggingface.co/datasets/aeslc/discussions/5

looks great! 

Do you encode the fact that you've already converted a dataset? (to not convert it twice) or do you base yourself on the info contained in `dataset_info`

I am only looping trough the dataset cards, assuming that all of them were created with MiB.

I agree we should only run the bulk edit once for all canonical datasets: I'm using a for-loop over canonical datasets.

yes, worst case, we have this in structured data:

<img width="337" alt="image" src="https://user-images.githubusercontent.com/326577/230037051-06caddcb-08c8-4953-a710-f3d122917db3.png">


I have just included as well the conversion from MB to GB if necessary. See: 
- https://huggingface.co/datasets/bookcorpus/discussions/2/files
- https://huggingface.co/datasets/asnq/discussions/2/files

Nice. Is it another loop? Because in https://huggingface.co/datasets/amazon_us_reviews/discussions/2/files we have `32377.29 MB` for example

First, I tested some batches to check the changes made. Then I incorporated the MB to GB conversion. Now I'm running the rest.

The bulk edit parsed 751 canonical datasets and updated 166.

Thanks a lot!

The sizes now match as expected!

<img width="1446" alt="Capture d’écran 2023-04-05 à 16 10 15" src="https://user-images.githubusercontent.com/1676121/230107044-ac2a76ea-a4fe-4e81-a925-f464b85f5edd.png">


I made another bulk edit of ancient canonical datasets that were moved to community organization. I have parsed 11 datasets and opened a PR on 3 of them:
- [x] "allenai/scicite": https://huggingface.co/datasets/allenai/scicite/discussions/3
- [x] "allenai/scifact": https://huggingface.co/datasets/allenai/scifact/discussions/2
- [x] "dair-ai/emotion": https://huggingface.co/datasets/dair-ai/emotion/discussions/6

should we force merge the PR and close this issue?

I merged the PRs for "scicite" and "scifact".
Getting next item from IterableDataset took forever.
Author: HongtaoYang
State: closed

### Describe the bug

I have a large dataset, about 500GB. The format of the dataset is parquet. 

I then load the dataset and try to get the first item
```python
def get_one_item():
    dataset = load_dataset("path/to/datafiles", split="train", cache_dir=".", streaming=True)
    dataset = dataset.filter(lambda example: example['text'].startswith('Ar'))
    print(next(iter(dataset)))
```

However, this function never finish. I waited ~10mins, the function was still running so I killed the process. I'm now using `line_profiler` to profile how long it would take to return one item. I'll be patient and wait for as long as it needs. 

I suspect the filter operation is the reason why it took so long. Can I get some possible reasons behind this?

### Steps to reproduce the bug

Unfortunately without my data files, there is no way to reproduce this bug.

### Expected behavior

With `IteralbeDataset`, I expect the first item to be returned instantly.

### Environment info

- datasets version: 2.11.0 
- python: 3.7.12

Comments: Hi! It can take some time to iterate over Parquet files as big as yours, convert the samples to Python, and find the first one that matches a filter predicate before yielding it...

Thanks @mariosasko, I figured it was the filter operation. I'm closing this issue because it is not a bug, it is the expected beheaviour.
Add Qdrant as another search index
Author: kacperlukawski
State: open
Labels: name: enhancement

### Feature request

I'd suggest adding Qdrant (https://qdrant.tech) as another search index available, so users can directly build an index from a dataset. Currently, FAISS and ElasticSearch are only supported: https://huggingface.co/docs/datasets/faiss_es 

### Motivation

ElasticSearch is a keyword-based search system, while FAISS is a vector search library. Vector database, such as Qdrant, is a different tool based on similarity (like FAISS) but is not limited to a single machine. It makes the vector database well-suited for bigger datasets and collaboration if several people want to access a particular dataset.

### Your contribution

I can provide a PR implementing that functionality on my own.

Comments: @mariosasko I'd appreciate your feedback on this.
Shuffle a sharded iterable dataset without seed can lead to duplicate data
Author: lhoestq
State: closed
Labels: name: bug

As reported in https://github.com/huggingface/datasets/issues/5360

If `seed=None` in `.shuffle()`, shuffled datasets don't use the same shuffling seed across nodes.

Because of that, the lists of shards is not shuffled the same way across nodes, and therefore some shards may be assigned to multiple nodes instead of exactly one.

This can happen only when you have a number of shards that is a factor of the number of nodes.

The current workaround is to always set a `seed` in `.shuffle()`
Dataset configuration
Author: lhoestq
State: open
Labels: name: generic discussion

Following discussions from https://github.com/huggingface/datasets/pull/5331

We could have something like `config.json` to define the configuration of a dataset.

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support a list for several configs with a 'config_name' field.

The alternative was to use YAML in the README.md.

I think it could also support a `dataset_type` field to specify which dataset builder class to use, and the other parameters would be the builder's parameters. Some parameters exist for all builders like `data_files` and `data_dir`, but some parameters are builder specific like `sep` for csv.

This format would be used in `push_to_hub` to be able to push multiple configs.

cc @huggingface/datasets 

EDIT: actually we're going for the YAML approach in README.md

Comments: Originally we also though about adding it to the YAML part of the README.md:

```yaml
builder_config:
  data_dir: data
  data_files:
  - split: train
    pattern: "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
```

Having it in the README.md could make it easier to modify it in the UI on HF, and for validation on commit

From internal discussions we agreed to go with the YAML approach, since it's the one that seems more appropriate to be modified by a human on the Hub or locally (while JSON e.g. for models are usually created programmatically).

Current format:
```yaml
builder_config:
  data_files:
  - split: train
    pattern: data/train-*
```
Document to compress data files before uploading
Author: albertvillanova
State: closed
Labels: name: documentation

In our docs to [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset), we tell users to upload directly their data files, like CSV, JSON, JSON-Lines, text,... However, these extensions are not tracked by Git LFS by default, as they are not in the `.giattributes` file. Therefore, if they are too large, Git will fail to commit/upload them.

I think for those file extensions (.csv, .json, .jsonl, .txt), we should better recommend to **compress** their data files (using ZIP for example) before uploading them to the Hub.
- Compressed files are tracked by Git LFS in our default `.gitattributes` file

What do you think?
CC: @stevhliu 

See related issue:
- https://huggingface.co/datasets/tcor0005/langchain-docs-400-chunksize/discussions/1

Comments: Great idea!

Should we also take this opportunity to include some audio/image file formats? Currently, it still reads very text heavy. Something like:

> We support many text, audio, and image data extensions such as `.zip`, `.rar`, `.mp3`, and `.jpg` among many others. For data extensions like `.csv`, `.json`, `.jsonl`, and `txt`, we recommend compressing them before uploading to the Hub. These file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of supported file extensions.

Hi @stevhliu, thanks for your suggestion.

I agree it is a good opportunity to mention that audio/image file formats are also supported.

Nit:
I would not mention .zip, .rar after "text, audio, and image data extensions". Those are "compression" extensions and not "text, audio, and image data extensions".

What about something similar to:
> We support many text, audio, and image data extensions such as `.csv`, `.mp3`, and `.jpg` among many others. For text data extensions like `.csv`, `.json`, `.jsonl`, and `.txt`, we recommend compressing them before uploading to the Hub (to `.zip` or `.gz` file extension for example). 
>
> Note that text file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of tracked file extensions by default.

Note that for compressions I have mentioned:
- gz, to compress individual files
- zip, to compress and archive multiple files; zip is preferred rather than tar because it supports streaming out of the box

Perfect, thanks for making the distinction between compression and data extensions!
Add information about patterns search order to the doc about structuring repo
Author: polinaeterna
State: closed
Labels: name: documentation

Following [this](https://github.com/huggingface/datasets/issues/5650) issue I think we should add a note about the order of patterns that is used to find splits, see [my comment](https://github.com/huggingface/datasets/issues/5650#issuecomment-1488412527). Also we should reference this page in pages about packaged loaders. 

I have a déjà vu that it had already been discussed as some point but I don't remember....

Comments: Good idea, I think I've seen this a couple of times before too on the forums. I can work on this :)

Closed in #5693