Minimartzz/github-issues huggingface dataset:
Fallback JSON Dataset loading does not load all values when features specified manually
State: closed

### Describe the bug

The fallback JSON dataset loader located here:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L130-L153

does not load the values of features correctly when features are specified manually and not all features have a value in the first entry of the dataset. I'm pretty sure this is not supposed to be expected bahavior?

To fix this you'd have to change this line:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L140

To pass a schema to pyarrow which has the same structure as the features argument passed to the load_dataset() method.

### Steps to reproduce the bug

Consider a dataset JSON like this:

```
[
{
	"instruction": "Do stuff",
	"output": "Answer stuff"
},
{
	"instruction": "Do stuff2",
	"input": "Additional Input2",
	"output": "Answer stuff2"
}
]
```

Using this code to load the dataset:

```
from datasets import load_dataset, Features, Value

features = {
    "instruction": Value("string"),
    "input": Value("string"),
    "output": Value("string")
}
features = Features(features)

ds = load_dataset("json", data_files="./ds.json", features=features)

for row in ds["train"]:
    print(row)
```

we get a dataset that looks like this:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | None               | "Answer Stuff2" |

### Expected behavior

The input column should contain values other than None for dataset entries that have the "input" attribute set:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | "Additional Input2" | "Answer Stuff2" |

### Environment info

Python 3.10.10
Datasets 2.11.0
Windows 10

Comments:
Thanks for reporting, @myluki2000.

I am working on a fix.
How to limit the number of examples in dataset, for testing?
State: closed

### Describe the bug

I am using this command:
`data = load_dataset("json", data_files=data_path)`
However, I want to add a parameter, to limit the number of loaded examples to be 10, for development purposes, but can't find this simple parameter.

### Steps to reproduce the bug

In the description.

### Expected behavior

To be able to limit the number of examples

### Environment info

Nothing special

Comments:
Hi! You can use the `nrows` parameter for this:
```python
data = load_dataset("json", data_files=data_path, nrows=10)
```

@mariosasko I get:

`TypeError: __init__() got an unexpected keyword argument 'nrows'`

I misread the format in which the dataset is stored -  the `nrows` parameter works for CSV, but not JSON.

This means the only option is first to create a DataFrame and then convert it to a Dataset object:
```python
import pandas as pd
from datasets import Dataset

df = pd.read_json(data_path, lines=True, nrows=10)
ds = Dataset.from_pandas(df)
```
Calling datasets.load_dataset("text" ...) results in a wrong split.
State: open

### Describe the bug

When creating a text dataset, the training split should have the bulk of the examples by default. Currently, testing does.

### Steps to reproduce the bug

I have a folder with 18K text files in it. Each text file essentially consists in a document or article scraped from online. Calling the following codeL
```
folder_path = "/home/cyril/Downloads/llama_dataset"
data = datasets.load_dataset("text", data_dir=folder_path)
data.save_to_disk("/home/cyril/Downloads/data.hf")

data = datasets.load_from_disk("/home/cyril/Downloads/data.hf")
print(data)
```

Results in the following split:
```
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 2114
    })
    test: Dataset({
        features: ['text'],
        num_rows: 200882
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 152
    })
})
```

It seems to me like the train/test/validation splits are in the wrong order since test split >>>> train_split

### Expected behavior

Train split should have the bulk of the training examples.

### Environment info

datasets 2.11.0, python 3.10.6

Comments:
Handle empty audio
State: closed

Some audio paths exist, but they are empty, and an error will be reported when reading the audio path.How to use the filter function to avoid the empty audio path?
when a audio is empty, when do resample , it will break:
`array, sampling_rate = sf.read(f) array = librosa.resample(array, orig_sr=sampling_rate, target_sr=self.sampling_rate)`

Comments:
Hi! Can you share one of the problematic audio files with us?

I tried to reproduce the error with the following code: 
```python
import soundfile as sf
import numpy as np
from datasets import Audio

sf.write("empty.wav",  np.array([]), 16000)
Audio(sampling_rate=24000).decode_example({"path": "empty.wav", "bytes": None})
```
But without success.

Also, what version of `librosa` is installed in your env? (You can get this info with `python -c "import librosa; print(librosa.__version__)`)



I'm closing this issue as the reproducer hasn't been provided.
Return Numpy Array (fixed length) Mode, in __get_item__, Instead of List
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

There are old known issues, but they can be easily forgettable problems in multiprocessing with pytorch-dataloader:

Too high usage of RAM or shared-memory in pytorch when we set num workers > 1 and returning type of dataset or dataloader is "List" or "Dict".
https://github.com/pytorch/pytorch/issues/13246 

With huggingface datasets, unfortunately, the default return type is the list, so the problem is raised too often if we do not set anything for the issue.


However, this issue can be released when the returning output is fixed in length.
Therefore, I request the mode, returning outputs with fixed length (e.g. numpy array) rather than list.

The design would be good when we load datasets as
```python
load_dataset(..., with_return_as_fixed_tensor=True)
```

### Motivation

The general solution for this issue is already in the comments:  https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 

: Numpy or Pandas seems not to have problems, while both have the string type.
(I'm not sure that the sequence of huggingface datasets can solve this problem as well)


### Your contribution

 I'll read it ! thanks

Comments:
Hi! 

You can use [`.set_format("np")`](https://huggingface.co/docs/datasets/process#format) to get NumPy arrays (or Pytorch tensors with `.set_format("torch")`) in `__getitem__`.

Also, have you been able to reproduce the linked PyTorch issue with a HF dataset?
Dataset sizes are in MiB instead of MB in dataset cards
State: closed
Labels: description: Something isn't working
name: bug

description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations

Comments:
Example of bulk edit: https://huggingface.co/datasets/aeslc/discussions/5

looks great! 

Do you encode the fact that you've already converted a dataset? (to not convert it twice) or do you base yourself on the info contained in `dataset_info`

I am only looping trough the dataset cards, assuming that all of them were created with MiB.

I agree we should only run the bulk edit once for all canonical datasets: I'm using a for-loop over canonical datasets.

yes, worst case, we have this in structured data:

<img width="337" alt="image" src="https://user-images.githubusercontent.com/326577/230037051-06caddcb-08c8-4953-a710-f3d122917db3.png">


I have just included as well the conversion from MB to GB if necessary. See: 
- https://huggingface.co/datasets/bookcorpus/discussions/2/files
- https://huggingface.co/datasets/asnq/discussions/2/files

Nice. Is it another loop? Because in https://huggingface.co/datasets/amazon_us_reviews/discussions/2/files we have `32377.29 MB` for example

First, I tested some batches to check the changes made. Then I incorporated the MB to GB conversion. Now I'm running the rest.

The bulk edit parsed 751 canonical datasets and updated 166.

Thanks a lot!

The sizes now match as expected!

<img width="1446" alt="Capture d’écran 2023-04-05 à 16 10 15" src="https://user-images.githubusercontent.com/1676121/230107044-ac2a76ea-a4fe-4e81-a925-f464b85f5edd.png">


I made another bulk edit of ancient canonical datasets that were moved to community organization. I have parsed 11 datasets and opened a PR on 3 of them:
- [x] "allenai/scicite": https://huggingface.co/datasets/allenai/scicite/discussions/3
- [x] "allenai/scifact": https://huggingface.co/datasets/allenai/scifact/discussions/2
- [x] "dair-ai/emotion": https://huggingface.co/datasets/dair-ai/emotion/discussions/6

should we force merge the PR and close this issue?

I merged the PRs for "scicite" and "scifact".
Getting next item from IterableDataset took forever.
State: closed

### Describe the bug

I have a large dataset, about 500GB. The format of the dataset is parquet. 

I then load the dataset and try to get the first item
```python
def get_one_item():
    dataset = load_dataset("path/to/datafiles", split="train", cache_dir=".", streaming=True)
    dataset = dataset.filter(lambda example: example['text'].startswith('Ar'))
    print(next(iter(dataset)))
```

However, this function never finish. I waited ~10mins, the function was still running so I killed the process. I'm now using `line_profiler` to profile how long it would take to return one item. I'll be patient and wait for as long as it needs. 

I suspect the filter operation is the reason why it took so long. Can I get some possible reasons behind this?

### Steps to reproduce the bug

Unfortunately without my data files, there is no way to reproduce this bug.

### Expected behavior

With `IteralbeDataset`, I expect the first item to be returned instantly.

### Environment info

- datasets version: 2.11.0 
- python: 3.7.12

Comments:
Hi! It can take some time to iterate over Parquet files as big as yours, convert the samples to Python, and find the first one that matches a filter predicate before yielding it...

Thanks @mariosasko, I figured it was the filter operation. I'm closing this issue because it is not a bug, it is the expected beheaviour.
5537 speedup load
State: open

I reimplemented fsspec.spec.glob() in `hffilesystem.py` as `_glob`,  used it in `_resolve_single_pattern_in_dataset_repository` only, and saw a 20% speedup in times to load the config, on average. 

That's not much when usually this step takes only 2-3 seconds for most datasets, but in this particular case,  `bigcode/the-stack-dedup` , the loading time to get the config (not download the entire 6tb dataset, of course), went from ~170 secs to ~20 secs. 

What makes this work is this code in `_glob`:

```  
      if self.dir_cache is not None:
            allpaths = self.dir_cache
        else:
            allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
```

I also had to `import glob.has_magic( )` for `_glob()` (confusing, I know).

I hope there is no issue with copying most of the code from `fsspec.spec.glob`, as it is a BSD 3-Clause License, 
and I left a comment about this in the docstring of` _glob()`, that we may want to delete.

As mentioned, I evaluated the speedup across a random selection of about 1000 datasets (not all 27k+), and verified that old_config.eq(new_method_config) with the build in method, but deleted this test and related code changes on the subsequent commit. It's in the commit history if anyone wants to see it. (Note this does not include the outlier of  `bigcode/the-stack-dedup`

|   |   old_time     |    new _time |     diff   |  pct_diff |
| -- | -- | -- | -- | -- |
| mean  |    3.340 |    2.642 |    0.698 |   18.404 |
| min   |    2.024   |  1.976   |  -0.840  | -37.634 |
| max   |  66.582  |   41.517 |   30.927  |  85.538 |

Comments:
Awesome ! cc @mariosasko :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5704). All of your documentation changes will be reflected on that endpoint.

Hi, thanks for working on this!

Your solution only works if the `root` is `""`, e.g., this would yield an incorrect result:
```python
dset = load_dataset("user/hf-dataset-repo", data_dir="path/to/data_dir")
```

Also, the `HfFileSystem` implementation in `datasets` will be replaced with the more powerful [one](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py) from `huggingface_hub` soon (I plan to open a PR that makes `find` much faster in the coming days). 

So I don't think we want to merge this PR in the current state, but thanks again for the effort.

 (I'll comment on the original issue to propose a cleaner solution)

Ooof. Sorry, I should have checked that more thoroughly then! I would say we could just add that check and only use my approach if the root is "", which would still be faster in many cases, but it sounds like you have a better solution on the way. Thanks for the feedback Mario.
[WIP][Test, Please ignore] Investigate performance impact of using multiprocessing only
State: closed


Comments:
`multiprocess` uses `dill` instead of `pickle` for pickling shared objects and, as such, can pickle more types than `multiprocessing`. And I don't think this is something we want to change :).

That makes sense to me, and I don't think you should merge this change. I was only curious about the performance impact. I saw the benchmarks that was produced in other PRs, and wanted to get a better understanding of it. I created this PR to see if it got automatically added here.

Is there a way I can generate those benchmarks myself?

You can find some speed comparisons between dill and pickle on SO if you google "dill vs pickle speed".

And for the benchmarks, you can generate them locally with DVC running this code from the repo root: https://github.com/huggingface/datasets/blob/0803a006db1c395ac715662cc6079651f77c11ea/.github/workflows/benchmarks.yaml#L23-L47.

Thanks for the help @mariosasko!
Add Qdrant as another search index
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

I'd suggest adding Qdrant (https://qdrant.tech) as another search index available, so users can directly build an index from a dataset. Currently, FAISS and ElasticSearch are only supported: https://huggingface.co/docs/datasets/faiss_es 

### Motivation

ElasticSearch is a keyword-based search system, while FAISS is a vector search library. Vector database, such as Qdrant, is a different tool based on similarity (like FAISS) but is not limited to a single machine. It makes the vector database well-suited for bigger datasets and collaboration if several people want to access a particular dataset.

### Your contribution

I can provide a PR implementing that functionality on my own.

Comments:
@mariosasko I'd appreciate your feedback on this.
Shuffle a sharded iterable dataset without seed can lead to duplicate data
State: closed
Labels: description: Something isn't working
name: bug

As reported in https://github.com/huggingface/datasets/issues/5360

If `seed=None` in `.shuffle()`, shuffled datasets don't use the same shuffling seed across nodes.

Because of that, the lists of shards is not shuffled the same way across nodes, and therefore some shards may be assigned to multiple nodes instead of exactly one.

This can happen only when you have a number of shards that is a factor of the number of nodes.

The current workaround is to always set a `seed` in `.shuffle()`

Comments:
Dataset configuration
State: open
Labels: description: Generic discussion on the library
name: generic discussion

Following discussions from https://github.com/huggingface/datasets/pull/5331

We could have something like `config.json` to define the configuration of a dataset.

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support a list for several configs with a 'config_name' field.

The alternative was to use YAML in the README.md.

I think it could also support a `dataset_type` field to specify which dataset builder class to use, and the other parameters would be the builder's parameters. Some parameters exist for all builders like `data_files` and `data_dir`, but some parameters are builder specific like `sep` for csv.

This format would be used in `push_to_hub` to be able to push multiple configs.

cc @huggingface/datasets 

EDIT: actually we're going for the YAML approach in README.md

Comments:
Originally we also though about adding it to the YAML part of the README.md:

```yaml
builder_config:
  data_dir: data
  data_files:
  - split: train
    pattern: "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
```

Having it in the README.md could make it easier to modify it in the UI on HF, and for validation on commit

From internal discussions we agreed to go with the YAML approach, since it's the one that seems more appropriate to be modified by a human on the Hub or locally (while JSON e.g. for models are usually created programmatically).

Current format:
```yaml
builder_config:
  data_files:
  - split: train
    pattern: data/train-*
```
Document to compress data files before uploading
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

In our docs to [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset), we tell users to upload directly their data files, like CSV, JSON, JSON-Lines, text,... However, these extensions are not tracked by Git LFS by default, as they are not in the `.giattributes` file. Therefore, if they are too large, Git will fail to commit/upload them.

I think for those file extensions (.csv, .json, .jsonl, .txt), we should better recommend to **compress** their data files (using ZIP for example) before uploading them to the Hub.
- Compressed files are tracked by Git LFS in our default `.gitattributes` file

What do you think?
CC: @stevhliu 

See related issue:
- https://huggingface.co/datasets/tcor0005/langchain-docs-400-chunksize/discussions/1

Comments:
Great idea!

Should we also take this opportunity to include some audio/image file formats? Currently, it still reads very text heavy. Something like:

> We support many text, audio, and image data extensions such as `.zip`, `.rar`, `.mp3`, and `.jpg` among many others. For data extensions like `.csv`, `.json`, `.jsonl`, and `txt`, we recommend compressing them before uploading to the Hub. These file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of supported file extensions.

Hi @stevhliu, thanks for your suggestion.

I agree it is a good opportunity to mention that audio/image file formats are also supported.

Nit:
I would not mention .zip, .rar after "text, audio, and image data extensions". Those are "compression" extensions and not "text, audio, and image data extensions".

What about something similar to:
> We support many text, audio, and image data extensions such as `.csv`, `.mp3`, and `.jpg` among many others. For text data extensions like `.csv`, `.json`, `.jsonl`, and `.txt`, we recommend compressing them before uploading to the Hub (to `.zip` or `.gz` file extension for example). 
>
> Note that text file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of tracked file extensions by default.

Note that for compressions I have mentioned:
- gz, to compress individual files
- zip, to compress and archive multiple files; zip is preferred rather than tar because it supports streaming out of the box

Perfect, thanks for making the distinction between compression and data extensions!
Add information about patterns search order to the doc about structuring repo
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

Following [this](https://github.com/huggingface/datasets/issues/5650) issue I think we should add a note about the order of patterns that is used to find splits, see [my comment](https://github.com/huggingface/datasets/issues/5650#issuecomment-1488412527). Also we should reference this page in pages about packaged loaders. 

I have a déjà vu that it had already been discussed as some point but I don't remember....

Comments:
Good idea, I think I've seen this a couple of times before too on the forums. I can work on this :)

Closed in #5693
Allow load_dataset to take a working dir for intermediate data
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

As a user, I can set a working dir for intermediate data creation. The processed files will be moved to the cache dir, like
```
load_dataset(…, working_dir=”/temp/dir”, cache_dir=”/cloud_dir”).
```

### Motivation

This will help the use case for using datasets with cloud storage as cache. It will help boost the performance.

### Your contribution

I can provide a PR to fix this if the proposal seems reasonable.

Comments:
Hi ! AFAIK a dataset must be present on a local disk to be able to efficiently memory map the datasets Arrow files. What makes you think that it is possible to load from a cloud storage and have good performance ?

Anyway it's already possible to download_and_prepare a dataset as Arrow files in a cloud storage with:
```python
builder = load_dataset_builder(..., cache_dir="/temp/dir")
builder.download_and_prepare("/cloud_dir")
```

but then 
```python
ds = builder.as_dataset()
```
would fail if "/cloud_dir" is not a local directory.

In my use case, I am trying to mount the S3 bucket as local system with S3FS-FUSE / [goofys](https://github.com/kahing/goofys). I want to use S3 to save the download data and save checkpoint for training for persistent. Setting the s3 location as cache directory is not fast enough. That is why I want to set a work directory for temp data for memory map and only save the final result to s3 cache. 

You can try setting `HF_DATASETS_DOWNLOADED_DATASETS_PATH` and `HF_DATASETS_EXTRACTED_DATASETS_PATH` to S3, and `HF_DATASETS_CACHE` to your local disk.

This way all your downloaded and extracted data are on your mounted S3, but the datasets Arrow files are on your local disk

If we hope to also persist the Arrow files on the mounted S3 but work with the efficiency of local disk, is there any recommended way to do this, other than copying the Arrow files from local disk to S3?