helmo/github-issues huggingface dataset:
train_test_split in arrow_dataset does not ensure to keep single classes in test set
State: closed | Author: Ulipenitz (NONE)

### Describe the bug

When I have a dataset with very few (e.g. 1) examples per class and I call the train_test_split function on it, sometimes the single class will be in the test set. thus will never be considered for training.

### Steps to reproduce the bug

```
import numpy as np
from datasets import Dataset

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.train_test_split(test_size=0.5)
    data_set["train"]

    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test) 
```

### Expected behavior

I expect to have every available class at least once in my training set.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 11.0.0
- Pandas version: 1.3.5


Comments: Hi! You can get this behavior by specifying `stratify_by_column="label"` in `train_test_split`.

This is the full example:
```python
import numpy as np
from datasets import Dataset, ClassLabel

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.cast_column("label", ClassLabel(num_classes=3))
    data_set = data_set.train_test_split(test_size=0.5, stratify_by_column="label")
    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test) 
```
Invalid Arrow data from JSONL
State: open | Author: lhoestq (MEMBER)
Labels: name: bug

This code fails:
```python
from datasets import Dataset

ds = Dataset.from_json(path_to_file)
ds.data.validate()
```
raises

```python
ArrowInvalid: Column 2: In chunk 1: Invalid: Struct child array #3 invalid: Invalid: Length spanned by list offsets (4064) larger than values array (length 4063)
```

This causes many issues for @TevenLeScao:

- `map` fails because it fails to rewrite invalid arrow arrays

    ```python
    ~/Desktop/hf/datasets/src/datasets/arrow_writer.py in write_examples_on_file(self)
        438             if all(isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) for row in self.current_examples):
        439                 arrays = [row[0][col] for row in self.current_examples]
    --> 440                 batch_examples[col] = array_concat(arrays)
        441             else:
        442                 batch_examples[col] = [

    ~/Desktop/hf/datasets/src/datasets/table.py in array_concat(arrays)
    1885 
    1886     if not _is_extension_type(array_type):
    -> 1887         return pa.concat_arrays(arrays)
    1888 
    1889     def _offsets_concat(offsets):

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays()

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

    ArrowIndexError: array slice would exceed array length
    ```


- `to_dict()` **segfaults** ⚠️

    ```python
    /Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/array/data.cc:99:  Check failed: (off) <= (length) Slice offset greater 
    than array length
    ```

To reproduce: unzip the archive and run the above code using `sanity_oscar_en.jsonl`
[sanity_oscar_en.jsonl.zip](https://github.com/huggingface/datasets/files/10734124/sanity_oscar_en.jsonl.zip)

PS: reading using pandas and converting to Arrow works though (note that the dataset lives in RAM in this case):
```python
ds = Dataset.from_pandas(pd.read_json(path_to_file, lines=True))
ds.data.validate()
```
[INVALID PR]
State: closed | Author: alvarobartt (MEMBER)

Hi to whoever is reading this! 🤗

## What's in this PR?

~~Basically, I've removed the 🤗`datasets` installation as `python -m pip install ".[quality]" in the `check_code_quality` job in `.github/workflows/ci.yaml`, as we don't need to install the whole package to run the CI, unless that's done on purpose e.g. to check that the Python package installation succeeds before running the tests over the matrix of os?~~

~~So I just wanted to check whether the time was reduced doing this (which I assume it will), plus whether this is something that can be improved, or just discarded in case you're also using that step to make sure that the package can be installed.~~

## What's missing?

~~I was just wondering whether you consider replacing `isort` and `flake8` with `ruff` (if possible), since it's way faster, more information at [`ruff`](https://github.com/charliermarsh/ruff). Before creating this PR the average time of the `check_code_quality` job was around 40s.~~

## Edit

Sorry for the inconvenience this may have caused, didn't realise that the config is defined in `setup.cfg` and `pyproject.toml`, so running those without installing the Python package leads to failure, my bad 😞 


Comments: _The documentation is not available anymore as the PR was closed or merged._
Checking that split name is correct happens only after the data is downloaded
State: open | Author: polinaeterna (CONTRIBUTOR)
Labels: name: bug

### Describe the bug

Verification of split names (=indexing data by split) happens after downloading the data. So when the split name is incorrect, users learn about that only after the data is fully downloaded, for large datasets it might take a lot of time.

### Steps to reproduce the bug

Load any dataset with random split name, for example:
```python
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_11_0", "en", split="blabla")
```
and the download will start smoothly, despite there is no split named "blabla".

### Expected behavior

Raise error when split name is incorrect. 

### Environment info

`datasets==2.9.1.dev0`
ClassLabel.cast_storage raises TypeError when called on an empty IntegerArray
State: closed | Author: marioga (CONTRIBUTOR)

### Describe the bug

`ClassLabel.cast_storage` raises `TypeError` when called on an empty `IntegerArray`.

### Steps to reproduce the bug

Minimal steps:
```python
import pyarrow as pa
from datasets import ClassLabel

ClassLabel(names=['foo', 'bar']).cast_storage(pa.array([], pa.int64()))
```

In practice, this bug arises in situations like the one below:
```python
from datasets import ClassLabel, Dataset, Features, Sequence

dataset = Dataset.from_dict({'labels': [[], []]}, features=Features({'labels': Sequence(ClassLabel(names=['foo', 'bar']))}))

# this raises TypeError
dataset.map(batched=True, batch_size=1)
```

### Expected behavior

`ClassLabel.cast_storage` should return an empty Int64Array.

### Environment info

- `datasets` version: 2.9.1.dev0
- Platform: Linux-4.15.0-1032-aws-x86_64-with-glibc2.27
- Python version: 3.10.6
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Reload features from Parquet metadata
State: closed | Author: MFreidank (CONTRIBUTOR)

Resolves #5482. 

Attaches feature metadata to parquet files serialised using `Dataset.to_parquet`. 

This allows retrieving data with "rich" feature types (e.g., `datasets.features.image.Image` or `datasets.features.audio.Audio`) from parquet files without cumbersome casting (for an example, see #5482).

@lhoestq It seems that it is sufficient to attach metadata to the schema prior to serialising and features are loaded back with correct types afterwards automatically. 

I used the following script to test the implementation:

```python
from pathlib import Path

import datasets

dataset_name = "Maysee/tiny-imagenet"
ds = datasets.load_dataset(dataset_name, split=datasets.Split.TRAIN)
output_directory_path = Path(__file__).parent.joinpath("example_test_outputs", dataset_name.replace("/", "_"))
output_directory_path.mkdir(exist_ok=True, parents=True)
output_filepath = output_directory_path.joinpath("ds.parquet")
ds.to_parquet(str(output_filepath))

reloaded_ds = datasets.load_dataset(str(output_directory_path), split=datasets.Split.TRAIN)

assert ds.features == reloaded_ds.features
```

Prior to the change in this PR this script raises an `AssertionError` and the `Image` features lose their type after serialisation. After the change in this PR, the assertion does not raise an error and manual inspection of the features shows type `Image` for the respective columns of `reloaded_ds `.

Some open questions:
* How/where can I best add new unit tests for this implementation?
* What dataset would I best use in the tests? I chose `Maysee/tiny-imagenet` mainly because it is small and contains an ?Image` feature that can be used to test, but I'd be happy for suggestions on a suitable data source to use.
* Currently I'm calling `datasets.arrow_writer.ArrowWriter._build_metadata` as I need the same logic. However, I'm not happy with the coupling between `datasets.io.parquet` and `datasets.arrow_writer` it leaves me with. Suggest to factor this common logic out into a helper function and reuse it from both of these. Do you agree and if yes, could you please guide me where I would best place this function?

Many thanks in advance and kind regards,
MFreidank


  

Comments: Thanks a lot for your help @lhoestq. I've simplified what turned out to be a simple fix and added the unit test.

Does this look ready to be merged or is there anything I'm still missing?

Cool ! I think you just need to remove the unused import in `io/parquet.py`
```
src/datasets/io/parquet.py:4:1: F401 'pyarrow as pa' imported but unused
```
and we're good to merge :)

_The documentation is not available anymore as the PR was closed or merged._

> Cool ! I think you just need to remove the unused import in `io/parquet.py`
> 
> ```
> src/datasets/io/parquet.py:4:1: F401 'pyarrow as pa' imported but unused
> ```
> 
> and we're good to merge :)

Done! Thanks a lot, this was fun :)
Improve inconsistency of `Dataset.map` interface for `load_from_cache_file`
State: closed | Author: HallerPatrick (CONTRIBUTOR)
Labels: name: enhancement

### Feature request


1. Replace the `load_from_cache_file` default value to `True`. 
2. Remove or alter checks from `is_caching_enabled` logic.

### Motivation

I stumbled over an inconsistency in the `Dataset.map` interface. The documentation (and source) states for the parameter `load_from_cache_file`:
```
load_from_cache_file (`bool`, defaults to `True` if caching is enabled):
    If a cache file storing the current computation from `function`
    can be identified, use it instead of recomputing.
```

1. `load_from_cache_file` default value is `None`, while being annotated as `bool`
2. It is inconsistent with other method signatures like `filter`, that have the default value `True`
3. The logic is inconsistent, as the `map` method checks if caching is enabled through `is_caching_enabled`. This logic is not used for other similar methods.

### Your contribution

I am not fully aware of the logic behind caching checks. If this is just a inconsistency that historically grew, I would suggest to remove the `is_caching_enabled` logic as the "default" logic. Maybe someone can give insights, if environment variables have a higher priority than local variables or vice versa. 

If this is clarified, I could adjust the source  according to the "Feature request" section of this issue.

Comments: Hi, thanks for noticing this! We can't just remove the cache control as this allows us to control where the arrow files generated by the ops are written (cached on disk if enabled or a temporary directory if disabled). The right way to address this inconsistency would be by having `load_from_cache_file=None` by default everywhere.

Hi! Yes, this seems more plausible. I can implement that. One last thing is the type annotation `load_from_cache_file: bool = None`. Which I then would change to `load_from_cache_file: Optional[bool] = None`.

PR #5515 

Yes, `Optional[bool]` is the correct type annotation and thanks for the PR.
Some functions use a param named `type` shouldn't that be avoided since it's a Python reserved name?
State: closed | Author: alvarobartt (MEMBER)

Hi @mariosasko, @lhoestq, or whoever reads this! :)

After going through `ArrowDataset.set_format` I found out that the `type` param is actually named `type` which is a Python reserved name as you may already know, shouldn't that be renamed to `format_type` before the 3.0.0 is released?

Just wanted to get your input, and if applicable, tackle this issue myself! Thanks 🤗

Comments: Hi! Let's not do this - renaming it would be a breaking change, and going through the deprecation cycle is only worth it if it improves user experience.

Hi @mariosasko, ok it makes sense. Anyway, don't you think it's worth it at some point to start a deprecation cycle e.g. `fs` in `load_from_disk`? It doesn't affect user experience but it's for sure a bad practice IMO, but's up to you 😄 Feel free to close this issue otherwise!

I don't think deprecating a param name in this particular instance is worth the hassle, so I'm closing the issue 🙂.

Sure, makes sense @mariosasko thanks!
Milvus integration for search
State: open | Author: filip-halt (NONE)

Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5510). All of your documentation changes will be reflected on that endpoint.

To the maintainer, sorry about the repeated run requests for formatting. Missed the `make style` outlined in contributing guidelines. 

Anything I can do to get the workflow to run? @lhoestq 

cc @mariosasko 

> Anything I can do to get the workflow to run?

You can merge `main` into your branch to fix code formatting (we switched from isort+flake8 to ruff this week), and then run `make style`

I believe that should be good. @mariosasko
Add a static `__all__` to `__init__.py` for typecheckers
State: open | Author: LoicGrobol (NONE)

This adds a static `__all__` field to `__init__.py`, allowing typecheckers to know which symbols are accessible from `datasets` at runtime. In particular [Pyright](https://github.com/microsoft/pylance-release/issues/2328#issuecomment-1029381258) seems to rely on this. At this point I have added all (modulo oversight) the symbols mentioned in the Reference part of [the docs](https://huggingface.co/docs/datasets), but that could be adjusted. As a side effect, only these symbols will be imported by `from datasets import *`, which may or may not be a good thing (and if it isn't, that's easy to fix).

Another option would be to add a pyi stub, but I think `__all__` should be the most pythonic solution.

This should fix #3841.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5509). All of your documentation changes will be reflected on that endpoint.

Hi! I've commented on the original issue to provide some context. Feel free to share your opinion there.
Saving a dataset after setting format to torch doesn't work, but only if filtering
State: closed | Author: joebhakim (NONE)

### Describe the bug

Saving a dataset after setting format to torch doesn't work, but only if filtering

### Steps to reproduce the bug

```
a = Dataset.from_dict({"b": [1, 2]})
a.set_format('torch')
a.save_to_disk("test_save") # saves successfully
a.filter(None).save_to_disk("test_save_filter") # does not
>> [...] TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types [<class 'torch.Tensor'>]. When using `batched=True`, make sure provided `function` returns a `dict` of types like `(<class 'list'>, <class 'numpy.ndarray'>)`.

# note: skipping the format change to torch lets this work.



### Expected behavior

Saving to work

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-6.1.9-arch1-1-x86_64-with-glibc2.36
- Python version: 3.10.9
- PyArrow version: 9.0.0
- Pandas version: 1.4.4

Comments: Hey, I'm a research engineer working on language modelling wanting to contribute to open source. I was wondering if I could give it a shot?

Hi! This issue was fixed in https://github.com/huggingface/datasets/pull/4972, so please install `datasets>=2.5.0` to avoid it.
Optimise behaviour in respect to indices mapping
State: open | Author: mariosasko (COLLABORATOR)
Labels: name: enhancement

_Originally [posted](https://huggingface.slack.com/archives/C02V51Q3800/p1675443873878489?thread_ts=1675418893.373479&cid=C02V51Q3800) on Slack_

Considering all this, perhaps for Datasets 3.0, we can do the following:
* [ ] have `continuous=True` by default in `.shard`  (requested in the survey and makes more sense for us since it doesn't create an indices mapping)
* [x] allow calling `save_to_disk` on "unflattened" datasets
* [ ] remove "hidden" expensive calls in `save_to_disk`, `unique`, `concatenate_datasets`, etc. For instance, instead of silently calling `flatten_indices`  where it's needed, it's probably better to be explicit (considering how expensive these ops can be) and raise an error instead
PyTorch BatchSampler still loads from Dataset one-by-one
State: closed | Author: davidgilbertson (NONE)

### Describe the bug

In [the docs here](https://huggingface.co/docs/datasets/use_with_pytorch#use-a-batchsampler), it mentions the issue of the Dataset being read one-by-one, then states that using a BatchSampler resolves the issue.

I'm not sure if this is a mistake in the docs or the code, but it seems that the only way for a Dataset to be passed a list of indexes  by PyTorch (instead of one index at a time) is to define a `__getitems__` method (note the plural) on the Dataset object, and since the HF Dataset doesn't have this, PyTorch executes [this line of code](https://github.com/pytorch/pytorch/blob/master/torch/utils/data/_utils/fetch.py#L58), reverting to fetching one-by-one.

### Steps to reproduce the bug

You can put a breakpoint in `Dataset.__getitem__()` or just print the args from there and see that it's called multiple times for a single `next(iter(dataloader))`, even when using the code from the docs:
```py
from torch.utils.data.sampler import BatchSampler, RandomSampler
batch_sampler = BatchSampler(RandomSampler(ds), batch_size=32, drop_last=False)
dataloader = DataLoader(ds, batch_sampler=batch_sampler)
```



### Expected behavior

The expected behaviour would be for it to fetch batches from the dataset, rather than one-by-one.

To demonstrate that there is room for improvement: once I have a HF dataset `ds`, if I just add this line:
```py
ds.__getitems__ = ds.__getitem__
```
...then the time taken to loop over the dataset improves considerably (for wikitext-103, from one minute to 13 seconds with batch size 32). Probably not a big deal in the grand scheme of things, but seems like an easy win.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Comments: This change seems to come from a few months ago in the PyTorch side. That's good news and it means we may not need to pass a batch_sampler as soon as we add `Dataset.__getitems__` to get the optimal speed :)

Thanks for reporting ! Would you like to open a PR to add `__getitems__` and remove this outdated documentation ?

Yeah I figured this was the sort of thing that probably once worked. I can confirm that you no longer need the batch sampler, just `batch_size=n` in the `DataLoader`.

I'll pass on the PR, I'm flat out right now, sorry.
`load_dataset` has ~4 seconds of overhead for cached data
State: open | Author: davidgilbertson (NONE)
Labels: name: enhancement

### Feature request

When loading a dataset that has been cached locally, the `load_dataset` function takes a lot longer than it should take to fetch the dataset from disk (or memory).
This is particularly noticeable for smaller datasets. For example, wikitext-2, comparing `load_data` (once cached) and `load_from_disk`, the `load_dataset` method takes 40 times longer.

⏱    4.84s ⮜ load_dataset
⏱    119ms ⮜ load_from_disk

### Motivation

I assume this is doing something like checking for a newer version.

If so, that's an age old problem: do you make the user wait _every single time they load from cache_ or do you do something like load from cache always, _then_ check for a newer version and alert if they have stale data. The decision usually revolves around what percentage of the time the data will have been updated, and how dangerous old data is.

For most datasets it's extremely unlikely that there will be a newer version on any given run, so 99% of the time this is just wasted time.

Maybe you don't want to make that decision for all users, but at least having the _option_ to not wait for checks would be an improvement.

### Your contribution

.

Comments: Hi ! To skip the verification step that checks if newer data exist, you can enable offline mode with `HF_DATASETS_OFFLINE=1`.

Although I agree this step should be much faster for datasets hosted on the HF Hub - we could just compare the commit hash from the local data and the remote git repository. We're not been leveraging the git commit hashes, since the library was built before we even had git repositories for each dataset on HF.

Thanks @lhoestq, for memory when I recorded those times I had `HF_DATASETS_OFFLINE` set.