Minimartzz/github-issues huggingface dataset:
Directly reading parquet files in a s3 bucket from the load_dataset method
State: open
Labels: description: This issue or pull request already exists
name: duplicate

description: New feature or request
name: enhancement

### Feature request

Right now, we have to read the get the parquet file to the local storage. So having ability to read given the bucket directly address would be benificial 

### Motivation

In a production set up, this feature can help us a lot. So we do not need move training datafiles in between storage.  

### Your contribution

I am willing to help if there's anyway. 

Comments:
Hi ! I think is in the scope of this other issue: to https://github.com/huggingface/datasets/issues/5281
Apply flake8-comprehensions to codebase
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

Apply ruff flake8 comprehension checks to codebase. 

### Motivation

This should strictly improve the performance / readability of the codebase by removing unnecessary iteration, function calls, etc. This should generate better Python bytecode which should strictly improve performance.

I already applied this fixes to PyTorch and Sympy with little issue and have opened PRs to diffusers and transformers todo this as well.

### Your contribution

Making a PR.

Comments:
Downloaded datasets do not cache at $HF_HOME
State: closed

### Describe the bug

In the huggingface course (https://huggingface.co/course/chapter3/2?fw=pt) it said that if we set HF_HOME, downloaded datasets would be cached at specified address but it does not. downloaded models from checkpoint names are downloaded and cached at HF_HOME but this is not the case for datasets, they are still cached at ~/.cache/huggingface/datasets.

### Steps to reproduce the bug

Run the following code
```
from datasets import load_dataset
raw_datasets = load_dataset("glue", "mrpc")
raw_datasets
```

it downloads and store dataset at ~/.cache/huggingface/datasets

### Expected behavior

to cache dataset at HF_HOME.

### Environment info

python 3.10.6
Kubuntu 22.04
HF_HOME located on a separate partition

Comments:
Hi ! Can you make sure you set `HF_HOME` before importing `datasets` ?

Then you can print
```python
print(datasets.config.HF_CACHE_HOME)
print(datasets.config.HF_DATASETS_CACHE)
```
Added return methods for URL-references to the pushed dataset
State: open

Hi,

I was missing the ability to easily open the pushed dataset and it seemed like a quick fix.
Maybe we also want to log this info somewhere, but let me know if I need to add that too.

Cheers,
David

Comments:
Hi ! Maybe we'd need to align with `transformers` and other libraries that implement `push_to_hub` to agree on what it should return.

e.g. in `transformers` the typing says it returns a string, but in practice it returns a `CommitInfo`.

Therefore I'd not add an output to `push_to_hub` here unless we had a chance to discuss more broadly.

Anyway in my opinion it should no just return the URL of the repository, but ideally the URL at the revision where the data were pushed

Perhaps a mixin or something similar could be defined on the `hfh` side to ensure the `push_to_hub` API is aligned across our projects. 

PS: this would also mean that the PRs such as https://github.com/huggingface/datasets/pull/5528 would no longer be our responsibility

cc @Wauplin 

I agree, with universability and the idea is more about returning at least something that references where to find the uploaded file/model or otherwise. 

Ideally, the referenced PR would work.

imo this would be a good use case to just use `huggingface_hub` and align to what we do there :)

@mariosasko, can you give me some pointers to where I might help implementing this for the `huggingface-hub`?

> @mariosasko: Perhaps a mixin or something similar could be defined on the hfh side to ensure the push_to_hub API is aligned across our projects.

> @julien-c: imo this would be a good use case to just use huggingface_hub and align to what we do there :)

I (finally) opened a PR to harmonize return types: https://github.com/huggingface/huggingface_hub/pull/1921. It should hopefully be shipped in next release later this week (:crossed_fingers:).
the pile datasets url seems to change back
State: closed

### Describe the bug

in #3627, the host url of the pile dataset became `https://mystic.the-eye.eu`. Now the new url is broken, but `https://the-eye.eu` seems to work again.

### Steps to reproduce the bug

```python3
from datasets import load_dataset
dataset = load_dataset("bookcorpusopen")
```
shows
```python3
ConnectionError: Couldn't reach https://mystic.the-eye.eu/public/AI/pile_preliminary_components/books1.tar.gz (ProxyError(MaxRetryError("HTTPSConnectionPool(host='mystic.the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_pr
eliminary_components/books1.tar.gz (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 504 Gateway Timeout')))")))
```

### Expected behavior

Downloading as normal.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- PyArrow version: 6.0.1
- Pandas version: 1.5.3

Comments:
Thanks for reporting, @wjfwzzc.

I am transferring this issue to the corresponding dataset on the Hub: https://huggingface.co/datasets/bookcorpusopen/discussions/1

Thank you. All fixes are done:
- [x] https://huggingface.co/datasets/bookcorpusopen/discussions/2
- [x] https://huggingface.co/datasets/the_pile/discussions/1
- [x] https://huggingface.co/datasets/the_pile_books3/discussions/1
- [x] https://huggingface.co/datasets/the_pile_openwebtext2/discussions/2
- [x] https://huggingface.co/datasets/the_pile_stack_exchange/discussions/2
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number
State: closed
Labels: description: Good for newcomers
name: good first issue

### Describe the bug

When dataset contains a 0-dim tensor, formatting.py raises a following error and fails.
```bash
Traceback (most recent call last):
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 501, in format_row
    return _unnest(formatted_batch)
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 137, in _unnest
    return {key: array[0] for key, array in py_dict.items()}
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 137, in <dictcomp>
    return {key: array[0] for key, array in py_dict.items()}
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number
```

### Steps to reproduce the bug

Load whichever dataset and add transform method to add 0-dim tensor. Or create/find a dataset containing 0-dim tensor. E.g.

```python
from datasets import load_dataset
import torch

dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
def t(batch):
    return {"test": torch.tensor(1)}
    
dataset.set_transform(t)
d_0 = dataset[0]
```

### Expected behavior

Extractor will correctly get a row from the dataset, even if it contains 0-dim tensor.

### Environment info

`datasets==2.8.0`, but it looks like it is also applicable to main branch version (as of 16th February)

Comments:
Hi! The `set_transform` does not apply a custom formatting transform on a single example but the entire batch, so the fixed version of your transform would look as follows:
```python
from datasets import load_dataset
import torch

dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
def t(batch):
    return {"test": torch.tensor([1] * len(batch[next(iter(batch))]))}
    
dataset.set_transform(t)
d_0 = dataset[0]
```

Still, the formatter's error message should mention that a dict of **sequences** is expected as the returned value (not just a dict) to make debugging easier.

I can take this

Fixed in #5553 

> Hi! The `set_transform` does not apply a custom formatting transform on a single example but the entire batch, so the fixed version of your transform would look as follows:
> 
> ```python
> from datasets import load_dataset
> import torch
> 
> dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
> def t(batch):
>     return {"test": torch.tensor([1] * len(batch[next(iter(batch))]))}
>     
> dataset.set_transform(t)
> d_0 = dataset[0]
> ```
> 
> Still, the formatter's error message should mention that a dict of **sequences** is expected as the returned value (not just a dict) to make debugging easier.

ok, will change it according to suggestion. Thanks for the reply!
Increase speed of data files resolution
State: closed
Labels: description: New feature or request
name: enhancement

description: Issues a bit more difficult than "Good First" issues
name: good second issue

Certain datasets like `bigcode/the-stack-dedup` have so many files that loading them takes forever right from the data files resolution step.

`datasets` uses file patterns to check the structure of the repository but it takes too much time to iterate over and over again on all the data files.

This comes from `resolve_patterns_in_dataset_repository` which calls `_resolve_single_pattern_in_dataset_repository`, which iterates on all the files at
```python
glob_iter = [PurePath(filepath) for filepath in fs.glob(PurePath(pattern).as_posix()) if fs.isfile(filepath)]
```
but calling `glob` on such a dataset is too expensive. Indeed it calls `ls()` in `hffilesystem.py` too many times.

Maybe `glob` can be more optimized in `hffilesystem.py`, or the data files resolution can directly be implemented in the filesystem by checking its `dir_cache` ?

Comments:
#self-assign

You were right, if `self.dir_cache` is not None in glob, it is exactly the same as what is returned by find, at least for all the tests we have, and some extended evaluation I did across a random sample of about 1000 datasets. 

Thanks for the nice hints, and let me know if this is not exactly what we want here!

see PR: https://github.com/huggingface/datasets/pull/5704



I think we can make the data files resolution (significantly) faster in 2 steps:

1. `glob` calls `find` (which in turn calls `ls`), so we need `find` to be fast, and this can be achieved by fetching all the entries in a single API call and avoiding calls to `ls`. Implementing this for `HfFileSystem.find` (the one in `huggingface_hub`) is on my TO-DO list.
2. caching the repeated `find` calls in `_get_data_files_patterns` when the `data_files` patterns are not provided in `load_dataset`. To address this, we can introduce a `_resolve_single_pattern` function that would accept a filesystem object and a list of regex patterns to resolve. Then we can wrap this filesystem object in `_get_data_files_patterns` with an object that would cache the find calls before resolving the patterns with `_resolve_single_pattern`. (Feel free to suggest a cleaner implementation)

WDYT?

Good idea :) 

For 2:

That would work ! It's also possible to have a FileSystem with a cache on `.find` and use it inside the resolver passed to `_get_data_files_patterns`. Right now they're pretty simple:

```python
# for remote repositories
resolver = partial(_resolve_single_pattern_in_dataset_repository, dataset_info, base_path=base_path)
# for local
resolver = partial(_resolve_single_pattern_locally, base_path)
```

something like this maybe (with Quentin's reimplementation of `HfFilesystem.find`)?

 ```
 @lru_cache(max_size=None)
 def _find(self, path, maxdepth=None, withdirs=False, detail=False, **kwargs):
```

In any case please let me know if I can help in any way!
train_test_split in arrow_dataset does not ensure to keep single classes in test set
State: closed

### Describe the bug

When I have a dataset with very few (e.g. 1) examples per class and I call the train_test_split function on it, sometimes the single class will be in the test set. thus will never be considered for training.

### Steps to reproduce the bug

```
import numpy as np
from datasets import Dataset

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.train_test_split(test_size=0.5)
    data_set["train"]

    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test) 
```

### Expected behavior

I expect to have every available class at least once in my training set.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 11.0.0
- Pandas version: 1.3.5


Comments:
Hi! You can get this behavior by specifying `stratify_by_column="label"` in `train_test_split`.

This is the full example:
```python
import numpy as np
from datasets import Dataset, ClassLabel

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.cast_column("label", ClassLabel(num_classes=3))
    data_set = data_set.train_test_split(test_size=0.5, stratify_by_column="label")
    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test) 
```
Invalid Arrow data from JSONL
State: open
Labels: description: Something isn't working
name: bug

This code fails:
```python
from datasets import Dataset

ds = Dataset.from_json(path_to_file)
ds.data.validate()
```
raises

```python
ArrowInvalid: Column 2: In chunk 1: Invalid: Struct child array #3 invalid: Invalid: Length spanned by list offsets (4064) larger than values array (length 4063)
```

This causes many issues for @TevenLeScao:

- `map` fails because it fails to rewrite invalid arrow arrays

    ```python
    ~/Desktop/hf/datasets/src/datasets/arrow_writer.py in write_examples_on_file(self)
        438             if all(isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) for row in self.current_examples):
        439                 arrays = [row[0][col] for row in self.current_examples]
    --> 440                 batch_examples[col] = array_concat(arrays)
        441             else:
        442                 batch_examples[col] = [

    ~/Desktop/hf/datasets/src/datasets/table.py in array_concat(arrays)
    1885 
    1886     if not _is_extension_type(array_type):
    -> 1887         return pa.concat_arrays(arrays)
    1888 
    1889     def _offsets_concat(offsets):

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays()

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

    ArrowIndexError: array slice would exceed array length
    ```


- `to_dict()` **segfaults** ⚠️

    ```python
    /Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/array/data.cc:99:  Check failed: (off) <= (length) Slice offset greater 
    than array length
    ```

To reproduce: unzip the archive and run the above code using `sanity_oscar_en.jsonl`
[sanity_oscar_en.jsonl.zip](https://github.com/huggingface/datasets/files/10734124/sanity_oscar_en.jsonl.zip)

PS: reading using pandas and converting to Arrow works though (note that the dataset lives in RAM in this case):
```python
ds = Dataset.from_pandas(pd.read_json(path_to_file, lines=True))
ds.data.validate()
```


Comments:
[INVALID PR]
State: closed

Hi to whoever is reading this! 🤗

## What's in this PR?

~~Basically, I've removed the 🤗`datasets` installation as `python -m pip install ".[quality]" in the `check_code_quality` job in `.github/workflows/ci.yaml`, as we don't need to install the whole package to run the CI, unless that's done on purpose e.g. to check that the Python package installation succeeds before running the tests over the matrix of os?~~

~~So I just wanted to check whether the time was reduced doing this (which I assume it will), plus whether this is something that can be improved, or just discarded in case you're also using that step to make sure that the package can be installed.~~

## What's missing?

~~I was just wondering whether you consider replacing `isort` and `flake8` with `ruff` (if possible), since it's way faster, more information at [`ruff`](https://github.com/charliermarsh/ruff). Before creating this PR the average time of the `check_code_quality` job was around 40s.~~

## Edit

Sorry for the inconvenience this may have caused, didn't realise that the config is defined in `setup.cfg` and `pyproject.toml`, so running those without installing the Python package leads to failure, my bad 😞 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Checking that split name is correct happens only after the data is downloaded
State: open
Labels: description: Something isn't working
name: bug

### Describe the bug

Verification of split names (=indexing data by split) happens after downloading the data. So when the split name is incorrect, users learn about that only after the data is fully downloaded, for large datasets it might take a lot of time.

### Steps to reproduce the bug

Load any dataset with random split name, for example:
```python
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_11_0", "en", split="blabla")
```
and the download will start smoothly, despite there is no split named "blabla".

### Expected behavior

Raise error when split name is incorrect. 

### Environment info

`datasets==2.9.1.dev0`

Comments:
ClassLabel.cast_storage raises TypeError when called on an empty IntegerArray
State: closed

### Describe the bug

`ClassLabel.cast_storage` raises `TypeError` when called on an empty `IntegerArray`.

### Steps to reproduce the bug

Minimal steps:
```python
import pyarrow as pa
from datasets import ClassLabel

ClassLabel(names=['foo', 'bar']).cast_storage(pa.array([], pa.int64()))
```

In practice, this bug arises in situations like the one below:
```python
from datasets import ClassLabel, Dataset, Features, Sequence

dataset = Dataset.from_dict({'labels': [[], []]}, features=Features({'labels': Sequence(ClassLabel(names=['foo', 'bar']))}))

# this raises TypeError
dataset.map(batched=True, batch_size=1)
```

### Expected behavior

`ClassLabel.cast_storage` should return an empty Int64Array.

### Environment info

- `datasets` version: 2.9.1.dev0
- Platform: Linux-4.15.0-1032-aws-x86_64-with-glibc2.27
- Python version: 3.10.6
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments:
Reload features from Parquet metadata
State: closed

Resolves #5482. 

Attaches feature metadata to parquet files serialised using `Dataset.to_parquet`. 

This allows retrieving data with "rich" feature types (e.g., `datasets.features.image.Image` or `datasets.features.audio.Audio`) from parquet files without cumbersome casting (for an example, see #5482).

@lhoestq It seems that it is sufficient to attach metadata to the schema prior to serialising and features are loaded back with correct types afterwards automatically. 

I used the following script to test the implementation:

```python
from pathlib import Path

import datasets

dataset_name = "Maysee/tiny-imagenet"
ds = datasets.load_dataset(dataset_name, split=datasets.Split.TRAIN)
output_directory_path = Path(__file__).parent.joinpath("example_test_outputs", dataset_name.replace("/", "_"))
output_directory_path.mkdir(exist_ok=True, parents=True)
output_filepath = output_directory_path.joinpath("ds.parquet")
ds.to_parquet(str(output_filepath))

reloaded_ds = datasets.load_dataset(str(output_directory_path), split=datasets.Split.TRAIN)

assert ds.features == reloaded_ds.features
```

Prior to the change in this PR this script raises an `AssertionError` and the `Image` features lose their type after serialisation. After the change in this PR, the assertion does not raise an error and manual inspection of the features shows type `Image` for the respective columns of `reloaded_ds `.

Some open questions:
* How/where can I best add new unit tests for this implementation?
* What dataset would I best use in the tests? I chose `Maysee/tiny-imagenet` mainly because it is small and contains an ?Image` feature that can be used to test, but I'd be happy for suggestions on a suitable data source to use.
* Currently I'm calling `datasets.arrow_writer.ArrowWriter._build_metadata` as I need the same logic. However, I'm not happy with the coupling between `datasets.io.parquet` and `datasets.arrow_writer` it leaves me with. Suggest to factor this common logic out into a helper function and reuse it from both of these. Do you agree and if yes, could you please guide me where I would best place this function?

Many thanks in advance and kind regards,
MFreidank


  

Comments:
Thanks a lot for your help @lhoestq. I've simplified what turned out to be a simple fix and added the unit test.

Does this look ready to be merged or is there anything I'm still missing?

Cool ! I think you just need to remove the unused import in `io/parquet.py`
```
src/datasets/io/parquet.py:4:1: F401 'pyarrow as pa' imported but unused
```
and we're good to merge :)

_The documentation is not available anymore as the PR was closed or merged._

> Cool ! I think you just need to remove the unused import in `io/parquet.py`
> 
> ```
> src/datasets/io/parquet.py:4:1: F401 'pyarrow as pa' imported but unused
> ```
> 
> and we're good to merge :)

Done! Thanks a lot, this was fun :)