ahmedfarazsyk/datasets-github-issues huggingface dataset:
the pile datasets url seems to change back
Author: wjfwzzc
State: closed

### Describe the bug

in #3627, the host url of the pile dataset became `https://mystic.the-eye.eu`. Now the new url is broken, but `https://the-eye.eu` seems to work again.

### Steps to reproduce the bug

```python3
from datasets import load_dataset
dataset = load_dataset("bookcorpusopen")
```
shows
```python3
ConnectionError: Couldn't reach https://mystic.the-eye.eu/public/AI/pile_preliminary_components/books1.tar.gz (ProxyError(MaxRetryError("HTTPSConnectionPool(host='mystic.the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_pr
eliminary_components/books1.tar.gz (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 504 Gateway Timeout')))")))
```

### Expected behavior

Downloading as normal.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- PyArrow version: 6.0.1
- Pandas version: 1.5.3

Comments: Thanks for reporting, @wjfwzzc.

I am transferring this issue to the corresponding dataset on the Hub: https://huggingface.co/datasets/bookcorpusopen/discussions/1

Thank you. All fixes are done:
- [x] https://huggingface.co/datasets/bookcorpusopen/discussions/2
- [x] https://huggingface.co/datasets/the_pile/discussions/1
- [x] https://huggingface.co/datasets/the_pile_books3/discussions/1
- [x] https://huggingface.co/datasets/the_pile_openwebtext2/discussions/2
- [x] https://huggingface.co/datasets/the_pile_stack_exchange/discussions/2
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number
Author: aalbersk
State: closed
Labels: name: good first issue

### Describe the bug

When dataset contains a 0-dim tensor, formatting.py raises a following error and fails.
```bash
Traceback (most recent call last):
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 501, in format_row
    return _unnest(formatted_batch)
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 137, in _unnest
    return {key: array[0] for key, array in py_dict.items()}
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 137, in <dictcomp>
    return {key: array[0] for key, array in py_dict.items()}
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number
```

### Steps to reproduce the bug

Load whichever dataset and add transform method to add 0-dim tensor. Or create/find a dataset containing 0-dim tensor. E.g.

```python
from datasets import load_dataset
import torch

dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
def t(batch):
    return {"test": torch.tensor(1)}
    
dataset.set_transform(t)
d_0 = dataset[0]
```

### Expected behavior

Extractor will correctly get a row from the dataset, even if it contains 0-dim tensor.

### Environment info

`datasets==2.8.0`, but it looks like it is also applicable to main branch version (as of 16th February)

Comments: Hi! The `set_transform` does not apply a custom formatting transform on a single example but the entire batch, so the fixed version of your transform would look as follows:
```python
from datasets import load_dataset
import torch

dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
def t(batch):
    return {"test": torch.tensor([1] * len(batch[next(iter(batch))]))}
    
dataset.set_transform(t)
d_0 = dataset[0]
```

Still, the formatter's error message should mention that a dict of **sequences** is expected as the returned value (not just a dict) to make debugging easier.

I can take this

Fixed in #5553 

> Hi! The `set_transform` does not apply a custom formatting transform on a single example but the entire batch, so the fixed version of your transform would look as follows:
> 
> ```python
> from datasets import load_dataset
> import torch
> 
> dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
> def t(batch):
>     return {"test": torch.tensor([1] * len(batch[next(iter(batch))]))}
>     
> dataset.set_transform(t)
> d_0 = dataset[0]
> ```
> 
> Still, the formatter's error message should mention that a dict of **sequences** is expected as the returned value (not just a dict) to make debugging easier.

ok, will change it according to suggestion. Thanks for the reply!
Increase speed of data files resolution
Author: lhoestq
State: closed
Labels: name: enhancement

name: good second issue

Certain datasets like `bigcode/the-stack-dedup` have so many files that loading them takes forever right from the data files resolution step.

`datasets` uses file patterns to check the structure of the repository but it takes too much time to iterate over and over again on all the data files.

This comes from `resolve_patterns_in_dataset_repository` which calls `_resolve_single_pattern_in_dataset_repository`, which iterates on all the files at
```python
glob_iter = [PurePath(filepath) for filepath in fs.glob(PurePath(pattern).as_posix()) if fs.isfile(filepath)]
```
but calling `glob` on such a dataset is too expensive. Indeed it calls `ls()` in `hffilesystem.py` too many times.

Maybe `glob` can be more optimized in `hffilesystem.py`, or the data files resolution can directly be implemented in the filesystem by checking its `dir_cache` ?

Comments: #self-assign

You were right, if `self.dir_cache` is not None in glob, it is exactly the same as what is returned by find, at least for all the tests we have, and some extended evaluation I did across a random sample of about 1000 datasets. 

Thanks for the nice hints, and let me know if this is not exactly what we want here!

see PR: https://github.com/huggingface/datasets/pull/5704



I think we can make the data files resolution (significantly) faster in 2 steps:

1. `glob` calls `find` (which in turn calls `ls`), so we need `find` to be fast, and this can be achieved by fetching all the entries in a single API call and avoiding calls to `ls`. Implementing this for `HfFileSystem.find` (the one in `huggingface_hub`) is on my TO-DO list.
2. caching the repeated `find` calls in `_get_data_files_patterns` when the `data_files` patterns are not provided in `load_dataset`. To address this, we can introduce a `_resolve_single_pattern` function that would accept a filesystem object and a list of regex patterns to resolve. Then we can wrap this filesystem object in `_get_data_files_patterns` with an object that would cache the find calls before resolving the patterns with `_resolve_single_pattern`. (Feel free to suggest a cleaner implementation)

WDYT?

Good idea :) 

For 2:

That would work ! It's also possible to have a FileSystem with a cache on `.find` and use it inside the resolver passed to `_get_data_files_patterns`. Right now they're pretty simple:

```python
# for remote repositories
resolver = partial(_resolve_single_pattern_in_dataset_repository, dataset_info, base_path=base_path)
# for local
resolver = partial(_resolve_single_pattern_locally, base_path)
```

something like this maybe (with Quentin's reimplementation of `HfFilesystem.find`)?

 ```
 @lru_cache(max_size=None)
 def _find(self, path, maxdepth=None, withdirs=False, detail=False, **kwargs):
```

In any case please let me know if I can help in any way!
train_test_split in arrow_dataset does not ensure to keep single classes in test set
Author: Ulipenitz
State: closed

### Describe the bug

When I have a dataset with very few (e.g. 1) examples per class and I call the train_test_split function on it, sometimes the single class will be in the test set. thus will never be considered for training.

### Steps to reproduce the bug

```
import numpy as np
from datasets import Dataset

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.train_test_split(test_size=0.5)
    data_set["train"]

    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test) 
```

### Expected behavior

I expect to have every available class at least once in my training set.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 11.0.0
- Pandas version: 1.3.5


Comments: Hi! You can get this behavior by specifying `stratify_by_column="label"` in `train_test_split`.

This is the full example:
```python
import numpy as np
from datasets import Dataset, ClassLabel

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.cast_column("label", ClassLabel(num_classes=3))
    data_set = data_set.train_test_split(test_size=0.5, stratify_by_column="label")
    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test) 
```
Invalid Arrow data from JSONL
Author: lhoestq
State: open
Labels: name: bug

This code fails:
```python
from datasets import Dataset

ds = Dataset.from_json(path_to_file)
ds.data.validate()
```
raises

```python
ArrowInvalid: Column 2: In chunk 1: Invalid: Struct child array #3 invalid: Invalid: Length spanned by list offsets (4064) larger than values array (length 4063)
```

This causes many issues for @TevenLeScao:

- `map` fails because it fails to rewrite invalid arrow arrays

    ```python
    ~/Desktop/hf/datasets/src/datasets/arrow_writer.py in write_examples_on_file(self)
        438             if all(isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) for row in self.current_examples):
        439                 arrays = [row[0][col] for row in self.current_examples]
    --> 440                 batch_examples[col] = array_concat(arrays)
        441             else:
        442                 batch_examples[col] = [

    ~/Desktop/hf/datasets/src/datasets/table.py in array_concat(arrays)
    1885 
    1886     if not _is_extension_type(array_type):
    -> 1887         return pa.concat_arrays(arrays)
    1888 
    1889     def _offsets_concat(offsets):

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays()

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

    ArrowIndexError: array slice would exceed array length
    ```


- `to_dict()` **segfaults** ⚠️

    ```python
    /Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/array/data.cc:99:  Check failed: (off) <= (length) Slice offset greater 
    than array length
    ```

To reproduce: unzip the archive and run the above code using `sanity_oscar_en.jsonl`
[sanity_oscar_en.jsonl.zip](https://github.com/huggingface/datasets/files/10734124/sanity_oscar_en.jsonl.zip)

PS: reading using pandas and converting to Arrow works though (note that the dataset lives in RAM in this case):
```python
ds = Dataset.from_pandas(pd.read_json(path_to_file, lines=True))
ds.data.validate()
```
Checking that split name is correct happens only after the data is downloaded
Author: polinaeterna
State: open
Labels: name: bug

### Describe the bug

Verification of split names (=indexing data by split) happens after downloading the data. So when the split name is incorrect, users learn about that only after the data is fully downloaded, for large datasets it might take a lot of time.

### Steps to reproduce the bug

Load any dataset with random split name, for example:
```python
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_11_0", "en", split="blabla")
```
and the download will start smoothly, despite there is no split named "blabla".

### Expected behavior

Raise error when split name is incorrect. 

### Environment info

`datasets==2.9.1.dev0`
ClassLabel.cast_storage raises TypeError when called on an empty IntegerArray
Author: marioga
State: closed

### Describe the bug

`ClassLabel.cast_storage` raises `TypeError` when called on an empty `IntegerArray`.

### Steps to reproduce the bug

Minimal steps:
```python
import pyarrow as pa
from datasets import ClassLabel

ClassLabel(names=['foo', 'bar']).cast_storage(pa.array([], pa.int64()))
```

In practice, this bug arises in situations like the one below:
```python
from datasets import ClassLabel, Dataset, Features, Sequence

dataset = Dataset.from_dict({'labels': [[], []]}, features=Features({'labels': Sequence(ClassLabel(names=['foo', 'bar']))}))

# this raises TypeError
dataset.map(batched=True, batch_size=1)
```

### Expected behavior

`ClassLabel.cast_storage` should return an empty Int64Array.

### Environment info

- `datasets` version: 2.9.1.dev0
- Platform: Linux-4.15.0-1032-aws-x86_64-with-glibc2.27
- Python version: 3.10.6
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Improve inconsistency of `Dataset.map` interface for `load_from_cache_file`
Author: HallerPatrick
State: closed
Labels: name: enhancement

### Feature request


1. Replace the `load_from_cache_file` default value to `True`. 
2. Remove or alter checks from `is_caching_enabled` logic.

### Motivation

I stumbled over an inconsistency in the `Dataset.map` interface. The documentation (and source) states for the parameter `load_from_cache_file`:
```
load_from_cache_file (`bool`, defaults to `True` if caching is enabled):
    If a cache file storing the current computation from `function`
    can be identified, use it instead of recomputing.
```

1. `load_from_cache_file` default value is `None`, while being annotated as `bool`
2. It is inconsistent with other method signatures like `filter`, that have the default value `True`
3. The logic is inconsistent, as the `map` method checks if caching is enabled through `is_caching_enabled`. This logic is not used for other similar methods.

### Your contribution

I am not fully aware of the logic behind caching checks. If this is just a inconsistency that historically grew, I would suggest to remove the `is_caching_enabled` logic as the "default" logic. Maybe someone can give insights, if environment variables have a higher priority than local variables or vice versa. 

If this is clarified, I could adjust the source  according to the "Feature request" section of this issue.

Comments: Hi, thanks for noticing this! We can't just remove the cache control as this allows us to control where the arrow files generated by the ops are written (cached on disk if enabled or a temporary directory if disabled). The right way to address this inconsistency would be by having `load_from_cache_file=None` by default everywhere.

Hi! Yes, this seems more plausible. I can implement that. One last thing is the type annotation `load_from_cache_file: bool = None`. Which I then would change to `load_from_cache_file: Optional[bool] = None`.

PR #5515 

Yes, `Optional[bool]` is the correct type annotation and thanks for the PR.
Some functions use a param named `type` shouldn't that be avoided since it's a Python reserved name?
Author: alvarobartt
State: closed

Hi @mariosasko, @lhoestq, or whoever reads this! :)

After going through `ArrowDataset.set_format` I found out that the `type` param is actually named `type` which is a Python reserved name as you may already know, shouldn't that be renamed to `format_type` before the 3.0.0 is released?

Just wanted to get your input, and if applicable, tackle this issue myself! Thanks 🤗

Comments: Hi! Let's not do this - renaming it would be a breaking change, and going through the deprecation cycle is only worth it if it improves user experience.

Hi @mariosasko, ok it makes sense. Anyway, don't you think it's worth it at some point to start a deprecation cycle e.g. `fs` in `load_from_disk`? It doesn't affect user experience but it's for sure a bad practice IMO, but's up to you 😄 Feel free to close this issue otherwise!

I don't think deprecating a param name in this particular instance is worth the hassle, so I'm closing the issue 🙂.

Sure, makes sense @mariosasko thanks!
Saving a dataset after setting format to torch doesn't work, but only if filtering
Author: joebhakim
State: closed

### Describe the bug

Saving a dataset after setting format to torch doesn't work, but only if filtering

### Steps to reproduce the bug

```
a = Dataset.from_dict({"b": [1, 2]})
a.set_format('torch')
a.save_to_disk("test_save") # saves successfully
a.filter(None).save_to_disk("test_save_filter") # does not
>> [...] TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types [<class 'torch.Tensor'>]. When using `batched=True`, make sure provided `function` returns a `dict` of types like `(<class 'list'>, <class 'numpy.ndarray'>)`.

# note: skipping the format change to torch lets this work.



### Expected behavior

Saving to work

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-6.1.9-arch1-1-x86_64-with-glibc2.36
- Python version: 3.10.9
- PyArrow version: 9.0.0
- Pandas version: 1.4.4

Comments: Hey, I'm a research engineer working on language modelling wanting to contribute to open source. I was wondering if I could give it a shot?

Hi! This issue was fixed in https://github.com/huggingface/datasets/pull/4972, so please install `datasets>=2.5.0` to avoid it.
Optimise behaviour in respect to indices mapping
Author: mariosasko
State: open
Labels: name: enhancement

_Originally [posted](https://huggingface.slack.com/archives/C02V51Q3800/p1675443873878489?thread_ts=1675418893.373479&cid=C02V51Q3800) on Slack_

Considering all this, perhaps for Datasets 3.0, we can do the following:
* [ ] have `continuous=True` by default in `.shard`  (requested in the survey and makes more sense for us since it doesn't create an indices mapping)
* [x] allow calling `save_to_disk` on "unflattened" datasets
* [ ] remove "hidden" expensive calls in `save_to_disk`, `unique`, `concatenate_datasets`, etc. For instance, instead of silently calling `flatten_indices`  where it's needed, it's probably better to be explicit (considering how expensive these ops can be) and raise an error instead
PyTorch BatchSampler still loads from Dataset one-by-one
Author: davidgilbertson
State: closed

### Describe the bug

In [the docs here](https://huggingface.co/docs/datasets/use_with_pytorch#use-a-batchsampler), it mentions the issue of the Dataset being read one-by-one, then states that using a BatchSampler resolves the issue.

I'm not sure if this is a mistake in the docs or the code, but it seems that the only way for a Dataset to be passed a list of indexes  by PyTorch (instead of one index at a time) is to define a `__getitems__` method (note the plural) on the Dataset object, and since the HF Dataset doesn't have this, PyTorch executes [this line of code](https://github.com/pytorch/pytorch/blob/master/torch/utils/data/_utils/fetch.py#L58), reverting to fetching one-by-one.

### Steps to reproduce the bug

You can put a breakpoint in `Dataset.__getitem__()` or just print the args from there and see that it's called multiple times for a single `next(iter(dataloader))`, even when using the code from the docs:
```py
from torch.utils.data.sampler import BatchSampler, RandomSampler
batch_sampler = BatchSampler(RandomSampler(ds), batch_size=32, drop_last=False)
dataloader = DataLoader(ds, batch_sampler=batch_sampler)
```



### Expected behavior

The expected behaviour would be for it to fetch batches from the dataset, rather than one-by-one.

To demonstrate that there is room for improvement: once I have a HF dataset `ds`, if I just add this line:
```py
ds.__getitems__ = ds.__getitem__
```
...then the time taken to loop over the dataset improves considerably (for wikitext-103, from one minute to 13 seconds with batch size 32). Probably not a big deal in the grand scheme of things, but seems like an easy win.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Comments: This change seems to come from a few months ago in the PyTorch side. That's good news and it means we may not need to pass a batch_sampler as soon as we add `Dataset.__getitems__` to get the optimal speed :)

Thanks for reporting ! Would you like to open a PR to add `__getitems__` and remove this outdated documentation ?

Yeah I figured this was the sort of thing that probably once worked. I can confirm that you no longer need the batch sampler, just `batch_size=n` in the `DataLoader`.

I'll pass on the PR, I'm flat out right now, sorry.
`load_dataset` has ~4 seconds of overhead for cached data
Author: davidgilbertson
State: open
Labels: name: enhancement

### Feature request

When loading a dataset that has been cached locally, the `load_dataset` function takes a lot longer than it should take to fetch the dataset from disk (or memory).
This is particularly noticeable for smaller datasets. For example, wikitext-2, comparing `load_data` (once cached) and `load_from_disk`, the `load_dataset` method takes 40 times longer.

⏱    4.84s ⮜ load_dataset
⏱    119ms ⮜ load_from_disk

### Motivation

I assume this is doing something like checking for a newer version.

If so, that's an age old problem: do you make the user wait _every single time they load from cache_ or do you do something like load from cache always, _then_ check for a newer version and alert if they have stale data. The decision usually revolves around what percentage of the time the data will have been updated, and how dangerous old data is.

For most datasets it's extremely unlikely that there will be a newer version on any given run, so 99% of the time this is just wasted time.

Maybe you don't want to make that decision for all users, but at least having the _option_ to not wait for checks would be an improvement.

### Your contribution

.

Comments: Hi ! To skip the verification step that checks if newer data exist, you can enable offline mode with `HF_DATASETS_OFFLINE=1`.

Although I agree this step should be much faster for datasets hosted on the HF Hub - we could just compare the commit hash from the local data and the remote git repository. We're not been leveraging the git commit hashes, since the library was built before we even had git repositories for each dataset on HF.

Thanks @lhoestq, for memory when I recorded those times I had `HF_DATASETS_OFFLINE` set.