josteece/github-issues huggingface dataset:
flatten_indices doesn't work with pandas format
Opened by alialamiidrissi (open)

### Describe the bug

Hi,
I noticed that `flatten_indices` throws an error when the batch format is `pandas`. This is probably due to the fact that flatten_indices uses map internally which doesn't accept dataframes as the transformation function output

### Steps to reproduce the bug

tabular_data = pd.DataFrame(np.random.randn(10,10))
tabular_data = datasets.arrow_dataset.Dataset.from_pandas(tabular_data)
tabular_data.with_format("pandas").select([0,1,2,3]).flatten_indices()

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.10.1
- Python version: 3.9.5
- PyArrow version: 11.0.0
- Pandas version: 1.4.1

Comments:
Thanks for reporting! This can be fixed by setting the format to `arrow` in `flatten_indices` and restoring the original format after the flattening. I'm working on a PR that reduces the number of the `flatten_indices` calls in our codebase and makes `flatten_indices` a no-op when a dataset does not have an indices mapping, so I'll incorporate the fix in that PR.
Features cannot be named "self"
Opened by alialamiidrissi (closed)

### Describe the bug

Hi,
I noticed that we cannot create a HuggingFace dataset from Pandas DataFrame with a column named `self`.
The error seems to be coming from arguments validation in the `Features.from_dict` function.



### Steps to reproduce the bug

```python
import datasets
dummy_pandas = pd.DataFrame([0,1,2,3], columns = ["self"])
datasets.arrow_dataset.Dataset.from_pandas(dummy_pandas)
```

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.8.0
- Python version: 3.9.5
- PyArrow version: 6.0.1
- Pandas version: 1.4.1

Comments:
Parquet file wrongly recognized as zip prevents loading a dataset
Opened by clefourrier (closed)

### Describe the bug

When trying to `load_dataset_builder` for `HuggingFaceGECLM/StackExchange_Mar2023`, extraction fails, because parquet file [devops-00000-of-00001-22fe902fd8702892.parquet](https://huggingface.co/datasets/HuggingFaceGECLM/StackExchange_Mar2023/resolve/1f8c9a2ab6f7d0f9ae904b8b922e4384592ae1a5/data/devops-00000-of-00001-22fe902fd8702892.parquet) is wrongly identified by python as being a zip not a parquet.
(Full thread on [Slack](https://huggingface.slack.com/archives/C02V51Q3800/p1678890880803599))

### Steps to reproduce the bug

```python
from datasets import load_dataset_builder

ds = load_dataset_builder("HuggingFaceGECLM/StackExchange_Mar2023")
```

### Expected behavior

Loading the file normally.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1058-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Comments:
xPath to implement all operations for Path
Opened by thomasw21 (closed)

### Feature request

Current xPath implementation is a great extension of Path in order to work with remote objects. However some methods such as `mkdir` are not implemented correctly. It should instead rely on `fsspec` methods, instead of defaulting do `Path` methods which only work locally.

### Motivation

I'm using xPath to interact with remote objects.

### Your contribution

I could try to make a PR. I'm a bit unfamiliar with chaining right now.

Comments:
 I think https://github.com/fsspec/universal_pathlib is the project you are looking for.

`xPath` has the methods often used in dataset scripts, and `mkdir` is not one of them (`dl_manager`'s role is to "interact" with the file system, so using `mkdir` is discouraged).

Right is there a difference between UPath and xPath? Typically is xPath less well implemented compared to Upath, ie missing some implementations of some methods? Or are there methods in xPath that are not implemented with UPath?

`xPath` is an internal component (it doesn't have a leading underscore in the name, but it should) not meant to be used outside of `datasets`, and it's only tested on HTTP URLs, not S3.



Okay I understand that xPath won't support my usecase. What I was perhaps getting to is why not use UPath in `datasets` instead of `xPath` if UPath seems to have strictly more robust implementations.

It seems like `universal_pathlib` does not support `fsspec` URL chaining (`::` is the chaining symbol) and "compression" filesystems (e.g., `zip`), but this is what we need to access and stream files from within an archive (e.g., we want to stream URLs such as this one: `zip://data.parquet::https://www.dummyurl.com/archive.zip`)
IterableDataset with_format does not support 'device' keyword for jax
Opened by Lime-Cakes (open)

### Describe the bug

As seen here: https://huggingface.co/docs/datasets/use_with_jax dataset.with_format() supports the keyword 'device', to put data on a specific device when loaded as jax. However, when called on an IterableDataset, I got the error `TypeError: with_format() got an unexpected keyword argument 'device'`

Looking over the code, it seems IterableDataset support only pytorch and no support for jax device keyword? 
https://github.com/huggingface/datasets/blob/fc5c84f36684343bff3e424cb0fd1ac5ecdd66da/src/datasets/iterable_dataset.py#L1029



### Steps to reproduce the bug

1. Load an IterableDataset (tested in streaming mode)
2. Call with_format('jax',device=device)

### Expected behavior

I expect to call `with_format('jax', device=device)`  as per [documentation](https://huggingface.co/docs/datasets/use_with_jax) without error

### Environment info

Tested with installing newest (dev) and also pip release (2.10.1).
- `datasets` version: 2.10.2.dev0
- Platform: Linux-5.15.89+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.3.5


Comments:
Hi! Yes, only `torch` is currently supported. Unlike `Dataset`, `IterableDataset` is not PyArrow-backed, so we cannot simply call `to_numpy` on the underlying subtables to format them numerically. Instead, we must manually convert examples to (numeric) arrays while preserving consistency with `Dataset`, which is not trivial, so this is still a to-do.

Any plans to support it in the future? Or would streaming dataset be left without support for jax and tensorflow?

@mariosasko 
`IterableDataset` and `with_format('torch', device=device)` also does not work (using `datasets==3.2.0`)
Pass custom metadata filename to Image/Audio folders
Opened by polinaeterna (open)

This is a quick fix. 
Now it requires to pass data via `data_files` parameters and include a required metadata file there and pass its filename as `metadata_filename` parameter.
For example, with the structure like:
```
data
   images_dir/
      im1.jpg
      im2.jpg
      ...
   metadata_dir/
      meta_file1.jsonl
      meta_file2.jsonl
      ...
```
to load data with `metadata_file1.jsonl` do:
```python
ds = load_dataset("imagefolder", data_files=["data/images_dir/**", "data/metadata_dir/meta_file1.jsonl"], metadata_filename="meta_file1.jsonl")
```

Note that if you have multiple splits, metadata file should be specified in each of them in `data_files`, smth like:
```python
data_files={
"train": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"], 
"test": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"]
}
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5635). All of your documentation changes will be reflected on that endpoint.

I'm not a big fan of this new param - I find assigning metadata files to splits via the `data_files` param cleaner. Also, assuming that the metadata filename is `metadata.json`/`metadata.csv` (I don't think we should allow other names), a user can do `load_dataset("imagefolder", data_dir="data")` to load a dataset with that structure.

@mariosasko  I don't really like this change in it's current state either but passing specific files with `data_files` also looks not quite user-friendly to me. The idea of providing specific parameter for metadata filename seems natural to me but I don't see a way for implementing it without some ugly changes in `load.py` (passing the param to factories and creating metadata patterns on the fly). Why don't you like this parameter?

For context: this PR emerged from the case where users wanted to use different metadata files with the same large set of images without copying directories on disk and it's not possible with `data_files` approach.

edit: ah no, it's possible if one  puts metadata files in different subdirs (so that the filenames can be left the same)

>For context: this PR emerged from the case where users wanted to use different metadata files with the same large set of images without copying directories on disk and it's not possible with data_files approach.
>
>edit: ah no, it's possible if one puts metadata files in different subdirs (so that the filenames can be left the same)

Seems low prio, but one way to address this would be by allowing to pass  "exclude patterns" to `data_files`
Dataset cannot convert too large dictionnary
Opened by MaraLac (open)

### Describe the bug

Hello everyone!

I tried to build a new dataset with the command "dict_valid = datasets.Dataset.from_dict({'input_values': values_array})".
However, I have a very large dataset (~400Go) and it seems that dataset cannot handle this.

Indeed, I can create the dataset until a certain size of my dictionnary, and then I have the error "OverflowError: Python int too large to convert to C long".

Do you know how to solve this problem?
Unfortunately I cannot give a reproductible code because I cannot share a so large file, but you can find the code below (it's a test on only a part of the validation data ~10Go, but it's already the case).

Thank you!

### Steps to reproduce the bug

SAVE_DIR = './data/'
features = h5py.File(SAVE_DIR+'features.hdf5','r')

valid_data = features["validation"]["data/features"]

v_array_values = [np.float32(item[()]) for item in valid_data.values()]
for i in range(len(v_array_values)):
    v_array_values[i] = v_array_values[i].round(decimals=5)

dict_valid = datasets.Dataset.from_dict({'input_values': v_array_values})

### Expected behavior

The code is expected to give me a Huggingface dataset.

### Environment info

python: 3.8.15
numpy: 1.22.3
datasets: 2.3.2
pyarrow: 8.0.0

Comments:
Answered on the forum:

> To fix the overflow error, we need to merge [support LargeListArray in pyarrow by xwwwwww · Pull Request #4800 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/4800), which adds support for the large lists. However, before merging it, we need to come up with a cleaner API for large lists. I hope to find some time to address this before Datasets 3.0.
Custom split names
Opened by ErfanMoosaviMonazzah (closed)

### Feature request

Hi,
I participated in multiple NLP tasks where there are more than just train, test, validation splits, there could be multiple validation sets or test sets. But it seems currently only those mentioned three splits supported. It would be nice to have the support for more splits on the hub. (currently i can have more splits when I am loading datasets from urls, but not hub)

### Motivation

Easier access to more splits

### Your contribution

No

Comments:
Hi!

You can also use names other than "train", "validation"  and "test". As an example, check the [script](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0/blob/e095840f23f3dffc1056c078c2f9320dad9ca74d/common_voice_11_0.py#L139) of the Common Voice 11 dataset.
adds early exit if url is `PathLike`
Opened by vvvm23 (open)

Closes #4864 

Should fix errors thrown when attempting to load `json` dataset using `pathlib.Path` in `data_files` argument.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5630). All of your documentation changes will be reflected on that endpoint.
load_dataset gives "403" error when using Financial phrasebank
Opened by Jimchoo91 (open)

When I try to load this dataset, I receive the following error: 

ConnectionError: Couldn't reach https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip (error 403)

Has this been seen before? Thanks. The website loads when I try to access it manually.

Comments:
Hi! You seem to be using an outdated version of `datasets` that downloads the older script version. To avoid the error, you can either pass `revision="main"` to `load_dataset` (this can fail if a script uses newer features of the lib) or update your installation with `pip install -U datasets` (better solution).
add kwargs to index search
Opened by SaulLu (closed)

This PR proposes to add kwargs to index search methods.

This is particularly useful for setting the timeout of a query on elasticsearch.

A typical use case would be:
```python
dset.add_elasticsearch_index("filename", es_client=es_client)
scores, examples = dset.get_nearest_examples("filename", "my_name-train_29", request_timeout=60)
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Allow "jsonl" data type signifier
Opened by BramVanroy (open)

### Feature request

`load_dataset` currently does not accept `jsonl` as type but only `json`. 

### Motivation

I was working with one of the `run_translation` scripts and used my own datasets (`.jsonl`) as train_dataset. But the default code did not work because

```
FileNotFoundError: Couldn't find a dataset script at jsonl\jsonl.py or any data file in the same directory. Couldn't find 'jsonl' on the Hugging Face Hub either: FileNotFoundError: Dataset 'jsonl' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

The reason is because the script has these lines to extract the data type by its extension. Therefore, the derived type is `jsonl` which is not recognized by datasets as the error above shows.

https://github.com/huggingface/transformers/blob/ade26bf9912f69e2110137443e4406d7dbe253e7/examples/pytorch/translation/run_translation.py#L342-L356

I suppose you could argue that this is the script's fault (in which case I'll do a PR over at `transformers`) but it makes sense to me to add `jsonl` as an alias to `json` in `datasets`.


### Your contribution

At the moment I cannot work on this. I think it can be as "easy" as having an alias for json, namely jsonl.

Comments:
You can use "json" instead. It doesn't work by extension names, but rather by dataset builder names, e.g. "text", "imagefolder", etc. I don't think the example in `transformers` is correct because of that

Yes, I understand the reasoning but this issue is to propose that the example in transformers (while incorrect) "makes sense" in terms of user expectation. So the question is whether it would be possible to add "aliases" for common types (like "json" and "text") based on common extensions (like jsonl and txt)?
glue datasets returning -1 for test split
Opened by lithafnium (closed)

### Describe the bug

Downloading any dataset from GLUE has -1 as class labels for test split. Train and validation have regular 0/1 class labels.  This is also present in the dataset card online. 

### Steps to reproduce the bug

```
dataset = load_dataset("glue", "sst2")
for d in dataset:
    # prints out -1
    print(d["label"]
```

### Expected behavior

Expected behavior should be 0/1 instead of -1. 

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- PyArrow version: 8.0.0
- Pandas version: 1.5.3



Comments:
Hi @lithafnium, thanks for reporting.

Please note that you can use the "Community" tab in the corresponding dataset page to start any discussion: https://huggingface.co/datasets/glue/discussions

Indeed this issue was already raised there (https://huggingface.co/datasets/glue/discussions/5) and answered: https://huggingface.co/datasets/glue/discussions/5#63907885937867f0cb3cde31
> The test labels are not public.
>
> Note this dataset belongs to a benchmark: people send their predictions for the test split to GLUE (https://gluebenchmark.com/) and then they get a score in their leaderboard...
Update README template to better template
Opened by emiltj (closed)


Comments:
IMO this template should stay generic.

Also, we now use [the card template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md) from `hugginface_hub` as the source of truth on the Hub (you now have the option to import it into the dataset card/README.md), so I think the next step would be deleting this template rather than updating it.

Agreed, the PR was a mistake and meant for my own repo. My bad

Feel free to close the PR then.
CI is broken after fsspec-2023.3.0 release
Opened by albertvillanova (closed)

As reported by @lhoestq, our CI is broken after `fsspec` 2023.3.0 release:
```
FAILED tests/test_filesystem.py::test_compression_filesystems[Bz2FileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
  At index 0 diff: {'name': 'file.txt', 'size': 70, 'type': 'file', 'created': 1678175677.1887748, 'islink': False, 'mode': 33188, 'uid': 1001, 'gid': 123, 'mtime': 1678175677.1887748, 'ino': 286957, 'nlink': 1} != 'file.txt'
  Full diff:
    [
  -  'file.txt',
  +  {'created': 1678175677.1887748,
  +   'gid': 123,
  +   'ino': 286957,
  +   'islink': False,
  +   'mode': 33188,
  +   'mtime': 1678175677.1887748,
  +   'name': 'file.txt',
  +   'nlink': 1,
  +   'size': 70,
  +   'type': 'file',
  +   'uid': 1001},
    ]
```
Also:
```
FAILED tests/test_filesystem.py::test_compression_filesystems[GzipFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[Lz4FileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[XzFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
FAILED tests/test_filesystem.py::test_compression_filesystems[ZstdFileSystem] - AssertionError: assert [{'created': ...: False, ...}] == ['file.txt']
===== 5 failed, 2134 passed, 18 skipped, 38 warnings in 157.21s (0:02:37) ======
```

See:
- fsspec/filesystem_spec#1205

Comments:
IterableDataset.add_column is unable to accept another IterableDataset as a parameter.
Opened by zsaladin (closed)

### Describe the bug

`IterableDataset.add_column` occurs an exception when passing another `IterableDataset` as a parameter.
The method seems to accept only eager evaluated values.

https://github.com/huggingface/datasets/blob/35b789e8f6826b6b5a6b48fcc2416c890a1f326a/src/datasets/iterable_dataset.py#L1388-L1391


I wrote codes below to make it.
```py
def add_column(dataset: IterableDataset, name: str, add_dataset: IterableDataset, key: str) -> IterableDataset:
    iter_add_dataset = iter(add_dataset)

    def add_column_fn(example):
        if name in example:
            raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
        return {name: next(iter_add_dataset)[key]}

    return dataset.map(add_column_fn)
```

Is there other way to do it? Or is it intended?

### Steps to reproduce the bug

Thie codes below occurs `NotImplementedError`
```py
from datasets import IterableDataset


def gen(num):
    yield {f"col{num}": 1}
    yield {f"col{num}": 2}
    yield {f"col{num}": 3}


ids1 = IterableDataset.from_generator(gen, gen_kwargs={"num": 1})
ids2 = IterableDataset.from_generator(gen, gen_kwargs={"num": 2})
new_ids = ids1.add_column("new_col", ids1)

for row in new_ids:
    print(row)
```

### Expected behavior

`IterableDataset.add_column` is able to task `IterableDataset` and lazy evaluated values as a parameter since IterableDataset is lazy evalued.

### Environment info
- `datasets` version: 2.8.0
- Platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.7
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


Comments:
Hi! You can use `concatenate_datasets([ids1, ids2], axis=1)` to do this.
Arrow map type in parquet files unsupported
Opened by TevenLeScao (open)

### Describe the bug

When I try to load parquet files that were processed with Spark, I get the following issue:

`ValueError: Arrow type map<string, string ('warc_headers')> does not have a datasets dtype equivalent.`

Strangely, loading the dataset with `streaming=True` solves the issue.

### Steps to reproduce the bug

The dataset is private, but this can be reproduced with any dataset that has Arrow maps.

### Expected behavior

Loading the dataset no matter whether streaming is True or not.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-5.15.0-1029-gcp-x86_64-with-glibc2.31
- Python version: 3.10.7
- PyArrow version: 8.0.0
- Pandas version: 1.4.2

Comments:
I'm attaching a minimal reproducible example:
```python
from datasets import load_dataset
import pyarrow as pa
import pyarrow.parquet as pq

table_with_map = pa.Table.from_pydict(
    {"a": [1, 2], "b": [[("a", 2)], [("b", 4)]]},
    schema=pa.schema({"a": pa.int32(), "b": pa.map_(pa.string(), pa.int32())})
)
pq.write_table(table_with_map, "parquet_with_map.parquet")
dset = load_dataset("parquet", data_files="parquet_with_map.parquet", split="train") # error unless streaming=True
``` 

For a dataset generated with the packaged loaders (CSV, JSON, Parquet), `streaming=True` sets the dataset's features to `None` (unless explicitly provided in `load_dataset`), hence no error will be thrown as long as the features stay "unresolved" (resolving the features with `_resolve_features` will lead to an error).

I've also been wondering about datasets support for Arrow Map datatypes. I had a situation where I had a pandas series of dict[str, float] with hundreds of different possible key values (ie. not bounded), and this got converted to a sequence of structs where every single struct had the entire set of keys.

I worked around it, by explicitly creating a sequence of [str, float], but given that pyarrow has an explicit Map datatype, it would be good to be able to explicitly cast/force this data type combination.

(feel free to ignore) polars will not support this type: https://github.com/pola-rs/polars/issues/3942#issuecomment-1202331210

> Polars will not add the map dtype. It's benefit do not outweigh the extra complexity. Maybe we can investigate conversion of maps to struct. But I will have to explore that.

Looks like they chose to convert every instance with https://github.com/pola-rs/polars/pull/4226