noamaanMulla-03/datasets-issues huggingface dataset:
IterableDataset missing column_names, differs from Dataset interface

### Describe the bug

The documentation on [Stream](https://huggingface.co/docs/datasets/v1.18.2/stream.html) seems to imply that IterableDataset behaves just like a Dataset. However, examples like
```
dataset.map(augment_data, batched=True, remove_columns=dataset.column_names, ...)
```
will not work because `.column_names` does not exist on IterableDataset. I cannot find any clear explanation on why this is not available, is it an oversight? We do have `iterable_ds.features` available.

### Steps to reproduce the bug

See above

### Expected behavior

Dataset and IterableDataset would be expected to have the same interface, with any differences noted in the documentation.

### Environment info

n/a

Labels: name: enhancement

name: good first issue

Another example is that `IterableDataset.map` does not have `fn_kwargs`, among other arguments. It makes it harder to convert code from Dataset to IterableDataset.

Hi! `fn_kwargs` was added to `IterableDataset.map` in `datasets 2.5.0`, so please update your installation (`pip install -U datasets`) to use it.

Regarding `column_names`, I agree we should add this property to `IterableDataset`. In the meantime, you can use `list(dataset.features.keys())` instead.

Thanks! That's great news.

On Thu, Dec 22, 2022, 07:48 Mario Šaško ***@***.***> wrote:

> Hi! fn_kwargs was added to IterableDataset.map in datasets 2.5.0, so
> please update your installation (pip install -U datasets) to use it.
>
> Regarding column_names, I agree we should add this property to
> IterableDataset. In the meantime, you can use
> list(dataset.features.keys()) instead.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5383#issuecomment-1362993633>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHD6N2EQUFEOUFDW3VHSILWORZ45ANCNFSM6AAAAAATGKWVGM>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


I'm marking this issue as a "good first issue", as it makes sense to have `IterableDataset.column_names` in the API. Besides the case when `features` are `None` (e.g., `features` are `None` after `map`), in which we can also return `column_names` as `None`, adding this property should be straightforward,

Hi @mariosasko, I can work on this if that's ok?

Yes! I've assigned you the issue.
Authorization error

### Describe the bug

Get `Authorization error` when try to push data into hugginface datasets hub.

### Steps to reproduce the bug

I did all steps in the [tutorial](https://huggingface.co/docs/datasets/share), 

1. `huggingface-cli login` with WRITE token
2. `git lfs install`
3. `git clone https://huggingface.co/datasets/namespace/your_dataset_name`
4.
  ```
  cp /somewhere/data/*.json .
  git lfs track *.json
  git add .gitattributes
  git add *.json
  git commit -m "add json files"
  ```

but when I execute `git push` I got the error:

```
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
batch response: Authorization error.
error: failed to push some refs to 'https://huggingface.co/datasets/zeusfsx/ukrainian-news'
```

Size of data ~100Gb.  I have five json files - different parts.

### Expected behavior

All my data pushed into hub

### Environment info

- `datasets` version: 2.10.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.10.10
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


Hi! 

It's better to report this kind of issue in the `huggingface_hub` repo, so if you still haven't resolved it, I suggest you open an issue there.

Yeah,  I solved it. Problem was in osxkeychain.  When I do `hugginface-cli login` it's add token with default account (username)`hg_user` but my repo contain other username.  When I changed username in keychain -  it works now.
add kwargs to index search

This PR proposes to add kwargs to index search methods.

This is particularly useful for setting the timeout of a query on elasticsearch.

A typical use case would be:
```python
dset.add_elasticsearch_index("filename", es_client=es_client)
scores, examples = dset.get_nearest_examples("filename", "my_name-train_29", request_timeout=60)
```


_The documentation is not available anymore as the PR was closed or merged._
Parquet file wrongly recognized as zip prevents loading a dataset

### Describe the bug

When trying to `load_dataset_builder` for `HuggingFaceGECLM/StackExchange_Mar2023`, extraction fails, because parquet file [devops-00000-of-00001-22fe902fd8702892.parquet](https://huggingface.co/datasets/HuggingFaceGECLM/StackExchange_Mar2023/resolve/1f8c9a2ab6f7d0f9ae904b8b922e4384592ae1a5/data/devops-00000-of-00001-22fe902fd8702892.parquet) is wrongly identified by python as being a zip not a parquet.
(Full thread on [Slack](https://huggingface.slack.com/archives/C02V51Q3800/p1678890880803599))

### Steps to reproduce the bug

```python
from datasets import load_dataset_builder

ds = load_dataset_builder("HuggingFaceGECLM/StackExchange_Mar2023")
```

### Expected behavior

Loading the file normally.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1058-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Dataset cannot convert too large dictionnary

### Describe the bug

Hello everyone!

I tried to build a new dataset with the command "dict_valid = datasets.Dataset.from_dict({'input_values': values_array})".
However, I have a very large dataset (~400Go) and it seems that dataset cannot handle this.

Indeed, I can create the dataset until a certain size of my dictionnary, and then I have the error "OverflowError: Python int too large to convert to C long".

Do you know how to solve this problem?
Unfortunately I cannot give a reproductible code because I cannot share a so large file, but you can find the code below (it's a test on only a part of the validation data ~10Go, but it's already the case).

Thank you!

### Steps to reproduce the bug

SAVE_DIR = './data/'
features = h5py.File(SAVE_DIR+'features.hdf5','r')

valid_data = features["validation"]["data/features"]

v_array_values = [np.float32(item[()]) for item in valid_data.values()]
for i in range(len(v_array_values)):
    v_array_values[i] = v_array_values[i].round(decimals=5)

dict_valid = datasets.Dataset.from_dict({'input_values': v_array_values})

### Expected behavior

The code is expected to give me a Huggingface dataset.

### Environment info

python: 3.8.15
numpy: 1.22.3
datasets: 2.3.2
pyarrow: 8.0.0


Answered on the forum:

> To fix the overflow error, we need to merge [support LargeListArray in pyarrow by xwwwwww · Pull Request #4800 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/4800), which adds support for the large lists. However, before merging it, we need to come up with a cleaner API for large lists. I hope to find some time to address this before Datasets 3.0.
Features cannot be named "self"

### Describe the bug

Hi,
I noticed that we cannot create a HuggingFace dataset from Pandas DataFrame with a column named `self`.
The error seems to be coming from arguments validation in the `Features.from_dict` function.



### Steps to reproduce the bug

```python
import datasets
dummy_pandas = pd.DataFrame([0,1,2,3], columns = ["self"])
datasets.arrow_dataset.Dataset.from_pandas(dummy_pandas)
```

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.8.0
- Python version: 3.9.5
- PyArrow version: 6.0.1
- Pandas version: 1.4.1
xPath to implement all operations for Path

### Feature request

Current xPath implementation is a great extension of Path in order to work with remote objects. However some methods such as `mkdir` are not implemented correctly. It should instead rely on `fsspec` methods, instead of defaulting do `Path` methods which only work locally.

### Motivation

I'm using xPath to interact with remote objects.

### Your contribution

I could try to make a PR. I'm a bit unfamiliar with chaining right now.

Labels: name: enhancement

 I think https://github.com/fsspec/universal_pathlib is the project you are looking for.

`xPath` has the methods often used in dataset scripts, and `mkdir` is not one of them (`dl_manager`'s role is to "interact" with the file system, so using `mkdir` is discouraged).

Right is there a difference between UPath and xPath? Typically is xPath less well implemented compared to Upath, ie missing some implementations of some methods? Or are there methods in xPath that are not implemented with UPath?

`xPath` is an internal component (it doesn't have a leading underscore in the name, but it should) not meant to be used outside of `datasets`, and it's only tested on HTTP URLs, not S3.



Okay I understand that xPath won't support my usecase. What I was perhaps getting to is why not use UPath in `datasets` instead of `xPath` if UPath seems to have strictly more robust implementations.

It seems like `universal_pathlib` does not support `fsspec` URL chaining (`::` is the chaining symbol) and "compression" filesystems (e.g., `zip`), but this is what we need to access and stream files from within an archive (e.g., we want to stream URLs such as this one: `zip://data.parquet::https://www.dummyurl.com/archive.zip`)
How to assign new values to Dataset?

![image](https://user-images.githubusercontent.com/37113676/179149159-bbbda0c8-a661-403c-87ed-dc2b4219cd68.png)

Hi, if I want to change some values of the dataset, or add new columns to it, how can I do it?

For example, I want to change all the labels of the SST2 dataset to `0`:
```python
from datasets import load_dataset
data = load_dataset('glue','sst2')

data['train']['label'] = [0]*len(data)
```

I will get the error:
```
TypeError: 'Dataset' object does not support item assignment
```

Labels: name: enhancement

Hi! One option is use `map` with a function that overwrites the labels (`dset = dset.map(lamba _: {"label": 0}, features=dset.features`)). Or you can use the `remove_column` + `add_column` combination (`dset = dset.remove_columns("label").add_column("label", [0]*len(data)).cast(dset.features)`, but note that this approach creates an in-memory table for the added column instead of writing to disk, which could be problematic for large datasets.

Hi! I tried your proposed solution, but it does not solve my problem unfortunately. I am working with a set of protein sequences that have been tokenized with ESM, but some sequences are longer than `max_length`, they have been truncated in the tokenization. So now I want to truncate my labels as well, but that does not work with a mapping (e.g. `dset.map` as you suggested). Specifically, what I did was the following:

```
def postprocess_tokenize(tokenized_data):
    """
    adjust label lengths if they dont match.
    """
    if len(tokenized_data['input_ids']) < len(tokenized_data['labels']):
        new_labels = tokenized_data['labels'][:len(tokenized_data['input_ids'])]
        tokenized_data["labels"] = new_labels
    return tokenized_data

tokenized_data = tokenized_data.map(postprocess_tokenize, batched=True) # this does not adjust the labels...
```

Any tips on how to do this properly?

More generally, I am wondering why the DataCollator supports padding but does not support truncation? Seems odd to me.

Thanks in advance!
flatten_indices doesn't work with pandas format

### Describe the bug

Hi,
I noticed that `flatten_indices` throws an error when the batch format is `pandas`. This is probably due to the fact that flatten_indices uses map internally which doesn't accept dataframes as the transformation function output

### Steps to reproduce the bug

tabular_data = pd.DataFrame(np.random.randn(10,10))
tabular_data = datasets.arrow_dataset.Dataset.from_pandas(tabular_data)
tabular_data.with_format("pandas").select([0,1,2,3]).flatten_indices()

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.10.1
- Python version: 3.9.5
- PyArrow version: 11.0.0
- Pandas version: 1.4.1

Labels: name: bug

Thanks for reporting! This can be fixed by setting the format to `arrow` in `flatten_indices` and restoring the original format after the flattening. I'm working on a PR that reduces the number of the `flatten_indices` calls in our codebase and makes `flatten_indices` a no-op when a dataset does not have an indices mapping, so I'll incorporate the fix in that PR.
Pass custom metadata filename to Image/Audio folders

This is a quick fix. 
Now it requires to pass data via `data_files` parameters and include a required metadata file there and pass its filename as `metadata_filename` parameter.
For example, with the structure like:
```
data
   images_dir/
      im1.jpg
      im2.jpg
      ...
   metadata_dir/
      meta_file1.jsonl
      meta_file2.jsonl
      ...
```
to load data with `metadata_file1.jsonl` do:
```python
ds = load_dataset("imagefolder", data_files=["data/images_dir/**", "data/metadata_dir/meta_file1.jsonl"], metadata_filename="meta_file1.jsonl")
```

Note that if you have multiple splits, metadata file should be specified in each of them in `data_files`, smth like:
```python
data_files={
"train": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"], 
"test": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"]
}
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5635). All of your documentation changes will be reflected on that endpoint.

I'm not a big fan of this new param - I find assigning metadata files to splits via the `data_files` param cleaner. Also, assuming that the metadata filename is `metadata.json`/`metadata.csv` (I don't think we should allow other names), a user can do `load_dataset("imagefolder", data_dir="data")` to load a dataset with that structure.

@mariosasko  I don't really like this change in it's current state either but passing specific files with `data_files` also looks not quite user-friendly to me. The idea of providing specific parameter for metadata filename seems natural to me but I don't see a way for implementing it without some ugly changes in `load.py` (passing the param to factories and creating metadata patterns on the fly). Why don't you like this parameter?

For context: this PR emerged from the case where users wanted to use different metadata files with the same large set of images without copying directories on disk and it's not possible with `data_files` approach.

edit: ah no, it's possible if one  puts metadata files in different subdirs (so that the filenames can be left the same)

>For context: this PR emerged from the case where users wanted to use different metadata files with the same large set of images without copying directories on disk and it's not possible with data_files approach.
>
>edit: ah no, it's possible if one puts metadata files in different subdirs (so that the filenames can be left the same)

Seems low prio, but one way to address this would be by allowing to pass  "exclude patterns" to `data_files`
CI is broken: ModuleNotFoundError: jax requires jaxlib to be installed

CI test_py310 is broken: see https://github.com/huggingface/datasets/actions/runs/4498945505/jobs/7916194236?pr=5662
```
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_jax_in_memory - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_jax_on_disk - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_audio - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_device - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_image - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_jnp_array_kwargs - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/features/test_features.py::CastToPythonObjectsTest::test_cast_to_python_objects_jax - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
===== 8 failed, 2147 passed, 10 skipped, 37 warnings in 228.69s (0:03:48) ======
```

Labels: name: bug
Add instructions to create `DataLoader` from augmented dataset in object detection guide

The following adds instructions on how to create a `DataLoader` from the guide on how to use object detection with augmentations (#4710). I am open to hearing any suggestions for improvement !


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5579). All of your documentation changes will be reflected on that endpoint.

I'm not sure we need this part as we provide a link to the notebook that shows how to train an object detection model, and this notebook instantiates a `DataLoader` before training the model. I'd like to hear what @stevhliu thinks.

PS: Your `collate_fn` calls `torch.stack` on the `bbox` tensors, which don't have the same shape, so this will fail.

I agree with @mariosasko; we also have a [Use with PyTorch](https://huggingface.co/docs/datasets/use_with_pytorch) guide that shows how you can create a `DataLoader`.
Support for downloading only provided split

We can pass split to `_split_generators()`.
But I'm not sure if it's possible to solve cache issues, mostly with `dataset_info.json`


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5668). All of your documentation changes will be reflected on that endpoint.

My previous comment didn't create the retro-link in the PR. I write it here again.

You can check the context and the discussions we had about this feature enhancement in this PR:
- #2249
Custom split names

### Feature request

Hi,
I participated in multiple NLP tasks where there are more than just train, test, validation splits, there could be multiple validation sets or test sets. But it seems currently only those mentioned three splits supported. It would be nice to have the support for more splits on the hub. (currently i can have more splits when I am loading datasets from urls, but not hub)

### Motivation

Easier access to more splits

### Your contribution

No

Labels: name: enhancement

Hi!

You can also use names other than "train", "validation"  and "test". As an example, check the [script](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0/blob/e095840f23f3dffc1056c078c2f9320dad9ca74d/common_voice_11_0.py#L139) of the Common Voice 11 dataset.
Doc: save_to_disk, `num_proc` will affect `num_shards`, but it's not documented

### Describe the bug

[`num_proc`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_proc) will affect `num_shards`, but it's not documented


### Steps to reproduce the bug

Nothing to reproduce

### Expected behavior

[document of `num_shards`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_shards) explicitly says that it depends on `max_shard_size`, it should also mention `num_proc`.


### Environment info

datasets main document

Labels: name: documentation

name: good first issue

I agree this should be documented
adding progress bar / ETA for `load_dataset`

Please consider:
```
Downloading and preparing dataset oscar/unshuffled_deduplicated_en (download: 462.40 GiB, generated: 1.18 TiB, post-processed: Unknown size, total: 1.63 TiB) to cache/oscar/unshuffled_deduplicated_en/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
HF google storage unreachable. Downloading and preparing it from source
```
and no indication whatsoever of whether things work well or when it'll be done. It's important to have an estimated completion time for when doing slurm jobs since some instances have a cap on run-time.

I think for this particular job it sat for 30min in total silence and then after 30min it started generating:
```
897850 examples [07:24, 10286.71 examples/s]
```
which is already great!

Request: 
1. ETA - knowing how many hours to allocate for a slurm job
2. progress bar - helps to know things are working and aren't stuck and where we are at.

Thank you!

@lhoestq 


Labels: name: enhancement

Is this done now? I see progress bars when using `load_dataset`.

There are progress bars when downloading data and when preparing them as Arrow files.

The "total silence" part mentioned in OP refer to checksums verifications which have had some changes in the latest release 2.10:
- they're disabled by default (other less costly verifications are still done like checking the generated dataset size)
- they're using a tqdm bar as well
Add `Dataset.to_list` to the API

Since there is `Dataset.from_list` in the API, we should also add `Dataset.to_list` to be consistent.
 
Regarding the implementation, we can re-use `Dataset.to_dict`'s code and replace the `to_pydict` calls with `to_pylist`.

Labels: name: enhancement

name: good first issue

Hello, I have an interest in this issue.
Is the `Dataset.to_dict` you are describing correct in the code here?

https://github.com/huggingface/datasets/blob/35b789e8f6826b6b5a6b48fcc2416c890a1f326a/src/datasets/arrow_dataset.py#L4633-L4667

Yes, this is where `Dataset.to_dict` is defined.

#self-assign
Load/Save FAISS index using fsspec

### Feature request

From what I understand `faiss` already support this [link](https://github.com/facebookresearch/faiss/wiki/Index-IO,-cloning-and-hyper-parameter-tuning#generic-io-support)

I would like to use a stream as input to `Dataset.load_faiss_index` and `Dataset.save_faiss_index`.

### Motivation

In my case, I'm saving faiss index in cloud storage and use `fsspec` to load them. It would be ideal if I could send the stream directly instead of copying the file locally (or mounting the bucket) and then load the index.

### Your contribution

I can submit the PR

Labels: name: enhancement

Hi! Sure, feel free to submit a PR. Maybe if we want to be consistent with the existing API, it would be cleaner to directly add support for `fsspec` paths in `Dataset.load_faiss_index`/`Dataset.save_faiss_index` in the same manner as it was done in `Dataset.load_from_disk`/`Dataset.save_to_disk`.

That's a great idea! I'll do that instead.