SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: [FR] `map` should reuse unchanged columns from the previous dataset to avoid disk usage

Comment:
You can use the `remove_columns` parameter in `map` to avoid duplicating the columns (and save disk space) and then concatenate the original dataset with the map result:
```python
from datasets import concatenate_datasets
# dummy example
ds_new = ds.map(lambda x: {"new_col": x["col"] + 2}, remove_columns=ds.column_names)
ds_combined = concatenate_datasets([ds, ds_new], axis=1)
```

Doing this automatically is hard to implement efficiently unless we know ahead of time which existing columns will be modified by a `map` transform. We have this info when `input_columns` are specified, so I think this is the only case we can optimize.
Issue: [FR] Transform Chaining, Lazy Mapping

Comment:
You can use `with_transform` to get a new dataset object.

Support for lazy `map` has already been discussed [here](https://github.com/huggingface/datasets/issues/3385) a little bit. Personally, I'm not a fan, as this would make `map` even more complex.
Issue: [FR] Transform Chaining, Lazy Mapping

Comment:
> You can use `with_transform` to get a new dataset object.
> 
> Support for lazy `map` has already been discussed [here](https://github.com/huggingface/datasets/issues/3385) a little bit. Personally, I'm not a fan, as this would make `map` even more complex.

I read about IterableDataset, and it seems to have lazy mapping. But I can't figure out how to convert an IterableDataset into a normal one when needed.

`with_transform` still does not chain AFAIU.
Issue: [FR] Transform Chaining, Lazy Mapping

Comment:
> I read about IterableDataset, and it seems to have lazy mapping. But I can't figure out how to convert an IterableDataset into a normal one when needed.

You must cache an `IterableDataset` to disk to load it as a `Dataset`. One way to do this is with `Dataset.from_generator`:
```python
from functools import partial
from datasets import Dataset

def gen_from_iterable_dataset(iterable_ds)
    yield from iterable_ds

ds = Dataset.from_generator(partial(gen_from_iterable_dataset, iterable_ds), features=iterable_ds.features})
```

> with_transform still does not chain AFAIU.

Yes, not supported yet - the solution is to combine the transforms into a single one.
Issue: [FR] Transform Chaining, Lazy Mapping

Comment:
I wonder if it would be beneficial to have a dedicated method to do that ? Maybe a `.save_to_disk()` so that the user can reload the resulting dataset later ?
Issue: [FR] Transform Chaining, Lazy Mapping

Comment:
> ```python
> from functools import partial
> from datasets import Dataset
> 
> def gen_from_iterable_dataset(iterable_ds)
>     yield from iterable_ds
> 
> ds = Dataset.from_generator(partial(gen_from_iterable_dataset, iterable_ds), features=iterable_ds.features})
> ```

@mariosasko With these complex mapping functions, what hash will be used to cache this dataset?
Issue: [FR] Transform Chaining, Lazy Mapping

Comment:
The params passed to `Dataset.from_generator` will be used to compute the hash (`partial` encapsulates the `iterable_ds` value, so changing it will also change the hash)
Issue: Documentation: wiki_dpr Dataset has no metric_type for Faiss Index

Comment:
Hi! You can do `ds.get_index("embeddings").faiss_index.metric_type` to get the metric type and then match the result with the FAISS metric [enum](https://github.com/facebookresearch/faiss/blob/43d86e30736ede853c384b24667fc3ab897d6ba9/faiss/MetricType.h#L22-L36) (should be L2).
Issue: Documentation: wiki_dpr Dataset has no metric_type for Faiss Index

Comment:
Ah! Thank you for pointing this out. FYI: the enum indicates it's using the inner product. Using `torch.inner` or `torch.dot` still produces a discrepancy compared to the built-in score. I think this is because of the compression/quantization that occurs with the FAISS index.
Issue: Improve `Dataset`'s string representation

Comment:
Yes, feel free to work on this.

You can check the PyArrow Table `__repr__`  and Polars DataFrame `__repr__`/`_repr_html_` implementations for some pointers/ideas.
Issue: Dataset.from_generator consistently freezes at ~1000 rows

Comment:
By default, we write data to disk (so it can be memory-mapped) every 1000 rows/samples. You can control this with the `writer_batch_size` parameter. Also, when working with fixed-size arrays, the `ArrayXD` feature types yield better performance (e.g., in your case, `features=datasets.Features({"i": datasets.Array3D(shape=(512,512,3), dtype="float32")})` should be faster).

Our support for multi-dim arrays could be better, and we plan to improve it as part of https://github.com/huggingface/datasets/issues/5272.
Issue: Dataset.from_generator consistently freezes at ~1000 rows

Comment:
> By default, we write data to disk (so it can be memory-mapped) every 1000 rows/samples. You can control this with the `writer_batch_size` parameter. Also, when working with fixed-size arrays, the `ArrayXD` feature types yield better performance (e.g., in your case, `features=datasets.Features({"i": datasets.Array3D(shape=(512,512,3), dtype="float32")})` should be faster).
> 
> Our support for multi-dim arrays could be better, and we plan to improve it as part of #5272.

Thanks for the explanation! The Image array was just for demonstration, I use PIL Images in practice. Does that make a difference? What's the best approach for a dataset with PIL Images as rows?
Issue: Get an error "OverflowError: Python int too large to convert to C long" when loading a large dataset

Comment:
This error means that one of the int32 (`Value("int32")`) columns in the dataset has a value that is out of the valid (int32) range.

I'll open a PR to print the name of a problematic column to make debugging such errors easier.
Issue: Get an error "OverflowError: Python int too large to convert to C long" when loading a large dataset

Comment:
`simhash` is the problematic column - it has values such as `18329103420363166823` that are out of the int64 range. You can fix this by setting the feature type to `Value("string")` (it's advised to use this type for hash values in general)

> Besides, it works fine when I am using streamed dataset.

Streaming yields Python dictionaries from the script without converting them to the Arrow representation, as this conversion step is not that cheap performance-wise.
Issue: Get an error "OverflowError: Python int too large to convert to C long" when loading a large dataset

Comment:
i am using uint64 for simhash

uint64 ranges up to about 3.69E19.

18329103420363166823 is less than this value.

moreover, our simhash algorithm use 64 bits. it should fit in uint64.
Issue: Get an error "OverflowError: Python int too large to convert to C long" when loading a large dataset

Comment:
You are right. I overlooked the feature type.

This is a reproducer:
```python
import pyarrow as pa
from datasets.arrow_writer import TypedSequence

pa.array(TypedSequence([18329103420363166823], type=Value("uint64")))
```

`pa.array([18329103420363166823])` also fails with the same error, so it seems PyArrow does not always infer the correct type as NumPy does (`uint64` in this case).

I'll report this issue in the Arrow repo.

`pa.array([18329103420363166823], pa.uint64)` works, so maybe we can implement a temporary fix (supporting complex input such as `[{"image": pil_image, "num": uint64_value}]` would be hard though).

In the meantime, you should be able to bypass this error by returning the `simhash` values as NumPy scalars in the script:
```python
def _generate_examples(self, ...):
    ...
    yield {...,  "simhash": np.uint64(simhash), ...}
```
Issue: Get an error "OverflowError: Python int too large to convert to C long" when loading a large dataset

Comment:
Thank you for checking this issue in detail.

However, it seems that using `np.uint64(simhash)` does not work. The same issue still exists.

https://huggingface.co/datasets/liwu/MNBVC/commit/1e44f1e400b7e61052647d44c99cdae3bae9c830

Anyway, we decide to use string type for these simhash values. Hope pyarrow can fix their bug soon.
Issue: extend the map function so it can wrap around long text that does not fit in the context window

Comment:
I just noticed the [docs](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L2881C11-L2881C200) say:

>If batched is `True` and `batch_size` is `n > 1`, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.

so maybe this is a bug then.
Issue: extend the map function so it can wrap around long text that does not fit in the context window

Comment:
All the values in a batch must be of the same length. So one solution is dropping all the input columns:
```python
data = data.map(lambda samples: tokenizer(samples["text"], max_length=tokenizer.model_max_length, truncation=True, stride=4, return_overflowing_tokens=True), batched=True, remove_columns=data.column_names)
```

Another is padding/transforming the input columns to the tokenizer output's length (447).
Issue: ValueError: Table schema does not match schema used to create file

Comment:
We'll do a new release of `datasets` soon to make the fix available :)

In the meantime you can use `datasets` from source (main)
Issue: Set a rule on the config and split names

Comment:
in this case we need to decide what to do with the existing datasets with white space characters (there shouldn't be a lot of them I think)
Issue: ConnectionError: Couldn't reach dataset_infos.json 

Comment:
Unfortunately, I can't reproduce the error. What does the following code return for you?
```python
import requests
from huggingface_hub import hf_hub_url
r = requests.get(hf_hub_url("codeparrot/codeparrot-clean-train", "dataset_infos.json", repo_type="dataset"))
```

Also, can you provide more info about your network (region, proxies, etc.)?
Issue: Why max_shard_size is not supported in load_dataset and passed to download_and_prepare

Comment:
Can you explain your use case for `max_shard_size`? 

On some systems, there is a limit to the size of a memory-mapped file, so we could consider exposing this parameter in `load_dataset`.
Issue: Why max_shard_size is not supported in load_dataset and passed to download_and_prepare

Comment:
In my use case, users may choose a proper size to balance the cost and benefit of using large shard size. (On azure blob or hdfs which may automatically download the shard from background)
Issue: Why max_shard_size is not supported in load_dataset and passed to download_and_prepare

Comment:
But `load_dataset` doesn't support caching (and reading) Arrow datasets from remote storage. 

`load_datset_builder` + `download_and_prepare` is not equal to `load_dataset`. The latter has one more step, `builder.as_dataset`, that memory-maps Arrow files, which only works for local files.
Issue: Why max_shard_size is not supported in load_dataset and passed to download_and_prepare

Comment:
Thanks. So if I want to use `IterableDataset` and control the size of single arrow file, how should I organize the data loader? Maybe `load_dataset_build` + `download_and_prepare` + `builder.as_dataset` + `dataset.to_iterable_dataset`?
Issue: Why max_shard_size is not supported in load_dataset and passed to download_and_prepare

Comment:
Yes, this should work.

I think we can expose `max_shard_size` in `load_dataset`, so feel free to open a PR.
Issue: Cannot reuse tokenizer object for dataset map

Comment:
This is a known issue: https://github.com/huggingface/datasets/issues/3847.

Fixing this requires significant work - rewriting the `tokenizers` lib to make them immutable.

The current solution is to pass `cache_file_name` to `map` to use that file for caching or calling a tokenizer before `map` (with the same set of parameters as the ones in the map transform)
Issue: AutoSharding IterableDataset's when num_workers > 1

Comment:
For this to be possible, we would have to switch from the "Streaming" Arrow format to the "Random Access" (IPC/Feather) format, which allows reading arbitrary record batches (explained [here](https://arrow.apache.org/docs/python/ipc.html)). We could then use these batches to construct shards.

@lhoestq @albertvillanova Do you think this use case is worth the switch? Also, we currently shard files, not inner row groups/chunks. Should we also support sharding row groups (e.g. if the number of input files is 1)?

PS: I don't expect significant speed-up for local, uncompressed Arrow files.
Issue: AutoSharding IterableDataset's when num_workers > 1

Comment:
Alternatively we could support multiprocessing map for iterable datasets and let the user do the CPU intensive task there ?

This way it would work on arrow data but also on any iterable dataset
Issue: AutoSharding IterableDataset's when num_workers > 1

Comment:
> For this to be possible, we would have to switch from the "Streaming" Arrow format to the "Random Access" (IPC/Feather) format, which allows reading arbitrary record batches (explained [here](https://arrow.apache.org/docs/python/ipc.html)). We could then use these batches to construct shards.
> 
> @lhoestq @albertvillanova Do you think this use case is worth the switch? Also, we currently shard files, not inner row groups/chunks. Should we also support sharding row groups (e.g. if the number of input files is 1)?
> 
> PS: I don't expect significant speed-up for local, uncompressed Arrow files.

Could you explain why you'd need to change the arrow format?

When we use streaming datasets we simply determine the number of worker shards and then add some modulo logic at the appropriate place. Worst case scenario, you'd skip streaming entries according to the number of shards.

For PyTorch, I'd be happy to provide an implementation or a sketch thereof, if you point me toward what the testing requirements would be for such a PR.
Issue: AutoSharding IterableDataset's when num_workers > 1

Comment:
> Could you explain why you'd need to change the arrow format?

This way workers have random access to the location of the file where its dataset subset starts. Currently we're using the Arrow streaming format which doesn't include the metadata of the record batches offsets. This is needed here to efficiently split a dataset made of one single file.
Issue: AutoSharding IterableDataset's when num_workers > 1

Comment:
> > Could you explain why you'd need to change the arrow format?
> 
> This way workers have random access to the location of the file where its dataset subset starts. Currently we're using the Arrow streaming format which doesn't include the metadata of the record batches offsets. This is needed here to efficiently split a dataset made of one single file.

I guess I don't understand why you'd need to subset the dataset in the first place. 
It seems sufficient to figure out how to offset or skip rows.

For instance, using pyArrow, you could use RecordBatchStreamReader to zero-copy iterate over records with read_next_batch and then only initiate the next step for records modulo worker shard.
That's one way to do it, where of course you'd need to account for gpu sharding as well.


Otherwise, how did you implement worker/node/GPU sharding for iterable/streaming data where you do not have index information or prior splits (e.g. files)?
Issue: AutoSharding IterableDataset's when num_workers > 1

Comment:
> For instance, using pyArrow, you could use RecordBatchStreamReader to zero-copy iterate over records with read_next_batch and then only initiate the next step for records modulo worker shard.

That works indeed ! And what we meant is that you can make it even faster to instantiate. Indeed using RecordBatchStreamReader you need to get the list of all the record batches in each worker, whereas you could just get the list of record batches per worker if you use the record batches locations in the Arrow IPC file footer. This would be especially appreciated to have a fast instantiation in case you have tens of thousands of Arrow files for example.
Issue: 404 on Datasets Documentation Page

Comment:
This wasn’t working for me a bit earlier, but it looks to be back up now
Issue: 404 on Datasets Documentation Page

Comment:
We had a minor issue updating the docs after the latest release. It should work now :).
Issue: Only two cores are getting used in sagemaker with pytorch 3.10 kernel

Comment:
I think it's more likely that this issue is related to PyTorch than Datasets, as PyTorch (on import) registers functions to execute when forking a process. Maybe this is the culprit: https://github.com/pytorch/pytorch/issues/99625
Issue: Only two cores are getting used in sagemaker with pytorch 3.10 kernel

Comment:
From reading that ticket, it may be down in mkl?  Is it worth hotfixing in the meantime, with the express intention of turning it off?  I know that's a horribly crufty solution, but it's also deeply frustrating to be limited to 2 cores for operations as simple as filtration.
Issue: Viewing dataset card returns “502 Bad Gateway”

Comment:
Yes, it seems to be working now. In case it's helpful, the outage lasted several days. It was failing as late as yesterday morning.
Issue: Streaming Dataset behind Proxy - FileNotFoundError

Comment:
Hi ! can you try to set the upper case environment variables `HTTP_PROXY` and `HTTPS_PROXY` ?

We use `aiohttp` for streaming and it uses case sensitive environment variables
Issue: Streaming Dataset behind Proxy - FileNotFoundError

Comment:
Hi, thanks for the quick reply.

I set the uppercase env variables with

`
os.environ['HTTP_PROXY'] = "http://example.com:xxxx" 
os.environ['HTTPS_PROXY'] = "http://example.com:xxxx" 
`

However, I still get the same error.

One thing that could be helpfull: When downloading a dataset without streaming i get the following message:
_HF google storage unreachable. Downloading and preparing it from source_.
The download does however work as expected.
Issue: Streaming Dataset behind Proxy - FileNotFoundError

Comment:
Are you able to use `aiohttp` to get the file at `https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json` using your proxy ?
Issue: Streaming Dataset behind Proxy - FileNotFoundError

Comment:
It only works when passing trust_env=True when creating the ClientSession, as well as setting ssl=False.

Working Example:

```
import os

os.environ['HTTP_PROXY'] = "xyz"
os.environ['HTTPS_PROXY'] = "xyz"

import asyncio
import aiohttp

async def download_pep(url):
    async with aiohttp.ClientSession(trust_env=True) as session:
        print("1")
        async with session.get(url, ssl=False) as resp:
            print("2")
            content = await resp.text()
            print(content)
            return content

asyncio.run(download_pep("https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json"))
```



SSL Verification has been a problem with other packages as well. Usually I circumvent the problem by setting
```
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
```
(probably not the best idea for security), although here aiohttp does not seem to use this default context.
Issue: Streaming Dataset behind Proxy - FileNotFoundError

Comment:
We do pass `trust_env` as well. Could you share the full stack trace you get when streaming using `datasets` ? That could help locate where we might have forgotten to pass `trust_env`
Issue: Streaming Dataset behind Proxy - FileNotFoundError

Comment:
> Is there a way to disable ssl verification when streaming a dataset.

I don't think so.

We use `fsspec` HTTPFileSystem implementation that is based on `aiohttp`. If you register a subclass of HTTPFileSystem that has SSL disabled by default it could work, but I wouldn't recommended it because it can raise security issues.
Issue: Streaming Dataset behind Proxy - FileNotFoundError

Comment:
Okay thanks for your help! I guess I have to figure out how to improve the proxy environment / see if I can make it work with ssl connections.
Issue: Docs: make "repository structure" easier to find

Comment:
Loading a local dataset also works the same way when `data_files` are not specified, so I agree we should make this info  easier to discover 

cc @stevhliu
Issue: Docs: make "repository structure" easier to find

Comment:
@benjaminbrown038 Yes, it is. Maybe @stevhliu can give some pointers on improving this doc page's discoverability.
Issue: Docs: make "repository structure" easier to find

Comment:
I think we can add a version of the [Main use-case](https://huggingface.co/docs/datasets/repository_structure#main-usecase) section to the [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset) tutorial. 

Currently, it doesn't tell you *how* to structure the repository; it only tells you how to create it. So adding the "main use-case" will help bridge the gap and make it easier to find. We should also add a link to the [Structure your repository](https://huggingface.co/docs/datasets/repository_structure) guide for users who want to learn about the other options.
Issue: description disappearing from Info when Uploading a Dataset Created with `from_dict`

Comment:
Here's a minimal way to reproduce the bug, for the sake of convenience.
````
from datasets import Dataset, DatasetInfo, load_dataset


episodes_dict = {"test":[1,2,3],"test2": [1,2,4]}

hugging_face_dataset = Dataset.from_dict(
    episodes_dict, info=DatasetInfo(description="test_str")
)
print(hugging_face_dataset.info)

hugging_face_dataset.push_to_hub("balisujohn/minari_test", private=True)

redownloaded_dataset= load_dataset("balisujohn/minari_test")["train"]


print(redownloaded_dataset.info)
````
Issue: description disappearing from Info when Uploading a Dataset Created with `from_dict`

Comment:
Thanks for reporting !

For now I would recommend uploading a separate JSON file for your metadata.

Alternatively you can upload a second configuration of the dataset containing your metadata but this feature is not released yet (though you can already use it from [here](https://github.com/huggingface/datasets/pull/5331), it will be released soon)
Issue: Common Voice datasets still need `use_auth_token=True`

Comment:
The issue commes from the dataset itself and is not related to the `datasets` lib

see https://huggingface.co/datasets/mozilla-foundation/common_voice_6_1/blob/2c475b3b88e0f2e5828f830a4b91618a25ff20b7/common_voice_6_1.py#L148-L152
Issue: Common Voice datasets still need `use_auth_token=True`

Comment:
Addressed in:

* `mozilla-foundation/common_voice_1_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_1_0/discussions/4)
* `mozilla-foundation/common_voice_2_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_2_0/discussions/3)
* `mozilla-foundation/common_voice_3_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_3_0/discussions/3)
* `mozilla-foundation/common_voice_4_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_4_0/discussions/3)
* `mozilla-foundation/common_voice_5_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_5_0/discussions/3)
* `mozilla-foundation/common_voice_5_1` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_5_1/discussions/3)
* `mozilla-foundation/common_voice_6_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_6_0/discussions/3)
* `mozilla-foundation/common_voice_6_1` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_6_1/discussions/3)
* `mozilla-foundation/common_voice_7_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0/discussions/3)
* `mozilla-foundation/common_voice_8_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0/discussions/7)
* `mozilla-foundation/common_voice_9_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_9_0/discussions/8)
* `mozilla-foundation/common_voice_10_0` [PR](https://huggingface.co/datasets/mozilla-foundation/common_voice_10_0/discussions/7)