SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Streaming IterableDataset not working with translation pipeline

Comment:
@mariosasko The map snippet didn't quite work, but gave me enough of a clue to get it working.  The following snippet does work:
```
def en_translation(x):
    return {"en":x['translation']['en']}
ks = ds.map(en_translation, remove_columns=['translation'])
test=[]
for x in iter(ks):
    test.append(x['en'])
xx= mt(test)
for x in xx:
    print(x)
```

I tried just returning `x['translation']['en`]` in the helper function instead of the dict, but that didn't give me an iterator over strings that pipeline would work with either.


The snippet as is gives the following error:
```
Traceback (most recent call last):
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/pdb.py", line 1704, in main
    pdb._runscript(mainpyfile)
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/pdb.py", line 1573, in _runscript
    self.run(statement)
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/jlquinn/models/hf/ende.t5.pipe.py", line 1, in <module>
    from transformers import pipeline
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/site-packages/transformers/pipelines/text2text_generation.py", line 335, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/site-packages/transformers/pipelines/text2text_generation.py", line 138, in __call__
    result = super().__call__(*args, **kwargs)
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1027, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1033, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/site-packages/transformers/pipelines/text2text_generation.py", line 287, in preprocess
    return super()._parse_and_tokenize(*args, truncation=truncation)
  File "/home/jlquinn/miniconda3/envs/watnlp/lib/python3.9/site-packages/transformers/pipelines/text2text_generation.py", line 100, in _parse_and_tokenize
    raise ValueError(
ValueError:  `args[0]`: <datasets.iterable_dataset.IterableDataset object at 0x7f5fd38ef1c0> have the wrong format. The should be either of type `str` or type `list`
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
```
Issue: Streaming IterableDataset not working with translation pipeline

Comment:
So perhaps there's no bug exactly, but I would love to see two things:  1) improve the documentation to better understand what's really getting returned.  2) update the example provided of using transformer pipeline with a dataset to include the oddball case that translation appears to be.
Issue: Streaming IterableDataset not working with translation pipeline

Comment:
Hi,

for the original snippet, the issue is that `streaming` datasets are not countable (they have no len) and therefore `KeyDataset` cannot work with them ( KeyDataset is a dataset and therefore requires a length).

I modified slightly the original snippet to make it work:

```python
from transformers import pipeline
from transformers.pipelines.pt_utils import KeyDataset
from datasets import load_dataset

ds = load_dataset(path="wmt14", name="fr-en", split="test", streaming=True)
bs = 1
mt = pipeline(
    "translation_en_to_fr", model="hf-internal-testing/tiny-random-T5ForConditionalGeneration", batch_size=bs
)


def ks(ds):
    for item in ds:
        yield item["translation"]["en"]


# print(f"{ks}")
xx = mt(ks(ds))
for x in xx:
    print(x)
```

This is what the first example in the docs suggests to use (as it's the most flexible): https://huggingface.co/docs/transformers/v4.29.1/en/pipeline_tutorial#using-pipelines-on-a-dataset

`KeyDataset` really exists only to get a `sized` dataset to work nicer with `tqdm` for instance.

@sgugger should we update the docs to remove `KeyDataset` entirely ? (We can add a note to pass manually the length of the data to tqdm so that the progress bar option can still be easy to use ?)
Issue: Streaming IterableDataset not working with translation pipeline

Comment:
Maybe moving `KeyDataset` later on in the guide and specify it's mostly for streaming then? Or is it also necessary for batch_size>1 (which is what the current doc implies)?
Issue: Streaming IterableDataset not working with translation pipeline

Comment:
Hmm

Iterator (`yield`) :
- Not countable
- Super flexible
- Cannot use `num_workers>1` (threading requires indexing at the correct location, iterators require to iterate in order,so each thread would iterate over the full thing being genuinely a bad idea)
- Can batch
- tqdm doesn't show a nice progress bar (it has no total)

KeyDataset (Or any PyTorch like Dataset returning the correct object for the pipeline):
- Countable
- Less flexible (not applicable to datasets with streaming), can only work on single keys. But should be easy to read and write your own (like @mariosasko did)
- Works with  `num_workers > 1` (Every worker can fetch exactly what's needed)
- Can batch 
- tqdm shows a nice progress bar

In the docs, if we update all the examples to use iterators, and include an example with

```
for item in tqdm.tqdm(pipe(iterator(), total=len(dataset))))
```

We can save the biggest feature that doesn't work out of the box with iterators which is the tqdm progress bar.

`num_workers>1` we can mention it, but it tends to be an issues only on CPU intensive loads, like image (and maybe audio)
Issue: load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!

Comment:
This is due to the slow resolution of the data files: https://github.com/huggingface/datasets/issues/5537.

We plan to switch to `huggingface_hub`'s `HfFileSystem` soon to make the resolution faster (will be up to 20x faster once we merge https://github.com/huggingface/huggingface_hub/pull/1443)
Issue: load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!

Comment:
You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.
Issue: load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!

Comment:
> You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.

That's unrelated to the problem discussed in this issue.
Issue: load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!

Comment:
> > You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.
> 
> That's unrelated to the problem discussed in this issue.

Sorry, I misunderstood it.
Issue: Abusurdly slow on iteration

Comment:
Hi ! You can try to use the [Image](https://huggingface.co/docs/datasets/v2.12.0/en/package_reference/main_classes#datasets.Image) type which [decodes images on-the-fly](https://huggingface.co/docs/datasets/v2.12.0/en/about_dataset_features#image-feature) into pytorch tensors :)

```python
ds = Dataset.from_dict({"tensor":a}).with_format("torch")
%time sum(1 for _ in ds)
# CPU times: user 5.04 s, sys: 96.5 ms, total: 5.14 s
# Wall time: 5.14 s
# 10000
```

```python
features = Features({"tensor": Image()})
ds = Dataset.from_dict({"tensor":a}, features=features).with_format("torch")
%time sum(1 for _ in ds)
# CPU times: user 1.86 s, sys: 49 ms, total: 1.91 s
# Wall time: 1.9 s
# 10000
```

-> Speed x2.7

And if you want to keep using arrays of integers, consider using the [Array2D](https://huggingface.co/docs/datasets/v2.12.0/en/package_reference/main_classes#datasets.Array2D) or [Array3D](https://huggingface.co/docs/datasets/v2.12.0/en/package_reference/main_classes#datasets.Array3D) types which are even faster (since it doesn't decode images):

```python
features = Features({"tensor": Array2D(shape=(100, 224), dtype="float32")})
ds = Dataset.from_dict({"tensor":a}, features=features).with_format("torch")
%time sum(1 for _ in ds)
# CPU times: user 828 ms, sys: 68.4 ms, total: 896 ms
# Wall time: 897 ms
# 10000
```

-> Speed x5.7

Batching also speeds up a lot

```python
from torch.utils.data import DataLoader
dl = DataLoader(ds, batch_size=100)
%time sum(1 for _ in dl)
# CPU times: user 564 ms, sys: 83.5 ms, total: 648 ms
# Wall time: 579 ms
# 100
```

-> Speed x8.9

```python
%time sum(1 for _ in ds.iter(batch_size=100))
# CPU times: user 119 ms, sys: 96.8 ms, total: 215 ms
# Wall time: 117 ms
# 100
```

-> Speed x46
Issue: Abusurdly slow on iteration

Comment:
Anyway, regarding the speed difference between numpy and pytorch, I think the issue is that we first convert numpy sub-arrays to pytorch and then consolidate into one tensor, while we should to the opposite. Indeed converting a numpy array to pytorch has a fix cost that seems to cause a slow down. The current pipeline is

```
arrow -> nested numpy arrays -> lists of torch tensors -> one torch tensor
```

and we should do

```
arrow -> nested numpy arrays -> one numpy array -> one torch tensor
```
Issue: Abusurdly slow on iteration

Comment:
I have a similar issue: iterating over a dataset takes 5s without applying any transform, but takes ~30s after applying a transform.
Here is the minimum code to reproduce the problem

```python
import numpy as np
from datasets import Dataset, DatasetDict, load_dataset, Array3D, Image, Features
from torch.utils.data import DataLoader
from tqdm import tqdm
import torchvision 
from torchvision.transforms import ToTensor, Normalize


#################################
# Without transform
#################################
    
train_dataset = load_dataset(
    'cifar100',
    split='train',
    use_auth_token=True,
)

train_dataset.set_format(type="numpy", columns=["img", "fine_label"])

train_loader= DataLoader(
    train_dataset,
    batch_size=100,
    pin_memory=False,
    shuffle=True,
    num_workers=8,
)

for batch in tqdm(train_loader, desc="Loading data, no transform"):
    pass


#################################
# With transform
#################################

transform_func = torchvision.transforms.Compose([
    ToTensor(), 
    Normalize(mean=[0.485, 0.456, 0.406], std= [0.229, 0.224, 0.225]),]          
)
    
train_dataset = train_dataset.map(
    desc=f"Preprocessing samples",
    function=lambda x: {"img": transform_func(x["img"])},
)

train_dataset.set_format(type="numpy", columns=["img", "fine_label"])


train_loader= DataLoader(
    train_dataset,
    batch_size=100,
    pin_memory=False,
    shuffle=True,
    num_workers=8,
)


for batch in tqdm(train_loader, desc="Loading data after transform"):
    pass 
```

I have also tried converting the Image column to an Array3D
```python
img_shape = train_dataset[0]["img"].shape

features = train_dataset.features.copy()
features["x"] = Array3D(shape=img_shape, dtype="float32")

train_dataset = train_dataset.map(
    desc=f"Preprocessing samples",
    function=lambda x: {"x": np.array(x["img"], dtype=np.uint8)},
    features=features,
)
train_dataset.cast_column("x", Array3D(shape=img_shape, dtype="float32"))
train_dataset.set_format(type="numpy", columns=["x", "fine_label"])
```
but to no avail. Any clue?
Issue: Remove columns in interable dataset

Comment:
This method has been recently added to the `IterableDataset`, so you need to update the `datasets`' installation (`pip install -U datasets`) to use it.
Issue: Streaming support for `load_from_disk`

Comment:
As the name says, `load_from_disk` load the data from your disk. If the data is hosted on S3, it is first downloaded locally and then loaded from your disk.

There is a discussion on streaming data from S3 here though: #5281
Issue: Streaming support for `load_from_disk`

Comment:
@lhoestq 
Thanks for your comment. I have checked out the discussion before and attempted at replicating the mentioned changes in the main branch (#5580). What I found was that if a dataset is saved using `save_to_disk`, it cannot be read by `load_dataset`. The error message asks me to to use `load_from_disk` instead. What would be the correct way of saving the data in this scenario?
Issue: Streaming support for `load_from_disk`

Comment:
Using `push_to_hub` you can save the dataset on the HF Hub as parquet files, and reload it / stream it using `load_dataset` :)

If you want to save your dataset somewhere else you can use `.to_parquet` to get a parquet file. If your dataset is big it's usually recommended to shard it into multi parquet files (around 1GB each).
Issue: Streaming support for `load_from_disk`

Comment:
@lhoestq
I tried the method you mentioned. This the current scenario I'm facing:

- The parquet file can be read from disk and streaming can be enabled.
- The parquet file can be read from `s3` (local MinIO).
- When `streaming=True` is enabled for `s3`, I get the error mentioned below:

```
File ~/.../lib/python3.8/site-packages/s3fs/core.py:502, in S3FileSystem.set_session(self, refresh, kwargs)
    500 conf = AioConfig(**config_kwargs)
    501 if self.session is None:
--> 502     self.session = aiobotocore.session.AioSession(**self.kwargs)
    504 for parameters in (config_kwargs, self.kwargs, init_kwargs, client_kwargs):
    505     for option in ("region_name", "endpoint_url"):

TypeError: __init__() got an unexpected keyword argument 'headers'
```

Does this mean there is a bug in the main branch?
Issue: Streaming support for `load_from_disk`

Comment:
Streaming from S3 is still experimental, there might be a few bugs unfortunately.

Can you share the full stack trace ?
Issue: Streaming support for `load_from_disk`

Comment:
Is `"data-parquet"` a file ? In `data_files` you should pass the paths to the parquet files (not to a directory). Glob patterns are not supported yet for S3 URLs.

The bug seems to happen because your provided data file has no extension. Because of that it tries to infer it from the file content, but fails because `_get_extraction_protocol` doesn't support S3 URLs yet.
Issue: Streaming support for `load_from_disk`

Comment:
@lhoestq 
Thank you for your answer. Saving the file with `.parquet` extension solved the issue! This is really great! Really appreciate all the help! 

Let me know if I should close the issue or feel free to close it if you want.
Issue: Streaming support for `load_from_disk`

Comment:
Cool ! I'm glad it worked out :)

Sure feel free to close the issue, since the original question about streaming with load_from_disk has been answered anyway
Issue: Use DeepSpeed load myself " .csv " dataset.

Comment:
Hi ! Doing `load_dataset("path/to/data.csv")` is not supported yet, but you can do

```python
ds =  load_dataset("csv", data_files=["path/to/data.csv"])
```
Issue: Is uint8 supported?

Comment:
Hi ! The numpy formatting detaults to int64 and float32 - but you can use uint8 using
```python
ds = ds.with_format("numpy", dtype=np.uint8)
```
Issue: Is uint8 supported?

Comment:
Thank you!
By setting `ds.with_format("numpy", dtype=np.uint8)`, the dataset returns the data in `uint8`.

However, `with_format` and `set_format` seem to cast the data on-the-fly.
I want to reduce the dataset size by using `uint8` instead of `int64` and I observe no difference between using `int64` and `uint8` for the vector.
Is there any way to actually store the data in `uint8` and save the disk space and the downloading time when loaded from the hub?
Issue: Is uint8 supported?

Comment:
If the feature type is `Value("uint8")` then it's written an uint8 on disk using the uint8 Arrow dtype.

e.g.
```python
ds = Dataset.from_dict({"a": range(10)}, features=Features({"a": Value("uint8")}))
ds.data.nbytes
# 10
```
Issue: Is uint8 supported?

Comment:
Oh, I understand now.
The data was stored in `uint8` from the beginning (when the dataset returns `int64`).

Thank you for your time!
My question is fully resolved.
Issue: Unable to push dataset - `create_pr` problem

Comment:
Thanks for reporting, @agombert.

In this case, I think the root issue is authentication: before pushing to Hub, you should authenticate. See our docs: https://huggingface.co/docs/datasets/upload_dataset#upload-with-python
> 2. To upload a dataset on the Hub in Python, you need to log in to your Hugging Face account:
   ```
   huggingface-cli login
   ```
Issue: Unable to push dataset - `create_pr` problem

Comment:
Hey @albertvillanova well I actually did :D 

<img width="1079" alt="Capture d’écran 2023-05-09 à 18 02 58" src="https://github.com/huggingface/datasets/assets/17645711/e091aa20-06b1-4dd3-bfdb-35e832c66f8d">
Issue: Unable to push dataset - `create_pr` problem

Comment:
That is weird that you get a Forbidden error if you are properly authenticated...

Today we had a big outage issue affecting the Hugging Face Hub. Could you please retry to push_to_hub your dataset? Maybe that was the cause...
Issue: Unable to push dataset - `create_pr` problem

Comment:
Login successful but also got this error "Forbidden: pass `create_pr=1` as a query parameter to create a Pull Request"
Issue: Unable to push dataset - `create_pr` problem

Comment:
Make sure your API token has a `write` role. I had the same issue as you with the `read` token. Creating a `write` token and using that solved the issue.
Issue: Unable to push dataset - `create_pr` problem

Comment:
> Make sure your API token has a `write` role. I had the same issue as you with the `read` token. Creating a `write` token and using that solved the issue.

I generate a token with write role. It works! thank you so much.
Issue: Unable to push dataset - `create_pr` problem

Comment:
@dmitrijsk amazing thanks so much ! 
The error should be clearer when the token is read-only – I wasted a lot of time there..
Issue: [Bug]504 Server Error when loading dataset which was already cached

Comment:
I am experiencing the same problem with the following environment:

* `datasets` version: 2.11.0
* Platform: `Linux 5.19.0-41-generic x86_64 GNU/Linux`
* Python version: `3.8.5`
* Huggingface_hub version: 0.13.3
* PyArrow version: `11.0.0`
* Pandas version: `1.5.3`

Trying to get some diagnostics, I got the following:  

```python
>>> from huggingface_hub import scan_cache_dir
>>> sd = scan_cache_dir()
>>> sd
HFCacheInfo(size_on_disk=0, repos=frozenset(), warnings=[CorruptedCacheException('Repo path is not a directory: /home/myname/.cache/huggingface/hub/version_diffusers_cache.txt')])

```
However, that might also be because I had tried to manually specify the `cache_dir` and that resulted in trying to download the dataset again ... but into a folder one level higher up than it should have.

Note that my issue is with the `huggan/wikiart` dataset, so it is not a dataset-specific issue.
Issue: [Bug]504 Server Error when loading dataset which was already cached

Comment:
same problem with a private dataset repo, seems the huggingface hub server got some connection problem?
Issue: [Bug]504 Server Error when loading dataset which was already cached

Comment:
@SingL3 You can avoid this error by setting the [`HF_DATASETS_OFFLINE`](https://huggingface.co/docs/datasets/v2.12.0/en/loading#offline) env variable to 1. By default,  if an internet connection is available, we check whether the cache of a cached dataset is up-to-date.

@lucidBrot `datasets`' cache is still not aligned with `huggigface_hub`'s. We plan to align it eventually.
Issue: [Bug]504 Server Error when loading dataset which was already cached

Comment:
Today we had a big issue affecting the Hugging Face Hub, thus all the `504 Server Error: Gateway Time-out` errors.

It is fixed now and loading your datasets should work as expected.
Issue: [Bug]504 Server Error when loading dataset which was already cached

Comment:
Hi, @albertvillanova.
If there is a locally cached version of datasets or something cache using huggingface_hub, when a network problem(either client or server) occurs, is it a better way to fallback to use the current cached version rather than raise a exception and exit?
Issue: (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

Comment:
That is weird. I can't reproduce it again after reboot.
```python
In [2]: import platform

In [3]: platform.platform()
Out[3]: 'macOS-13.2-arm64-arm-64bit'

In [4]: from datasets import load_dataset
   ...:
   ...: jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
Found cached dataset parquet (/Users/sarit/.cache/huggingface/datasets/nomic-ai___parquet/nomic-ai--gpt4all-j-prompt-generations-a3b62015e2e52043/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 63.25it/s]
```
Issue: Stream data concatenation issue

Comment:
Hi! 

You can call `map` as follows to avoid the error:
```python
augmented_dataset_cln = dataset_cln['train'].map(augment_dataset, features=dataset_cln['train'].features)
```
Issue: Stream data concatenation issue

Comment:
Hi! 
I have run into the same problem with you. Could you please let me know how you solve it? Thanks!
Issue: load json dataset interrupt when dtype cast problem occured

Comment:
Indeed the JSON dataset builder raises an error when it encounters an unexpected type.

There's an old PR open to add away to ignore such elements though, if it can help: https://github.com/huggingface/datasets/pull/2838
Issue: FileNotFound even though exists

Comment:
Hi! 

This would only work if `bigscience/xP3` was a no-code dataset, but it isn't (it has a Python builder script).

But this should work: 
```python
load_dataset("json", data_files="https://huggingface.co/datasets/bigscience/xP3/resolve/main/ur/xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl")
```
Issue: FileNotFound even though exists

Comment:
> I see, it's not compatible w/ regex right? e.g. `load_dataset("json", data_files="https://huggingface.co/datasets/bigscience/xP3/resolve/main/ur/*")`

It should work for patterns that "reference" the local filesystem, but to make this work with the Hub, we must implement https://github.com/huggingface/datasets/issues/5281 first.

In the meantime, you can fetch these glob files with `HfFileSystem` and pass them as a list to `load_dataset`:
```python
from datasets import load_dataset
from huggingface_hub import HfFileSystem, hf_hub_url # `HfFileSystem` requires the latest version of `huggingface_hub`

fs = HfFileSystem()
glob_files = fs.glob("datasets/bigscience/xP3/ur/*")
# convert fsspec URLs to HTTP URLs
resolved_paths = [fs.resolve_path(file) for file in glob_files]
data_files = [hf_hub_url(resolved_path.repo_id, resolved_path.path_in_repo, repo_type=resolved_path.repo_type) for resolved_path in resolved_paths]

ds = load_dataset("json", data_files=data_files)
```
Issue: Incomplete docstring for `BuilderConfig`

Comment:
Thanks for reporting! You are more than welcome to improve `BuilderConfig`'s docstring.

This class serves an identical purpose as `tensorflow_datasets`'s `BuilderConfig`, and its docstring is [here](https://github.com/tensorflow/datasets/blob/a95e38b5bb018312c3d3720619c2a8ef83ebf57f/tensorflow_datasets/core/dataset_builder.py#L81), so feel free to re-use parts of it.
Issue: Cannot pickle error in Dataset.from_generator()

Comment:
Hi! It should work if you put `model = torch.compile(model)` inside the `generate_data` function. If a referenced object is outside, it needs to be pickable, and that's not the case for the compiled models (or functions).