SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Dataset Viewer issue for gsarti/flores_101

Comment:
I'm just wondering why we don't have this dataset under:
- the `facebook` namespace
- or the canonical dataset `flores`: why does this only have 2 languages?
Issue: Dataset sharding non-contiguous?

Comment:
This was silly; I was sure I'd looked for a `contiguous` argument, and was certain there wasn't one the first time I looked :smile:

Sorry about that.
Issue: Dataset sharding non-contiguous?

Comment:
Hi! You can pass `contiguous=True` to `.shard()` get contiguous shards. More info on this and the default behavior can be found in the [docs](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.shard).

EDIT: Answered as you closed the thread 😄
Issue: Dataset sharding non-contiguous?

Comment:
Hahaha I'm sorry; my excuse is: it's Sunday. (Which makes me all the more grateful for your response :smiley:
Issue: Dataset sharding non-contiguous?

Comment:
@mariosasko Sorry for reviving this, but I was curious as to why `contiguous=False` was the default. This might be a personal bias, but I feel that a user would expect the opposite to be the default. :thinking:
Issue: Dataset Viewer issue for sst2

Comment:
Hi @lewtun, thanks for reporting.

I have checked locally and refreshed the preview and it seems working smooth now:
```python
In [8]: ds
Out[8]: 
DatasetDict({
    train: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 872
    })
    test: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 1821
    })
})
```

Could you confirm?
Issue: Dataset Viewer issue for sst2

Comment:
Thanks @albertvillanova - it is indeed working now (not sure what caused the error in the first place). Closing this :)
Issue: XNLI cache reload is very slow

Comment:
Hi,
Could you tell us how you are running this code?
I tested on my machine (M1 Mac). And it is running fine both on and off internet.

<img width="1033" alt="Screen Shot 2022-07-03 at 1 32 25 AM" src="https://user-images.githubusercontent.com/8711912/177026364-4ad7cedb-e524-4513-97f7-7961bbb34c90.png">
Tested on both stable and dev version.
Issue: XNLI cache reload is very slow

Comment:
Sure, I was running it on a Linux machine.
I found that if I turn the Internet off, it would still try to make a HTTPS call which would slow down the cache loading. If you can't reproduce then we can close the issue.
Issue: XNLI cache reload is very slow

Comment:
Hi @Muennighoff! You can set the env variable `HF_DATASETS_OFFLINE` to `1` to avoid this behavior in offline mode. More info is available [here](https://huggingface.co/docs/datasets/master/en/loading#offline).
Issue: Document link #load_dataset_enhancing_performance points to nowhere

Comment:
Hi! This is indeed the link the docstring should point to. Are you interested in submitting a PR to fix this?
Issue: Document link #load_dataset_enhancing_performance points to nowhere

Comment:
https://github.com/huggingface/datasets/blame/master/docs/source/cache.mdx#L93

There seems already an anchor here. Somehow it doesn't work. I am not very familiar with how this online documentation works.
Issue: Dataset Viewer issue for allocine

Comment:
Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content *sequentially* (no random access).
Issue: Dataset Viewer issue for allocine

Comment:
> Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content _sequentially_ (no random access).

Ah thanks for the clarification! I'll look out for this next time and implement the fix myself :)
Issue: imdb source error

Comment:
Thanks for reporting, @Muhtasham.

Indeed IMDB dataset is not accessible from yesterday, because the data is hosted on the data owners servers at Stanford (http://ai.stanford.edu/) and these are down due to a power outage originated by a fire: https://twitter.com/StanfordAILab/status/1539472302399623170?s=20&t=1HU1hrtaXprtn14U61P55w

As a temporary workaroud, you can load the IMDB dataset with this tweak:
```python
ds = load_dataset("imdb", revision="tmp-fix-imdb")
```
Issue: Metadata.jsonl for Imagefolder is ignored if it's in a parent directory to the splits directories/do not have "{split}_" prefix

Comment:
I agree it would be nice to support this. It doesn't fit really well in the current data_files.py, where files of each splits are separated in different folder though, maybe we have to modify a bit the logic here. 

One idea would be to extend `get_patterns_in_dataset_repository` and `get_patterns_locally` to additionally check for `metadata.json`, but feel free to comment if you have better ideas (I feel like we're reaching the limits of what the current implementation IMO, so we could think of a different way of resolving the data files if necessary)
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
This has so much potential to be great! Also I think you tagged some poor random dude on the internet whose name is also Joao, lol, edited that for you!
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Noted and I will look into the thread in detail tomorrow once I log back in.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
@lhoestq I have used TFRecords with `tf.data` for both vision and text and I can say that they are quite performant. I haven't worked with Feather yet as similarly as I have with TFRecords. If you haven't started the benchmarking script yet, I can prepare a Colab notebook that loads Feather files, converts them into a `tf.data` pipeline, and does some basic preprocessing. 

But in my limited understanding, Feather might be better suited for CSV files. Not yet sure if it's good for modalities like images.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
> Not yet sure if it's good for modalities like images.

We store images pretty much the same way as tensorflow_datasets (i.e. storing the encoded image bytes, or a path to the local image, so that the image can be decoded on-the-fly), so as long as we use something similar as TFDS for image decoding it should be ok
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
So for image datasets, we could potentially store the paths in the feather format and decode and read them on the fly? But it introduces an I/O redundancy of having to read the images every time.

With caching it could be somewhat mitigated but it's not a good solution for bigger image datasets.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
> So for image datasets, we could potentially store the paths in the feather format and decode and read them on the fly?

hopefully yes :) 

I double-checked the TFDS source code and they always save the bytes actually, not the path. Anyway we'll see if we run into issues or not (as a first step we can require the bytes to be in the feather file)
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Yes. For images, TFDS actually prepares TFRecords first for encoding and then reuses them for every subsequent call.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
@lhoestq @Rocketknight1 I worked on [this PoC](https://gist.github.com/sayakpaul/f7d5cc312cd01cb31098fad3fd9c6b59) that

* Creates Feather files from a medium resolution dataset (`tf_flowers`).
* Explores different options with TensorFlow IO to load the Feather files. 

I haven't benchmarked those different options yet. There's also a gotcha that I have noted in the PoC. I hope it gets us started but I'm sorry if this is redundant.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Cool thanks ! If I understand correctly in your PoC you store the flattened array of pixels in the feather file. This will take a lot of disk space.

Maybe we could just save the encoded bytes and let users apply a `map` to decode/transform them into the format they need for training ? Users can use tf.image to do so for example
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
@lhoestq I think I was able to make it work in the way you were envisioning. Here's the PoC:
https://gist.github.com/sayakpaul/f7d5cc312cd01cb31098fad3fd9c6b59#file-feather-tf-poc-bytes-ipynb

Some details:

* I am currently serializing the images as strings with `base64`). In comparison to the flattened arrays as before, the size of the individual feather files has reduced (144 MB -> 85 MB, largest).
* When decoding, I am first decoding the base64 string and then decoding that string (with `tf.io.decode_base64`) as an image with `tf.image.decode_png()`. 
* The entire workflow (from generating the Feather files to loading them and preparing the batched `tf.data` pipeline) involves the following libraries: `pyarraow`, `tensorflow-io`, and `tensorflow`. 

Cc: @Rocketknight1 @gante
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Cool thanks ! Too bad the Arrow binary type doesn't seem to be supported in `arrow_io.ArrowFeatherDataset` :/ We would also need it to support Arrow struct type. Indeed images in `datasets` are represented using an Arrow type
```python
pa.struct({"path": pa.string(), "bytes": pa.binary()})
```
not sure yet how hard it is to support this though.

Changing the typing on our side would create concerning breaking changes, that's why it would be awesome if it could work using these types
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
If the ArrowFeatherDataset doesn't yet support it, I guess our hands are a bit tied at the moment. 

IIUC, in my [latest PoC notebook](https://gist.github.com/sayakpaul/f7d5cc312cd01cb31098fad3fd9c6b59#file-feather-tf-poc-bytes-ipynb), you wanted to see each entry in the feather file to be represented like so?

```
pa.struct({"path": pa.string(), "bytes": pa.binary()})
``` 

In that case, `pa.binary()` isn't yet supported.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
> IIUC, in my [latest PoC notebook](https://gist.github.com/sayakpaul/f7d5cc312cd01cb31098fad3fd9c6b59#file-feather-tf-poc-bytes-ipynb), you wanted to see each entry in the feather file to be represented like so?
> 
> pa.struct({"path": pa.string(), "bytes": pa.binary()})

Yea because that's the data format we're using. If we were to use base64, then we would have to process the full dataset to convert it, which can take some time. Converting to TFRecords would be simpler than converting to base64 in Feather files.

Maybe it would take too much time to be worth exploring, but according to https://github.com/tensorflow/io/issues/1361#issuecomment-819029002 it's possible to add support for binary type in ArrowFeatherDataset. What do you think ? Any other alternative in mind ?
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
> Maybe it would take too much time to be worth exploring, but according to https://github.com/tensorflow/io/issues/1361#issuecomment-819029002 it's possible to add support for binary type in ArrowFeatherDataset.

Should be possible as per the comment but there hasn't been any progress and it's been more than a year. 

>  If we were to use base64, then we would have to process the full dataset to convert it, which can take some time.

I don't understand this. I would think TFRecords would also need something similar but I need the context you're coming from. 

> What do you think ? Any other alternative in mind ?

TFRecords since the TensorFlow ecosystem has developed good support for it over the years.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
> I don't understand this. I would think TFRecords would also need something similar but I need the context you're coming from.

Users already have a copy of the dataset in Arrow format (we can change this to Feather). So to load the Arrow/feather files to a TF dataset we need TF IO or something like that. Otherwise the user has to convert all the files from Arrow to TFRecords to use TF data efficiently. But the conversion needs resources: CPU, disk, time. Converting the images to base64 require the same sort of resources.

So the issue we're trying to tackle is how to load the Arrow data in TF without having to convert anything ^^
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Yeah, it looks like in its current state the tfio support for `Feather` is incomplete, so we'd end up having to write a lot of it, or do a conversion that defeats the whole point (because if we're going to convert the whole dataset we might as well convert to `TFRecord`).
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
@lhoestq Although I think this is a dead-end for now unfortunately, because of the limitations at TF's end, we could still explore automatic conversion to TFRecord, or I could dive into refining `to_tf_dataset()` to yield unbatched samples and/or load samples with multiprocessing to improve throughput. Do you have any preferences there?
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
> @lhoestq Although I think this is a dead-end for now unfortunately, because of the limitations at TF's end, we could still explore automatic conversion to TFRecord, or I could dive into refining `to_tf_dataset()` to yield unbatched samples and/or load samples with multiprocessing to improve throughput. Do you have any preferences there?

Happy to take part there @Rocketknight1.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
If `to_tf_dataset` can be unbatched, then it should be fairly easy for users to convert the TF dataset to TFRecords right ?
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
> If to_tf_dataset can be unbatched, then it should be fairly easy for users to convert the TF dataset to TFRecords right ?

Sort of! A `tf.data.Dataset` is more like an iterator, and does not support sample indexing. `to_tf_dataset()` creates an iterator, but to convert that to `TFRecord`, the user would have to iterate over the whole thing and manually save the stream of samples to files.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Someone would like to try to dive into tfio to fix this ? Sounds like a good opportunity to learn what are the best ways to load a dataset for TF, and also the connections between Arrow and TF.

If we can at least have the Arrow `binary` type working for TF that would be awesome already (issue https://github.com/tensorflow/io/issues/1361)

also cc @nateraw in case you'd be interested ;)
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
> Sounds like a good opportunity to learn what are the best ways to load a dataset for TF

The recommended way would likely be a combination of TFRecords and `tf.data`. 

Exploring the connection between Arrow and TensorFlow is definitely worth pursuing though. But I am not sure about the implications of storing images in a format supported by Arrow. I guess we'll know more once we have at least figured out the support for `binary` type for TFIO. I will spend some time on it and keep this thread updated.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
I am currently working on a fine-tuning notebook for the TFSegFormer model (Semantic Segmentation). The resolution is high for both the input images and the labels - (512, 512, 3). Here's the [Colab Notebook](https://colab.research.google.com/drive/1jAtR7Z0lYX6m6JsDI5VByh5vFaNhHIbP?usp=sharing) (it's a WIP so please bear that in mind).

I think the current implementation of `to_tf_dataset()` does create a bottleneck here since the GPU utilization is quite low.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Here's a notebook showing the performance difference: https://colab.research.google.com/gist/sayakpaul/d7ca67c90beb47e354942c9d8c0bd8ef/scratchpad.ipynb. 

Note that I acknowledge that it's not an apples-to-apples comparison in many aspects (the dataset isn't the same, data serialization format isn't the same, etc.) but this is the best I could do.
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Thanks ! I think the speed difference can be partly explained: you use ds.shuffle in your dataset, which is an exact shuffling (compared to TFDS which does buffer shuffling): it slows down query time by 2x to 10x since it has to play with data that are not contiguous.

The rest of the speed difference seems to be caused by image decoding (from 330µs/image to 30ms/image)
Issue: [to_tf_dataset] Use Feather for better compatibility with TensorFlow ?

Comment:
Fair enough. Can do one without shuffling too. But it's an important one to consider I guess.
Issue: Avoid splitting by` .py` for the file.

Comment:
Hi @espoirMur, thanks for reporting.

You are right: that code line could be improved and made more generically valid.

On the other hand, I would suggest using `os.path.splitext` instead.

Are you willing to open a PR? :)
Issue: Dataset Viewer issue for Pile of Law

Comment:
Thanks so much for adding the docs. I was able to successfully hide the viewer using the 
```
viewer: false
```
flag in the README.md of the dataset. I'm closing the issue because this is resolved. Thanks again!
Issue: Dataset Viewer issue for Pile of Law

Comment:
Just for the record:

- the doc
  
<img width="1430" alt="Capture d’écran 2022-06-27 à 09 29 27" src="https://user-images.githubusercontent.com/1676121/175884089-bca6c0d5-6387-473e-98ca-86a910ede4bd.png">

- the dataset main page

<img width="1134" alt="Capture d’écran 2022-06-27 à 09 29 05" src="https://user-images.githubusercontent.com/1676121/175884152-5f285bf0-3471-45de-927a-e141b00ebb33.png">

- the dataset viewer page

<img width="567" alt="Capture d’écran 2022-06-27 à 09 29 16" src="https://user-images.githubusercontent.com/1676121/175884191-ab6a297b-1c11-417e-bbde-0b7623278a79.png">
Issue: Dataset Viewer issue for CSV datasets

Comment:
Confirmed, it's fixed now. Thanks for reporting, and thanks @coyotte508 for fixing it

<img width="1123" alt="Capture d’écran 2022-06-21 à 10 28 05" src="https://user-images.githubusercontent.com/1676121/174753833-1b453a5a-6a90-4717-bca1-1b5fc6b75e4a.png">
Issue: Ecoset

Comment:
Hi! Very cool dataset! I answered your questions on the forum. Also, feel free to comment `#self-assign` on this issue to self-assign it.
Issue: Memory leak when iterating a Dataset

Comment:
Here is a modified repro example that makes it easier to see the leak:

```
$ cat ds2.py
import gc, sys
import time
from datasets import load_dataset
import os, psutil

process = psutil.Process(os.getpid())

print(process.memory_info().rss/2**20)

corpus = load_dataset("BeIR/msmarco", 'corpus', keep_in_memory=False, streaming=False)['corpus']
corpus = corpus.select(range(200000))

print(process.memory_info().rss/2**20)

batch = None

mem_before_start = psutil.Process(os.getpid()).memory_info().rss / 2**20

step = 20000
for i in range(0, 10*step, step):
    mem_before = psutil.Process(os.getpid()).memory_info().rss / 2**20
    batch = corpus[i:i+step]
    import objgraph
    #objgraph.show_refs([batch])
    #objgraph.show_refs([corpus])
    #sys.exit()
    gc.collect()

    mem_after = psutil.Process(os.getpid()).memory_info().rss / 2**20
    print(f"{i:6d} {mem_after - mem_before:12.4f} {mem_after - mem_before_start:12.4f}")

```

Let's run:

```
$ python ds2.py
     0      36.5391      36.5391
 20000      10.4609      47.0000
 40000       5.9766      52.9766
 60000       7.8906      60.8672
 80000       6.0586      66.9258
100000       8.4453      75.3711
120000       6.7422      82.1133
140000       8.5664      90.6797
160000       5.7344      96.4141
180000       8.3398     104.7539
```

You can see the last column of total RSS memory keeps on growing in MBs. The mid column is by how much it was grown during a single iteration of the repro script (20000 items)
Issue: Memory leak when iterating a Dataset

Comment:
@NouamaneTazi, please check my analysis here https://github.com/huggingface/datasets/issues/4883#issuecomment-1242599722 so if you agree with my research this Issue can be closed as well.

I also made a suggestion at how to proceed to hunt for a real leak here https://github.com/huggingface/datasets/issues/4883#issuecomment-1242600626

you may find this one to be useful as well https://github.com/huggingface/datasets/issues/4883#issuecomment-1242597966
Issue: Memory leak when iterating a Dataset

Comment:
Amazing job! Thanks for taking time to debug this 🤗

For my side, I tried to do some more research as well, but to no avail. https://github.com/huggingface/datasets/issues/4883#issuecomment-1243415957
Issue: split cache used when processing different split

Comment:
I was not able to reproduce this behavior (I tried without using pytorch lightning though, since I don't know what code you ran in pytorch lightning to get this).

If you can provide a MWE that would be perfect ! :)
Issue: split cache used when processing different split

Comment:
Hi, I think the issue happened because I was loading datasets under an `if` ... `else` statement and the condition would change the dataset I would need to load but instead the cached one was always returned. However, I believe that is expected behaviour, if so I'll close the issue.

Otherwise I will try to provide a MWE
Issue: Out of memory error on workers while running Beam+Dataflow

Comment:
Some naive ideas to cope with this:
- enable more RAM on each worker
- force the spanning of more workers
- others?
Issue: Out of memory error on workers while running Beam+Dataflow

Comment:
@albertvillanova We were finally able to process the full NQ dataset on our machines using 600 gb with 5 workers. Maybe these numbers will work for you as well.
Issue: Out of memory error on workers while running Beam+Dataflow

Comment:
Thanks a lot for the hint, @seirasto.

I have one question: what runner did you use? Direct, Apache Flink/Nemo/Samza/Spark, Google Dataflow...? Thank you.
Issue: Out of memory error on workers while running Beam+Dataflow

Comment:
@albertvillanova  Since we have already processed the NQ dataset on our machines can we upload it to datasets so the NQ PR can be merged?
Issue: Out of memory error on workers while running Beam+Dataflow

Comment:
Maybe @lhoestq can give a more accurate answer as I am not sure about the authentication requirements to upload those files to our cloud bucket.

Anyway I propose to continue this discussion on the dedicated PR for Natural questions dataset:
- #4368
Issue: Out of memory error on workers while running Beam+Dataflow

Comment:
> I asked my colleague who ran the code and he said apache beam.

He looked into it further and he just used DirectRunner. @albertvillanova
Issue: Out of memory error on workers while running Beam+Dataflow

Comment:
OK, thank you @seirasto for your hint.

That explains why you did not encounter the out of memory error: this only appears when the processing is distributed (on workers memory) and DirectRunner does not distribute the processing (all is done in a single machine).