SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Download only split data

Comment:
Hi @capsabogdan! Unfortunately, it's hard to implement because quite often datasets data is being hosted in a single archive for all splits :( So we have to download the whole archive to split it into splits. This is the case for CommonVoice too. 

However, for cases when data is distributed in separate archives ащк different splits I suppose it can (and will) be implemented someday. 


Btw for quick check of the dataset you can use [streaming](https://huggingface.co/docs/datasets/stream):
```python
cv = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)
cv = iter(cv)
print(next(cv))

>> {'client_id': 'a07b17f8234ded5e847443ea6f423cef745cbbc7537fb637d58326000aa751e829a21c4fd0a35fc17fb833aa7e95ebafce5efd19beeb8d843887b85e4eb35f5b',
>>  'path': None,
>>  'audio': {'path': 'cv-corpus-11.0-2022-09-21/en/clips/common_voice_en_100363.mp3',
>>  'array': array([ 0.0000000e+00,  1.1748125e-14,  1.5450088e-14, ...,
>>          1.3011958e-06, -6.3548953e-08, -9.9098514e-08], dtype=float32),
>> ...}

```
Issue: Download only split data

Comment:
thank you for the answer but am not sure if this will not be helpful, as we
need maybe just 10% of the datasets for some experiment

can we get just a portion of the dataset with stream?


is there really no solution? :(

Am Di., 15. Nov. 2022 um 16:55 Uhr schrieb Polina Kazakova <
***@***.***>:

> Hi @capsabogdan <https://github.com/capsabogdan>! Unfortunately, it's
> hard to implement because quite often datasets data is being hosted in a
> single archive for all splits :( So we have to download the whole archive
> to split it into splits. This is the case for CommonVoice too.
>
> However, for cases when data is distributed in separate archives in
> different splits I suppose it can be implemented someday.
>
> Btw for quick check of the dataset you can use streaming
> <https://huggingface.co/docs/datasets/stream>:
>
> cv = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)cv = iter(cv)print(next(cv))
> >> {'client_id': 'a07b17f8234ded5e847443ea6f423cef745cbbc7537fb637d58326000aa751e829a21c4fd0a35fc17fb833aa7e95ebafce5efd19beeb8d843887b85e4eb35f5b',>>  'path': None,>>  'audio': {'path': 'cv-corpus-11.0-2022-09-21/en/clips/common_voice_en_100363.mp3',>>  'array': array([ 0.0000000e+00,  1.1748125e-14,  1.5450088e-14, ...,>>          1.3011958e-06, -6.3548953e-08, -9.9098514e-08], dtype=float32),>> ...}
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5243#issuecomment-1315512887>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOC3JYRCTH54OBRUJULWIOW6PANCNFSM6AAAAAASAYO2LY>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Issue: Download only split data

Comment:
maybe it would be nice if you guys ould do some sort of shard before
loading the dataset, so users can download just chunks of data :)

I think this would be very helpful

Am Di., 15. Nov. 2022 um 19:24 Uhr schrieb Bogdan Capsa <
***@***.***>:

> thank you for the answer but am not sure if this will not be helpful, as
> we need maybe just 10% of the datasets for some experiment
>
> can we get just a portion of the dataset with stream?
>
>
> is there really no solution? :(
>
> Am Di., 15. Nov. 2022 um 16:55 Uhr schrieb Polina Kazakova <
> ***@***.***>:
>
>> Hi @capsabogdan <https://github.com/capsabogdan>! Unfortunately, it's
>> hard to implement because quite often datasets data is being hosted in a
>> single archive for all splits :( So we have to download the whole archive
>> to split it into splits. This is the case for CommonVoice too.
>>
>> However, for cases when data is distributed in separate archives in
>> different splits I suppose it can be implemented someday.
>>
>> Btw for quick check of the dataset you can use streaming
>> <https://huggingface.co/docs/datasets/stream>:
>>
>> cv = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)cv = iter(cv)print(next(cv))
>> >> {'client_id': 'a07b17f8234ded5e847443ea6f423cef745cbbc7537fb637d58326000aa751e829a21c4fd0a35fc17fb833aa7e95ebafce5efd19beeb8d843887b85e4eb35f5b',>>  'path': None,>>  'audio': {'path': 'cv-corpus-11.0-2022-09-21/en/clips/common_voice_en_100363.mp3',>>  'array': array([ 0.0000000e+00,  1.1748125e-14,  1.5450088e-14, ...,>>          1.3011958e-06, -6.3548953e-08, -9.9098514e-08], dtype=float32),>> ...}
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/datasets/issues/5243#issuecomment-1315512887>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ALSIFOC3JYRCTH54OBRUJULWIOW6PANCNFSM6AAAAAASAYO2LY>
>> .
>> You are receiving this because you were mentioned.Message ID:
>> ***@***.***>
>>
>
Issue: Download only split data

Comment:
+1 on this feature request - I am running into the same problem, where I only need the test set for a dataset that has a huge training set
Issue: Download only split data

Comment:
Hey, I'm also interested in that as a feature. I'm having the same problem with Common Voice 13.0. The dataset is super big but I only want the test data to benchmark multilingual models, but I don't have much Terabytes to store all the dataset...
Issue: Incompatible dill versions in datasets 2.6.1

Comment:
Thanks for reporting, @vinaykakade.

We are discussing about making a release early this week.

Please note that in the meantime, in your specific case (as we also pointed out here: https://github.com/huggingface/datasets/issues/5162#issuecomment-1291720293), you can circumvent the issue by pinning `multiprocess` to 0.70.13 version (instead of using latest 0.70.14).

Duplicate of:
- https://github.com/huggingface/datasets/issues/5162
Issue: Incompatible dill versions in datasets 2.6.1

Comment:
You can also make `pip-compile` work by using the backtracking resolver (instead of the legacy one): https://pip-tools.readthedocs.io/en/latest/#a-note-on-resolvers
```
pip-compile --resolver=backtracking requirements.in
```
This resolver will automatically use `multiprocess` 0.70.13 version.
Issue: Using `set_format(type='torch', columns=columns)` makes Array2D/3D columns stop formatting correctly

Comment:
In case others find this, the problem was not with set_format, but my usages of `to_pandas()` and `from_pandas()` which I was using during dataset splitting; somewhere in the chain of converting to and from pandas the `Array2D/Array3D` types get converted to series of `Sequence()` types
Issue: dataclasses error when importing the library in python 3.11

Comment:
I opened [this issue](https://github.com/python/cpython/issues/99401).
Python's maintainers say that the issue is caused by [this change](https://docs.python.org/3.11/whatsnew/3.11.html#dataclasses).
I believe adding a `__hash__` method to `datasets.utils.version.Version` should solve (at least partially) this issue.
Issue: dataclasses error when importing the library in python 3.11

Comment:
Has this been fixed? I am running into this issue now. 

If this has been fixed, could have a new release with this?
Issue: dataclasses error when importing the library in python 3.11

Comment:
Hi, I am getting error while training 

(tensorflow) C:\tensorflow\models\research\object_detection>python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
Traceback (most recent call last):
  File "C:\tensorflow\models\research\object_detection\train.py", line 54, in <module>
    from object_detection.legacy import trainer
  File "C:\tensorflow\models\research\object_detection\legacy\trainer.py", line 27, in <module>
    from object_detection.builders import optimizer_builder
  File "C:\tensorflow\models\research\object_detection\builders\optimizer_builder.py", line 25, in <module>
    from official.modeling.optimization import ema_optimizer
  File "C:\tensorflow\models\official\modeling\optimization\__init__.py", line 19, in <module>
    from official.modeling.optimization.configs.optimization_config import *
  File "C:\tensorflow\models\official\modeling\optimization\configs\optimization_config.py", line 31, in <module>
    @dataclasses.dataclass
     ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x0133252\AppData\Local\anaconda3\envs\tensorflow\Lib\dataclasses.py", line 1223, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "C:\Users\x0133252\AppData\Local\anaconda3\envs\tensorflow\Lib\dataclasses.py", line 1213, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x0133252\AppData\Local\anaconda3\envs\tensorflow\Lib\dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x0133252\AppData\Local\anaconda3\envs\tensorflow\Lib\dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'official.modeling.optimization.configs.optimizer_config.SGDConfig'> for field sgd is not allowed: use default_factory
Issue: dataclasses error when importing the library in python 3.11

Comment:
@Jayanth1812 and anyone else receiving a similar issue, it most likely has to do with your Python version. Downgrading to Python 3.9 works for me, but doing a downgrade might impact a lot of things. So to be safe and what worked for me was creating a new conda environment and following the installations here: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/install.html

And for Tensorflow GPU compatibility, after installing TensorFlow follow the instructions in section 4 'GPU Setup' in this document: https://www.tensorflow.org/install/pip
Issue: dataclasses error when importing the library in python 3.11

Comment:
@Jayanth1812, you can see in your error stack trace, that the error is caused by the `tensorflow` library, not by the `datasets` library. See:
```
File "C:\Users\x0133252\AppData\Local\anaconda3\envs\tensorflow\Lib\dataclasses.py"
```

You should open an issue in their repository instead: https://github.com/tensorflow/tensorflow
Issue: Type error when calling `map` over dataset containing 0-d tensors

Comment:
Hi! 

We could address this by calling `.item()` on such tensors to extract the value, but this would lose us the type, which could lead to storing the generated dataset in a suboptimal format. Considering this, I think the only proper fix would be implementing support for 0-D tensors on Apache Arrow's side (Arrow is the underlying format we use to store datasets on disk/in memory). WDYT @lhoestq?
Issue: Type error when calling `map` over dataset containing 0-d tensors

Comment:
I think we can just convert the item to a numpy typed scalar using `.numpy()` ?

For example this works:
```python
import numpy as np
import pyarrow as pa

assert pa.array([np.float64(1.0)]).type == pa.float64()
assert pa.array([np.float32(1.0)]).type == pa.float32()
assert pa.array([np.int32(1)]).type == pa.int32()
assert pa.array([np.int64(1)]).type == pa.int64()
```

And therefore it would work the same as for PyTorch N-D Tensors: convert to Numpy Array to keep the type in `_cast_to_python_objects`, then convert to Arrow
Issue: Loading a dataset from the hub fails if you happen to have a folder of the same name

Comment:
`load_dataset` first checks for a local directory before checking for the Hub.

To make it explicit that it has to fetch the Hub, we could support the `hffs` syntax:
```python
load_dataset("hf://datasets/glue")
```

would that work for you ? Also cc @mariosasko who's leading the `hffs` project
Issue: Loading a dataset from the hub fails if you happen to have a folder of the same name

Comment:
This still has no proper solution in 2.11

perhaps have a `download_config="force_remote"` or just backtrack once you reach `EmptyDatasetError` locally and then try to load it from the hub (or a local cache, as that only gets checked if there is no local folder...?)
Issue: Q: Memory release when removing the column?

Comment:
Hi ! Datasets are memory mapped from your disk, i.e. they're not loaded in RAM. This is possible thanks to the Arrow data format.

Therefore the column you remove is not in RAM, so removing it doesn't cause the RAM to decrease.
Issue: Q: Memory release when removing the column?

Comment:
Thanks for the explanation! @lhoestq 
I wonder since it is memory mapped, can we reduce or remove this memory map?
Issue: Add video feature

Comment:
@NielsRogge @rwightman may have additional requirements regarding this feature.

When adding a new (decodable) type, the hardest part is choosing the right decoding library. What I mean by "right" here is that it has all the features we need and is easy to install (with GPU support?).

Some candidates/options:
* [`decord`](https://github.com/dmlc/decord): no longer [maintained](https://github.com/dmlc/decord/issues/214), not trivial to install with GPU support
* [`pyAV`](https://github.com/PyAV-Org/PyAV):  used for CPU decoding in `torchvision`, GPU decoding not supported if I'm not mistaken, otherwise the best candidate probably
* [`video_reader`](https://github.com/pytorch/vision/blob/de350bc01ad2193ea2888f0ce8a6a346d3cba5a9/torchvision/csrc/io/video_reader/video_reader.cpp): used for GPU decoding in `torchvision`, depends on `torch'
* OpenCV: uses `ffmpeg` for video decoding under the hood
* ...

And the last resort is building our own library, which is the most flexible solution but also requires the most work.

PS: I'm adding a link to an article that compares various video decoding libraries: https://towardsdatascience.com/lightning-fast-video-reading-in-python-c1438771c4e6
Issue: Add video feature

Comment:
@mariosasko is GPU decoding a hard requirement here? Do we really need it? (I don't know)

Something to consider with `decord` is that it doesn't (AFAIK) support writing videos, so you'd still need something else for that. also I've noticed [issues](https://github.com/dmlc/decord/issues/242) with decord's ability to decode stereo audio streams along side the video (which you don't run into with PyAV).

---

I think PyAV should be able to do the job just fine to start. If we write the video io utilities as their own functions, we can hot swap them later if we find/write a different solution that's faster/better.
Issue: Add video feature

Comment:
Video is still a bit of a mess, but I'd say pyAV is likely the best approach (or supporting all three via pytorchvideo, but that adds a middle man dependency).

Being able to decode on the GPU, into memory that could be passed off to a Tensor in whatever framework is being used would be the dream, I don't think there is any interop of that nature working right now. Number of decoder instances per GPU is limited so it's not clear if balancing load btw GPU decoders and CPUs would be needed in say large scale video training.

Any of these solutions is less than ideal due to the nature of video, having a simple Python interface video / start -> end results in lots of extra memory (you need to decode whole range of the clips into a buffer before using anything). Any scalable video system would be streaming on the fly (issuing frames via callbacks as soon as the stream is far enough along to have re-ordered the frames and synced audio+video+other metadata (sensors, CC, etc).
Issue: Add video feature

Comment:
For standalone usage, decoding on GPU could be ideal but isn't async processing of inputs on CPUs while letting the accelerator busy for training the de-facto? Of course, I am aware of other advanced mechanisms such as CPU offloading, but I think my point is conveyed.
Issue: Add video feature

Comment:
Here's a minimal implementation of the helper functions we'd need from PyAV, a lot of which I borrowed from `pytorchvideo`, stripping out the `torch` specific stuff:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/c327cb6ff6b074e6ddc8068d19c0367d/pyav-io.ipynb)
    
It's not too much code...@mariosasko we could probably just maintain these helper fns within the `datasets` library, right?
Issue: Add video feature

Comment:
Also wanted to note I added a PR for video classification in `transformers` here, which uses `decord`. It's still open...should we make a decision now to align the libraries we are using between `datasets` and `transformers`? (CC @Narsil )

https://github.com/huggingface/transformers/pull/20151
Issue: Add video feature

Comment:
Fully agree on at least trying to unite things.

Making clear function boundaries to help us change dependency if needed seems like a good idea since there doesn't seem to be a clear winner.

I also happen to like directly calling ffmpeg. For some reason it was a lot faster than pyav.
Issue: Seems to freeze when loading audio dataset with wav files from local folder

Comment:
I just tried to do the same but changing the `.wav` files to `.mp3` files and that doesn't fix it.
Issue: Seems to freeze when loading audio dataset with wav files from local folder

Comment:
I don't know if anyone will ever read this but I've tried to upload the same dataset with google colab and the output seems more clarifying. I didn't specify the train/test split so the dataset wasn't fully uploaded (or that is what I understood, might be wrong!!).

Now, including the `drop_metadata` flag I can load the dataset normally (at least with colab notebook):

```python
from datasets import load_dataset

dataset = load_dataset("audiofolder", data_dir="../archive/Dataset", , drop_metadata=True)
```

I'll close the issue.
Issue: Seems to freeze when loading audio dataset with wav files from local folder

Comment:
Yes but it didn't matter. I don't remember which of them I ended up working with.
Issue: HuggingFace website is incorrectly reporting that my datasets are pickled

Comment:
Yes I think I know what is happening. We check in zips for pickles, and the UI must display the pickle jar when a scan has an associated list of imports, even when empty.
~I'll fix ASAP !~
Issue: HuggingFace website is incorrectly reporting that my datasets are pickled

Comment:
>  I'll fix ASAP !

Actually I'd rather leave it like that for now, as it indicates that we checked for pickles and nothing dangerous appeared :)
Issue: Cannot push 

Comment:
Did you run `huggingface-cli lfs-enable-largefiles` before committing or before adding ? Maybe you can try before adding

Anyway I'd encourage you to split your data into several TAR archives if possible, this way the dataset can loaded faster using multiprocessing (by giving each process a subset of shards to process)
Issue: Cannot push 

Comment:
@lhoestq 
Thanks for the help!
> Maybe you can try before adding

It did not help

But I totally got your point about split into multiple TAR archives. It really helped!
Issue: Implicit type conversion of lists in to_pandas

Comment:
I think this behavior comes from PyArrow:
```python
import pyarrow as pa
t = pa.table({"a": [[0]]})
t.to_pandas().a.values[0]
# array([0])
```

I believe this has to do with zero-copy: you can get a pandas DataFrame without copying the buffers from arrow, and therefore end up with numpy arrays.
Issue: Delta Tables usage using Datasets Library

Comment:
Hi ! Interesting :) Can you provide concrete examples of cases where it can be useful ?
Issue: Delta Tables usage using Datasets Library

Comment:
Few example blogs and posts that might help on this - 

1. https://hevodata.com/learn/databricks-delta-tables/
2. https://docs.databricks.com/delta/index.html

Basically, we are looking at utility of Datasets library with Delta Lake Tables.
Issue: Delta Tables usage using Datasets Library

Comment:
`datasets` can already read/write from parquet from/to a cloud storage using fsspec, if I understand correctly it's should be possible to load parquet files as delat lake tables no ? :) Or is there someting missing ?
Issue: Delta Tables usage using Datasets Library

Comment:
@lhoestq Per my understanding, delta lake table is a bunch of paruqet files together with the meta to support ACID. For example file 1 contains v0.1 of record A while file 2 contains v0.2 of record A. I am assuming the Hugging face dataset would delegate the read/write delta table to 3rd party lib, maybe pyarrow. Correct me if I was wrong @reichenbch 

And I am assuming, people are asking the versioning of Hugging face datasets. But I am assuming Hugging face delegate this function to github and it is not the key requirement for Public Data set. It actually the key function of ML Ops, I am not sure whether hugging face would like expand to that area.
Issue: save_elasticsearch_index

Comment:
Hi ! I think there exist tools to dump and reload an index in your elastic search but I'm not super familiar with it.

Anyway after reloading an index in elastic search you can call `ds.load_elasticsearch_index` which will connect the index to the dataset without re-indexing
Issue: Implement ability to define splits in metadata section of dataset card

Comment:
@merveenoyan Do you want different files to be splits or configurations?

From [what you specified in `Readme.md`](https://huggingface.co/datasets/inria-soda/tabular-benchmark/commit/fb4575853772c62a20203bdd6cc0202f5db4ce4e) I hypothesize that you want to have 4 **configs** corresponding to directories: `"clf_cat", "clf_num", "reg_cat", "reg_num"`. And inside each config you require to have as many splits as there are `csv` files
so if you run 
```python
load_dataset("inria-soda/tabular-benchmark", "clf_cat", split="compass")
```
you will generate the data only from `compass.csv` file.
In this case, running `load_dataset("inria-soda/tabular-benchmark", "clf_cat"`) without split parameter will return `DatasetDict` object with `"KDDCup09_upselling", "cat_compass", "cat_covertype", ... "road_safety"` keys (which values are splits - `Dataset` objects)

**or**
do you want each file to be a separate config? Like:
```python
load_dataset("inria-soda/tabular-benchmark", "clf_cat_compass")   # returns DatasetDict with a single "train" split
```
**or**
maybe smth completely different? :smile:  

Anyway, now I have an impression that this is probably rather a matter of automatically inferring configs from repository structure rather than providing parameters in metadata yaml.
Issue: Implement ability to define splits in metadata section of dataset card

Comment:
@polinaeterna I want the latter where you can think of every CSV file as a config, like MNLI from GLUE.