SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Automatically revert to last successful commit to hub when a push_to_hub is interrupted

Comment:
> Ideally we would want to upload the files iteratively - and then once everything is uploaded we proceed to commit. 

Oh I see. So maybe this has to be done in an implementation specific to `datasets/` as it is not a very common case (upload a bunch of files on the fly).

You can maybe have a look at how `huggingface_hub` is implemented for LFS files (arrow shards are LFS anyway, right?).
In [`upload_lfs_files`](https://github.com/huggingface/huggingface_hub/blob/e28646c977fc9304a4c3576ce61ff07f9778950b/src/huggingface_hub/_commit_api.py#L164) LFS files are uploaded 1 by 1 (multithreaded) and then [the commit is pushed](https://github.com/huggingface/huggingface_hub/blob/e28646c977fc9304a4c3576ce61ff07f9778950b/src/huggingface_hub/hf_api.py#L1926) to the Hub once all files have been uploaded. This is pretty much what you need, right ?

I can help you if you have questions how to do it in `datasets`. If that makes sense we could then move the implementation from `datasets` to `huggingface_hub` once it's mature. Next week I'm on holidays but feel free to start without my input.

(also cc @coyotte508 and @SBrandeis who implemented LFS upload in `hfh`)
Issue: Automatically revert to last successful commit to hub when a push_to_hub is interrupted

Comment:
> Could you share the error you got please ? Maybe the full stack trace if you have it ?

Here’s part of the stack trace, that I can reproduce at the moment from a photo I took (potential typos from OCR):
```
ValueError
Traceback (most recent call last)
<ipython-input-4-274613b7d3f5> in <module>
from datasets import load dataset
ds = load_dataset('jrahn/chessv6', use_auth_token-True)

/us/local/1ib/python3.7/dist-packages/datasets/table.py in cast_table _to_schema (table, schema)
Line 2005 raise ValueError()

ValueError: Couldn't cast 
fen: string 
move: string 
res: string 
eco: string 
move_id: int64
res_num: int64 to
{ 'fen': Value(dtype='string', id=None), 
'move': Value(dtype=' string', id=None),
'res': Value(dtype='string', id=None),
'eco': Value(dtype='string', id=None), 
'hc': Value(dtype='string', id=None), 
'move_ id': Value(dtype='int64', id=None),
'res_num': Value(dtype= 'int64' , id=None) }
because column names don't match 
```

The column 'hc' was removed before the interrupted push_to_hub(). It appears in the column list in curly brackets but not in the column list above.

Let me know, if I can be of any help.
Issue: integrate `load_from_disk` into `load_dataset` 

Comment:
I agree the situation is not ideal and it would be awesome to use `load_dataset` to reload a dataset saved locally !

For context:

- `load_dataset` works in three steps: download the dataset, then prepare it as an arrow dataset, and finally return a memory mapped arrow dataset. In particular it creates a cache directory to store the arrow data and the subsequent cache files for `map`.

- `load_from_disk` directly returns a memory mapped dataset from the arrow file (similar to `Dataset.from_file`). It doesn't create a cache diretory, instead all the subsequent `map` calls write in the same directory as the original data. 

If we want to keep the download_and_prepare step for consistency, it would unnecessarily copy the arrow data into the datasets cache. On the other hand if we don't do this step, the cache directory doesn't exist which is inconsistent.

I'm curious, what would you expect to happen in this situation ?
Issue: integrate `load_from_disk` into `load_dataset` 

Comment:
Thank you for the detailed breakdown, @lhoestq 

> I'm curious, what would you expect to happen in this situation ?

1. the simplest solution is to add a flag to the dataset saved by `save_to_disk` and have `load_dataset` check that flag - if it's set simply switch control to `load_from_disk` behind the scenes. So `load_dataset` detects it's a local filesystem, looks inside to see whether it's something it can cache or whether it should use it directly as is and continues accordingly with one of the 2 dataset-type specific APIs.

2. the more evolved solution is to look at a dataset produced by `save_to_disk` as a remote resource like hub. So the first time `load_dataset` sees it, it'll take a fingerprint and create a normal cached dataset. On subsequent uses it'll again discover it as a remote resource, validate that it has it cached via the fingerprint and serve as a normal dataset. 

As you said the cons of approach 2 is that if the dataset is huge it'll make 2 copies on the same machine. So it's possible that both approaches can be integrated. Say if `save_to_disc(do_not_cache=True)` is passed it'll use solution 1, otherwise solution 2. or could even symlink the huge arrow files to the cache instead? or perhaps it's more intuitive to use `load_dataset(do_not_cache=True)` instead. So that one can choose whether to make a cached copy or not for the locally saved dataset. i.e. a simple at use point user control.

Surely there are other ways to handle it, this is just one possibility.
Issue: integrate `load_from_disk` into `load_dataset` 

Comment:
I think the simplest is to always memory map the local file without copy, but still have a cached directory in the cache at `~/.cache/huggingface` instead of saving `map` results next to the original data.

In practice we can even use symlinks if it makes the implementation simpler
Issue: integrate `load_from_disk` into `load_dataset` 

Comment:
Yes, so that you always have the cached entry for any dataset, but the "payload" doesn't have to be physically in the cache if it's already on the local filesystem. As you said a symlink will do.
Issue: integrate `load_from_disk` into `load_dataset` 

Comment:
We haven't had the bandwidth to implement this so far. Let me know if you'd be interested in contributing this feature :)
Issue: integrate `load_from_disk` into `load_dataset` 

Comment:
@lhoestq I can jump into that. What I don't like is having functions with many parameters input. Even though they are optional, it's always harder to reason about and test such cases.
If there are more features worth to work on, feel free to ping me. It's a lot of fun to help :smile:
Issue: integrate `load_from_disk` into `load_dataset` 

Comment:
Thanks a lot for your help @mariusz-jachimowicz-83 :)

I think as a first step we could implement an Arrow dataset builder to be able to load and stream Arrow datasets locally or from Hugging Face. Maybe something similar to the Parquet builder at [src/datasets/packaged_modules/parquet/parquet.py](https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/parquet/parquet.py) ?

And we can deal with the disk space optimization as a second step. What do you think ?

(this issue is also related to https://github.com/huggingface/datasets/issues/3035)
Issue: integrate `load_from_disk` into `load_dataset` 

Comment:
@lhoestq Let me know if you have further recommendations or anything that you would like to add but you don't have bandwith for.
Issue: `Dataset.unique` showing wrong output after filtering

Comment:
Hi! It seems like `flatten_indices` (called in `unique`) doesn't know how to handle empty indices mappings. I'm working on the fix.
Issue: new dataset type: single-label and multi-label video classification

Comment:
Hi ! You can in the `features` folder how we implemented the audio and image feature types.

We can have something similar to videos. What we need to decide:
- the video loading library to use
- the output format when a user accesses a video type object
- what parameters a `Video()` feature type needs

also cc @nateraw who also took a look at what we can do for video
Issue: new dataset type: single-label and multi-label video classification

Comment:
Hi ! I think we just missing which lib we're going to use to decode the videos + which parameters must go in the `Video` type
Issue: new dataset type: single-label and multi-label video classification

Comment:
pytorchvideo uses [pyav](https://github.com/PyAV-Org/PyAV) as the default decoder: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/labeled_video_dataset.py#L37

Also it would be great if `optionally` audio can also be decoded from the video as in pytorchvideo: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/labeled_video_dataset.py#L35

Here are the other decoders supported in pytorchvideo: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/encoded_video.py#L17
Issue: new dataset type: single-label and multi-label video classification

Comment:
@sayakpaul I did do quite a bit of work on [this PR](https://github.com/huggingface/datasets/pull/4532) a while back to add a video feature. It's outdated, but uses my `encoded_video` [package](https://github.com/nateraw/encoded-video) under the hood, which is basically a wrapper around PyAV stolen from [pytorchvideo](https://github.com/facebookresearch/pytorchvideo/) that gets rid of the `torch` dependency. 

would be really great to get something like this in...it's just a really tricky and time consuming feature to add.
Issue: passing parameters to the method passed to Dataset.from_generator()

Comment:
Hi! Yes, you can either use the `gen_kwargs` param in `Dataset.from_generator` (`ds = Dataset.from_generator(gen, gen_kwargs={"param1": val})`) or wrap the generator function with `functools.partial`
(`ds = Dataset.from_generator(functools.partial(gen, param1="val"))`) to pass custom parameters to it.
Issue: Custom Json Dataset Throwing Error when batch is False

Comment:
Hi! Our processors are meant to be used in `batched` mode, so if `batched` is `False`, you need to drop the batch dimension (the error message warns you that the array has an extra dimension meaning it's 4D instead of 3D) to avoid the error:
```python
def prepare_examples(examples):
    #Some preporcessing for each image and text as all my data saved in cloud
    #For this reason I couldn't set the batch to True. 
    encoding = processor(img_as_tensor, words, boxes=boxes, word_labels=labels,
                       truncation=True, padding="max_length", return_tensors="np")
    # drop extra dim
    for k in encoding.items():
        encoding[k]=encoding[k][0]
    return encoding
```
Issue: Custom Json Dataset Throwing Error when batch is False

Comment:
> Hi! Our processors are meant to be used in `batched` mode, so if `batched` is `False`, you need to drop the batch dimension (the error message warns you that the array has an extra dimension meaning it's 4D instead of 3D) to avoid the error:
> 
> ```python
> def prepare_examples(examples):
>     #Some preporcessing for each image and text as all my data saved in cloud
>     #For this reason I couldn't set the batch to True. 
>     encoding = processor(img_as_tensor, words, boxes=boxes, word_labels=labels,
>                        truncation=True, padding="max_length", return_tensors="np")
>     # drop extra dim
>     for k in encoding.items():
>         encoding[k]=encoding[k][0]
>     return encoding
> ```

Thank you it did work

```
for k,v in encoding.items():
        encoding[k]=encoding[k][0]
```
Issue: Split is inferred from filename and overrides metadata.jsonl

Comment:
Hi! What's the structure of your image folder? `datasets` by default tries to infer to what split each file belongs based on directory/file names. If it's OK to load all the images inside the `dataset` folder in the `train` split, you can do the following:
```python
dataset = load_dataset("imagefolder", data_files="dataset/**")
```
Issue: Split is inferred from filename and overrides metadata.jsonl

Comment:
Thanks! Specifying `data_files` worked for that case.

I'm new to the library, so let me try rephrasing the issue. If there's no actual bug here, sorry for the trouble.

I've uploaded an example [here](https://files.catbox.moe/nfj2pd.zip) with the following files: 

```
.
├── bug.py
└── imagefolder
   ├── test
   │  ├── metadata.jsonl
   │  ├── dog.jpg
   │  └── personal trainer.jpg
   └── train
      ├── metadata.jsonl
      ├── cat.jpg
      └── testing center.jpg
```

`bug.py`
```
from datasets import load_dataset

dataset = load_dataset("imagefolder")

print(dataset)
# DatasetDict({
#     test: Dataset({
#         features: ['image', 'text'],
#         num_rows: 1
#     })
# })

for split in dataset:
    print("Split:", split)
    for n in dataset[split]:
        print(n['text'])


# Split: test
# testing center
```

As far as I can tell, this conforms with the example given here: https://huggingface.co/docs/datasets/image_dataset#imagefolder. It appears to me that, even though `metadata.jsonl` is present, the inferred labels from the path are taking precedent. Does this sound like a bug/undocumented behavior?
Issue: Split is inferred from filename and overrides metadata.jsonl

Comment:
This looks like a duplicate of https://github.com/huggingface/datasets/issues/4895 (the problem is explained in this comment: https://github.com/huggingface/datasets/issues/4895#issuecomment-1248269550).

In the meantime, you can do the following to fetch all the splits:
```python
dataset = load_dataset("imagefolder", data_files={"train": "imagefolder/train/**", "test": "imagefolder/test/**"})
```
Issue: I need to read the custom dataset in conll format

Comment:
Hi! We don't currently have a builder for parsing custom `conll` datasets, but I guess we could add one as a packaged module (similarly to what [TFDS](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/dataset_builders/conll/conll_dataset_builder.py) did). @lhoestq @albertvillanova WDYT?

In the meantime, you can use `Dataset.from_generator` to create a dataset as follows:
```python
from datasets import Dataset

# 2009 version
INPUT_COLUMNS = "ID FORM LEMMA PLEMMA POS PPOS FEAT PFEAT HEAD PHEAD DEPREL PDEPREL".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

# (optional) pass custom features with `features=Features(...)`
dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "path/to/conll/file"})   
```
Issue: I need to read the custom dataset in conll format

Comment:
I think we could add a dedicated builder if you think this format is general enough.
Issue: I need to read the custom dataset in conll format

Comment:



> I think we could add a dedicated builder if you think this format is general enough.

I think its functions are incomplete. It should have to_ Conll and from_ There are two methods of conll.
Issue: would huggingface like publish cpp binding for datasets package ?

Comment:
Hi ! Can you share more information about your use case ? How could it help you to have cpp bindings versus using the python libraries ?
Issue: would huggingface like publish cpp binding for datasets package ?

Comment:
> Hi ! Can you share more information about your use case ? How could it help you to have cpp bindings versus using the python libraries ?

for example ,the huggingface  load_model() and load_dataset()  can  execute in cpp env
Issue: would huggingface like publish cpp binding for datasets package ?

Comment:
If it's a viable option for you, you can check [tch-rs](https://github.com/LaurentMazare/tch-rs) to load models in Rust. Regarding datasets, you can first download them in python and then use Arrow C++ or Rust to load them
Issue: would huggingface like publish cpp binding for datasets package ?

Comment:
If you are more adventurous, another option is to embed python calls inside c++ e.g. with `pybind11`.
Issue: Force JSON format regardless of file naming on S3

Comment:
Hi ! Support for URIs like `s3://...` is not implemented yet in `data_files=`. You can use the HTTP URL instead if your data is public in the meantime
Issue: Force JSON format regardless of file naming on S3

Comment:
Hi,
I want to make sure I understand this response. I have a set of files on S3 that are private for security reasons. Because they are not public files  I cannot read those files (many are parquet) into my hf notebooks in Kaggle? That can't be correct, can it?
Issue: Force JSON format regardless of file naming on S3

Comment:
Hi ! There is a discussion at https://github.com/huggingface/datasets/issues/5281

Using the latest `datasets` 2.11 you can try passing fsspec URLs to private buckets to `data_files` in `load_dataset()`. Though this is still experimental and undocumented, so feedback is welcome. You may not have the best experience though, since anything related to performance and caching hasn't been tested properly yet.
Issue: Error loading StonyBrookNLP/tellmewhy dataset from hub even though local copy loads correctly

Comment:
I think this is because some columns are mostly empty lists. In particular the train and validation splits only have empty lists for `val_ann`. Therefore the type inference doesn't know which type is inside (or it would have to scan the other splits first before knowing).

You can fix that by specifying the features types explicitly.
Then you can save the feature types inside the dataset repository, so that you won't need to specify the features in subsequent calls:
```python
from datasets import load_dataset, Features, Sequence, Value
from datasets.info import DatasetInfosDict

features = Features({
 'narrative': Value('string'),
 'question': Value('string'),
 'original_sentence_for_question': Value('string'),
 'narrative_lexical_overlap': Value('float64'),
 'is_ques_answerable': Value('string'),
 'answer': Value('string'),
 'is_ques_answerable_annotator': Value('string'),
 'original_narrative_form': Sequence(Value('string')),
 'question_meta': Value('string'),
 'helpful_sentences': Sequence(Value('int64')),
 'human_eval': Value('bool'),
 'val_ann': Sequence(Value('int64')),
 'gram_ann': Sequence(Value('int64'))
})
ds = load_dataset('StonyBrookNLP/tellmewhy', features=features)
DatasetInfosDict({"default": ds["train"].info}).write_to_directory("path/to/local/tellmewhy")
```
and then after pushing the change to the dataset repository on the Hub, `load_dataset("StonyBrookNLP/tellmewhy")` will work directly`
Issue: Error loading StonyBrookNLP/tellmewhy dataset from hub even though local copy loads correctly

Comment:
`gram_ann` and `val_ann` are annotations that only exist for part of the test set. I wanted to keep all the columns consistent across all files, so I added them to train and validation as well. I'll check if removing them from those files is still compliant with this repo. Otherwise, I will do as you suggested. Thanks @lhoestq !
Issue: Error loading StonyBrookNLP/tellmewhy dataset from hub even though local copy loads correctly

Comment:
@lhoestq I followed the exact steps you described but it seems like I'm getting the same error unfortunately. Any other ideas? Thanks in advance
Issue: Error loading StonyBrookNLP/tellmewhy dataset from hub even though local copy loads correctly

Comment:
Hi ! If you move `dataset_infos.json` from `data/` to the root of your dataset repository if should work :)
Issue: Error loading StonyBrookNLP/tellmewhy dataset from hub even though local copy loads correctly

Comment:
I tried that and pushed to the [hub](https://huggingface.co/datasets/StonyBrookNLP/tellmewhy/tree/main). Now, there is a new error.
```
  File "/home/yklal95/tellmewhy/src/prepare_data.py", line 67, in main
    dataset = load_dataset('StonyBrookNLP/tellmewhy')
  File "/home/yklal95/anaconda3/envs/tmw-generalization/lib/python3.9/site-packages/datasets/load.py", line 1746, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/yklal95/anaconda3/envs/tmw-generalization/lib/python3.9/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/yklal95/anaconda3/envs/tmw-generalization/lib/python3.9/site-packages/datasets/builder.py", line 775, in _download_and_prepare
    verify_checksums(
  File "/home/yklal95/anaconda3/envs/tmw-generalization/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 33, in verify_checksums
    raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
datasets.utils.info_utils.ExpectedMoreDownloadedFiles: {'/home/yklal95/tellmewhy/data/test.json', '/home/yklal95/tellmewhy/data/validation.json', '/home/yklal95/tellmewhy/data/train.json'}
```
No changes were made to any of the other files and they are still on the hub. Let me know if you have any ideas @lhoestq Thanks!
Issue: Error loading StonyBrookNLP/tellmewhy dataset from hub even though local copy loads correctly

Comment:
Oh I see - the code I gave you returns local paths instead of URLs to store metadata about files to download.
I opened a PR in your repo here to remove this: https://huggingface.co/datasets/StonyBrookNLP/tellmewhy/discussions/1
sorry for the inconvenience !
Issue: Dataset Viewer issue for asapp/slue

Comment:
<img width="519" alt="Capture d’écran 2022-09-20 à 22 33 47" src="https://user-images.githubusercontent.com/1676121/191358952-1220cb7d-745a-4203-a66b-3c707b25038f.png">

```
Not found.

Error code:   SplitsResponseNotFound
```

https://datasets-server.huggingface.co/splits?dataset=asapp/slue

```json
{"error":"Not found."}
```
Issue: Dataset Viewer issue for asapp/slue

Comment:
I just launched a refresh. It's weird, I don't see any entry for this dataset in the cache, it's a bug on our side. In order to try to understand what happened, did you change the visibility status from private to public, by any chance?
Issue: Dataset Viewer issue for asapp/slue

Comment:
The dataset is being refreshed, please retry later.

<img width="802" alt="Capture d’écran 2022-09-20 à 22 39 46" src="https://user-images.githubusercontent.com/1676121/191360072-7cc86486-4e84-4b47-8f9a-4a69fe84a5ac.png">