SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
Hi! Yes, we should override `DatasetDict.__setitem__` and throw an error if features dictionaries are different. `DatasetDict` is a subclass of `dict`, so `DatasetDict.{update/setdefault}` need to be overridden as well. We could avoid this by subclassing `UserDict`, but then we would get the name collision - `DatasetDict.data` vs. `UserDict.data`. This makes me think we should rename the `data` attribute of `DatasetDict`/`Dataset` for easier dict subclassing (would also simplify https://github.com/huggingface/datasets/pull/3997) and to follow good Python practices. Another option is to have a custom `UserDict` class in `py_utils`, but it can be hard to keep this class consistent with the built-in `UserDict`. 

@albertvillanova @lhoestq wdyt?
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
I would keep things simple and keep subclassing dict. Regarding the features check, I guess this can be done only for `push_to_hub` right ? It is the only function right now that requires the underlying datasets to be splits (e.g. train/test) and have the same features.

Note that later you will be able to push datasets with different features as different dataset **configurations** (similarly to the [GLUE subsets](https://huggingface.co/datasets/glue) for example). We will work on this soon
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
Hi @lhoestq,

Returning to this thread to ask whether the possibility to create `DatasetDict` with different configurations will be supported in the future.

Best,
Pietro
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
DatasetDict is likely to always require the datasets to have the same columns and types, while different configurations may have different columns and types.

Why would you like to see that ?
If it's related to push_to_hub, we plan to allow pushing several configs, but not using DatasetDict
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
Hi @lhoestq and @pietrolesci,

I have been curious about this question as well. I don't have experience working with different configurations, but I can give a bit more detail on the work flow that I have been using with `Dataset_dict`.

As @pietrolesci mentions, I have been using `push_to_hub` to quickly iterate on dataset curation for different ML experiments - locally I create a set of dataset splits e.g. `train/val/test/inference`, then convert them to `HF_Datasets` and finally a to `Dataset_Dict` to `push_to_hub`. Where I have run into issues is when I want to include different metadata for different splits. For example, I have situations where I only have meta-data for one of the splits (e.g. test) or situations where I am working with `inference` data that does not have labels. Currently I use a rather hacky work around by adding "dummy" columns for missing columns to avoid the error:

```
ValueError: All datasets in `DatasetDict` should have the same features
```

I am curious why `DatasetDict` will likely not support this functionality? I don't know much about working with different configurations, but allowing for different columns between datasets / splits would be a very helpful use-case for me. Are there any docs for using different configuration OR a more info about incorporating it with `push_to_hub`.

Best wishes,
Jonathan
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
> I am curious why DatasetDict will likely not support this functionality?

There's a possibility we may merge the Dataset and DatasetDict classes. The DatasetDict purpose was to define a way to get the train/test splits of a dataset.

see the discussions at https://github.com/huggingface/datasets/issues/5189

> Are there any docs for using different configuration OR a more info about incorporating it with push_to_hub.

There's a PR open to allow to upload a dataset with a certain configuration name. Then later you can reload this specific configuration using `load_dataset(ds_name, config_name)`

see the PR at https://github.com/huggingface/datasets/pull/5213
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
Hi, regarding the following information:

> Please note that this is a design purpose: a `DatasetDict` has the same features for all its datasets. Normally, a `DatasetDict` is composed of several sub-datasets each corresponding to a different **split**.
> 
> To handle sub-datasets with different features, we use another approach: use different **configurations** instead of **splits**.

Altough this is often implied (such as how else would `DatasetDict` be able to process multiple splits in the same way?), I would expect it to be written somewhere in the docs plainly and maybe even in bold. Also I would expect to see it in multiple places such as:

- in docstring of `DatasetDict`
- in nlp/image/audio guides on how to create a dataset
- [in conceptual guide on how to create a loading script](https://huggingface.co/docs/datasets/main/en/about_dataset_load)


I think this addition would benefit the docs, especially when you guide a newbie (such as me) through the process of creating a dataset. As I said, you somehow suspect that this is in fact the case, but without reading it in the docs you cannot be sure.
Issue: TypeError: Cannot cast array data from dtype('O') to dtype('int64') according to the rule 'safe'

Comment:
@albertvillanova @mariosasko thank you, with that change now I get
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-9-eeb68eeb9bec>](https://localhost:8080/#) in <module>()
     11 )
     12 # You can make this part faster with num_proc=<some int>
---> 13 sentences = sentences.map(lambda ex: features["label"].str2int(ex["label"]) if ex["label"] is not None else None, features=features)
     14 sentences = sentences.shuffle()

8 frames
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in validate_function_output(processed_inputs, indices)
   2193             if processed_inputs is not None and not isinstance(processed_inputs, (Mapping, pa.Table)):
   2194                 raise TypeError(
-> 2195                     f"Provided `function` which is applied to all elements of table returns a variable of type {type(processed_inputs)}. Make sure provided `function` returns a variable of type `dict` (or a pyarrow table) to update the dataset or `None` if you are only interested in side effects."
   2196                 )
   2197             elif isinstance(indices, list) and isinstance(processed_inputs, Mapping):

TypeError: Provided `function` which is applied to all elements of table returns a variable of type <class 'int'>. Make sure provided `function` returns a variable of type `dict` (or a pyarrow table) to update the dataset or `None` if you are only interested in side effects.
```

the error is raised by [this](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L2221)

```
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in validate_function_output(processed_inputs, indices)
```
Issue: TypeError: Cannot cast array data from dtype('O') to dtype('int64') according to the rule 'safe'

Comment:
@mariosasko changed it like

```python
sentences = sentences.map(lambda ex: {"label" : features["label"].str2int(ex["label"]) if ex["label"] is not None else None}, features=features)
```

to avoid the above errorr.
Issue: TypeError: Cannot cast array data from dtype('O') to dtype('int64') according to the rule 'safe'

Comment:
Any update on this? Is this correct ?
> @mariosasko changed it like
> 
> ```python
> sentences = sentences.map(lambda ex: {"label" : features["label"].str2int(ex["label"]) if ex["label"] is not None else None}, features=features)
> ```
> 
> to avoid the above errorr.
Issue: Cache miss during reload for datasets using image fetch utilities through map 

Comment:
Hi ! Maybe one of the objects in the function is not deterministic across sessions ? You can read more about it and how to investigate here: https://huggingface.co/docs/datasets/about_cache
Issue: Cache miss during reload for datasets using image fetch utilities through map 

Comment:
Hi @apsdehal! Can you verify that replacing
```python
def fetch_single_image(image_url, timeout=None, retries=0):
    for _ in range(retries + 1):
        try:
            request = urllib.request.Request(
                image_url,
                data=None,
                headers={"user-agent": get_datasets_user_agent()},
            )
            with urllib.request.urlopen(request, timeout=timeout) as req:
                image = PIL.Image.open(io.BytesIO(req.read()))
            break
        except Exception:
            image = None
    return image
```
with 
```python
USER_AGENT = get_datasets_user_agent()

def fetch_single_image(image_url, timeout=None, retries=0):
    for _ in range(retries + 1):
        try:
            request = urllib.request.Request(
                image_url,
                data=None,
                headers={"user-agent": USER_AGENT},
            )
            with urllib.request.urlopen(request, timeout=timeout) as req:
                image = PIL.Image.open(io.BytesIO(req.read()))
            break
        except Exception:
            image = None
    return image
```
fixes the issue?
Issue: Cache miss during reload for datasets using image fetch utilities through map 

Comment:
Thanks @mariosasko. That does fix the issue. In general, I think these image downloading utilities since they are being used by a lot of image dataset should be provided as a part of `datasets` library right to keep the logic consistent and READMEs smaller? If they already exists, that is also great, please point me to those. I saw that `http_get` does exist.
Issue: Cache miss during reload for datasets using image fetch utilities through map 

Comment:
You can find my rationale (and a proposed solution) for why these utilities are not a part of `datasets` here: https://github.com/huggingface/datasets/pull/4100#issuecomment-1097994003.
Issue: Cache miss during reload for datasets using image fetch utilities through map 

Comment:
Makes sense. But, I think as the number of image datasets as grow, more people are copying pasting original code from docs to work as it is while we make fixes to them later. I think we do need a central place for these to avoid that confusion as well as more easier access to image datasets. Should we restart that discussion, possible on slack?
Issue: load_dataset can't load local dataset,Unable to find ...

Comment:
Hi! :)

I believe that should work unless `dataset_infos.json` isn't actually a dataset. For Hugging Face datasets, there is usually a file named `dataset_infos.json` which contains metadata about the dataset (eg. the dataset citation, license, description, etc). Can you double-check that `dataset_infos.json` isn't just metadata please?
Issue: load_dataset can't load local dataset,Unable to find ...

Comment:
Hi @ahf876828330, 

As @stevhliu pointed out, the proper way to load a dataset is not trying to load its metadata file.

In your case, as the dataset script is local, you should better point to your local loading script:
```python
dataset = load_dataset("dataset/opus_books.py")
```

Please, feel free to re-open this issue if the previous code snippet does not work for you.
Issue: load_dataset can't load local dataset,Unable to find ...

Comment:
> Hi! :)
> 
> I believe that should work unless `dataset_infos.json` isn't actually a dataset. For Hugging Face datasets, there is usually a file named `dataset_infos.json` which contains metadata about the dataset (eg. the dataset citation, license, description, etc). Can you double-check that `dataset_infos.json` isn't just metadata please?

Yes，you are right!So if I have a metadata dataset local,How can I turn it to a dataset that can be used by the load_dataset() function？Are there some examples?
Issue: load_dataset can't load local dataset,Unable to find ...

Comment:
The metadata file isn't a dataset so you can't turn it into one. You should try @albertvillanova's code snippet above (now merged in the docs [here](https://huggingface.co/docs/datasets/master/en/loading#local-loading-script)), which uses your local loading script `opus_books.py` to:

1. Download the actual dataset. 
2. Once the dataset is downloaded, `load_dataset` will load it for you.
Issue: feat: create an `Array3D` column from a list of arrays of dimension 2

Comment:
Hi @SaulLu, thanks for your proposal.

Just I got a bit confused about the dimensions...
- For the 2D case, you mention it is possible to create an `Array2D` from a list of arrays of dimension 1
- However, you give an example of creating an `Array2D` from arrays of dimension 2:
  - the values of `data_map` are arrays of dimension 2
  - the outer list in `prepare_dataset_2D` should not be taken into account in the dimension counting, as it is used because in `map` you pass `batched=True`

Note that for the 3D alternatives you mention:
- In `prepare_dataset_3D_ter`, you create an `Array3D` from arrays of dimension 3:
  - the array `data_map[index][np.newaxis, :, :]` has dimension 3
  - the outer list in `prepare_dataset_3D_ter` is the one used by `batched=True`
- In `prepare_dataset_3D_bis`, you create an `Array3D` from a list of list of lists:
  - the value of `data_map[index].tolist()` is a list of lists
  - it is enclosed by another list `[data_map[index].tolist()]`, thus giving a list of list of lists
  - the outer list is the one used by `batched=True`

Therefore, if I understand correctly, your request would be to be able to create an `Array3D` from a list of an array of dimension 2:
- In `prepare_dataset_3D`, `data_map[index]` is an array of dimension 2
- it is enclosed by a list `[data_map[index]]`, thus giving a list of an array of dimension 2
- the outer list is the one used by `batched=True`

Please, feel free to tell me if I did not understand you correctly.
Issue: feat: create an `Array3D` column from a list of arrays of dimension 2

Comment:
Hi @albertvillanova ,

Indeed my message was confusing and you guessed right :smile: : I think would be interesting to be able to create an Array3D from a list of an array of dimension 2. 

For the 2D case I should have given as a "similar" example:
```python

data_map_1D = {
    1: np.array([0.2, 0.4]),
    2: np.array([0.1, 0.4]),
}

def prepare_dataset_2D(batch):
    batch["pixel_values"] = [[data_map_1D[index]] for index in batch["id"]]
    return batch
    
ds_2D = ds.map(
    prepare_dataset_2D, 
    batched=True, 
    remove_columns=ds.column_names, 
    features=features.Features({"pixel_values": features.Array2D(shape=(1, 2), dtype="float32")})
)
```
Issue: Librispeech documentation, clarification on format

Comment:
The documentation in the code is definitely outdated - thanks for letting me know, I'll remove it in https://github.com/huggingface/datasets/pull/4184 .

You're exactly right `audio` `array` already decodes the audio file to the correct waveform. This is done on the fly, which is also why one should **not** do `ds["audio"]["array"][0]` as this will decode all dataset samples, but instead `ds[0]["audio"]["array"]` see: https://huggingface.co/docs/datasets/audio_process#audio-datasets
Issue: Librispeech documentation, clarification on format

Comment:
So, again to clarify: On disk, only the raw flac file content is stored? Is this also the case after `save_to_disk`?

And is it simple to also store it re-encoded as ogg or mp3 instead?
Issue: Librispeech documentation, clarification on format

Comment:
Hey, 

Sorry yeah I was just about to look into this! We actually had an outdated version of Librispeech ASR that didn't save any files, but instead converted the audio files to a byte string, then was then decoded on-the-fly. This however is not very user-friendly so we recently decided to instead show the full path of the audio files with the `path` parameter.

I'm currently changing this for Librispeech here: https://github.com/huggingface/datasets/pull/4184 .
You should be able to see the audio file in the original `flac` format under `path` then. I don't think it's a good idea to convert to MP3 out-of-the-box, but we could maybe think about some kind of convert function for audio datasets cc @lhoestq ?
Issue: Librispeech documentation, clarification on format

Comment:
> I don't think it's a good idea to convert to MP3 out-of-the-box, but we could maybe think about some kind of convert function for audio datasets cc @lhoestq ?

Sure, I would expect that `load_dataset("librispeech_asr")` would give you the original (not re-encoded) data (flac or already decoded). So such re-encoding logic would be some separate generic function. So I could do sth like `dataset.reencode_as_ogg(**ogg_encode_opts).save_to_disk(...)` or so.
Issue: Librispeech documentation, clarification on format

Comment:
A follow-up question: I wonder whether a Parquet dataset is maybe more what we actually want to have? (Following also my comment here: https://github.com/huggingface/datasets/pull/4184#issuecomment-1105045491.) Because I think we actually would prefer to embed the data content in the dataset.

So, instead of `save_to_disk`/`load_from_disk`, we would use `to_parquet`,`from_parquet`? Is there any downside? Are arrow files more efficient?

Related is also the doc update in #4193.
Issue: Librispeech documentation, clarification on format

Comment:
`save_to_disk` saves the dataset as an Arrow file, which is the format we use to load a dataset using memory mapping. This way the dataset does not fill your RAM, but is read from your disk instead.

Therefore you can directly reload a dataset saved with `save_to_disk` using `load_from_disk`.

Parquet files are used for cold storage: to use memory mapping on a Parquet dataset, you first have to convert it to Arrow. We use Parquet to reduce the I/O when pushing/downloading data from the Hugging face Hub. When you load a Parquet file from the Hub, it is converted to Arrow on the fly during the download.
Issue: Zenodo.org download is not responding

Comment:
Hi @dkajtoch, please note that at HuggingFace we are not hosting this dataset: we are just using a script to download their data file and create a dataset from it.

It was the dataset owners decision to host their data at Zenodo. You can see this on their website: https://marcobaroni.org/composes/sick.html

And yes, you are right: Zenodo is currently having some incidents and people are reporting problems from it.

On the other hand, we could contact the data owners and propose them to host their data at our Hugging Face Hub.

@julien-c I guess so.
Issue: Zenodo.org download is not responding

Comment:
Thanks @albertvillanova. I know that the problem lies in the source data. I just wanted to point out that these kind of problems are unavoidable without having one place where data sources are cached. Websites may go down or data sources may move. Having a copy in Hugging Face Hub would be a great solution.
Issue: Zenodo.org download is not responding

Comment:
Definitely, @dkajtoch! But we have to ask permission to the data owners. And many dataset licenses directly forbid data redistribution: in those cases we are not allowed to host their data on our Hub.
Issue: Support streaming FLEURS dataset

Comment:
Yes, you just have to use `dl_manager.iter_archive` instead of `dl_manager.download_and_extract`.

That's because `download_and_extract` doesn't support TAR archives in streaming mode.
Issue: Support streaming FLEURS dataset

Comment:
Tried to make it streamable, but I don't think it's really possible. @lhoestq @polinaeterna maybe you guys can check: 
https://huggingface.co/datasets/google/fleurs/commit/dcf80160cd77977490a8d32b370c027107f2407b 

real quick. 

I think the problem is that we cannot ensure that the metadata file is found before the audio. Or is this possible somehow @lhoestq ?
Issue: Support streaming FLEURS dataset

Comment:
@patrickvonplaten I think the metadata file should be found first because the audio files are contained in a folder next to the metadata files (just as in common voice), so the metadata files should be "on top of the list"   as they are closer to the root in the directories hierarchy
Issue: Support streaming FLEURS dataset

Comment:
The order of the files is determined when the TAR archive is created, depending on the commands the creator ran.
If the metadata file is not at the beginning of the file, that makes streaming completely inefficient. In this case the TAR archive needs to be recreated in an appropriate order.
Issue: Support streaming FLEURS dataset

Comment:
Actually we could maybe just host the metadata file ourselves and then stream the audio data only. Don't think that this would be a problem for the FLEURS authors (I can ask them :-))
Issue: Support streaming FLEURS dataset

Comment:
I made a PR to their repo to support streaming (by uploading the metadata file to the Hub). See:
- https://huggingface.co/datasets/google/fleurs/discussions/4
Issue: Add some iteration method on a dataset column (specific for inference)

Comment:
Thanks for the suggestion ! I agree it would be nice to have something directly in `datasets` to do something as simple as that

cc @albertvillanova @mariosasko @polinaeterna What do you think if we have something similar to pandas `Series` that wouldn't bring everything in memory when doing `dataset["audio"]` ? Currently it returns a list with all the decoded audio data in memory.

It would be a breaking change though, since `isinstance(dataset["audio"], list)` wouldn't work anymore, but we could implement a `Sequence` so that `dataset["audio"][0]` still works and only loads one item in memory.

Your alternative suggestion with `iterate` is also sensible, though maybe less satisfactory in terms of experience IMO
Issue: Add some iteration method on a dataset column (specific for inference)

Comment:
I agree that current behavior (decoding all audio file sin the dataset when accessing `dataset["audio"]`) is not useful, IMHO. Indeed in our docs, we are constantly warning our collaborators not to do that.

Therefore I upvote for a "useful" behavior of `dataset["audio"]`. I don't think the breaking change is important in this case, as I guess no many people use it with its current behavior. Therefore, for me it seems reasonable to return a generator (instead of an in-memeory list) for "special" features, like Audio/Image.

@lhoestq on the other hand I don't understand your proposal about Pandas-like...
Issue: Add some iteration method on a dataset column (specific for inference)

Comment:
I recall I had the same idea while working on the `Image` feature, so I agree implementing something similar to `pd.Series` that lazily brings elements in memory would be beneficial.
Issue: Add some iteration method on a dataset column (specific for inference)

Comment:
@lhoestq @mariosasko Could you please give a link to that new feature of `pandas.Series`? As far as I remember since I worked with pandas for more than 6 years, there was no lazy in-memory feature; it was everything in-memory; that was the reason why other frameworks were created, like Vaex or Dask, e.g.
Issue: Add some iteration method on a dataset column (specific for inference)

Comment:
Yea pandas doesn't do lazy loading. I was referring to pandas.Series to say that they have a dedicated class to represent a column ;)