SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Discrepancy in `nyu_depth_v2` dataset

Comment:
I think depth_map still shows discretized version. It would be nice to have corrected one.
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/depth_est_target_viz.png" width = 300>
Issue: Discrepancy in `nyu_depth_v2` dataset

Comment:
Also, I think we need to make some changes in the code to visualize depth_map as it is `float32` . `plot.imshow()` supports either [0, 1] +  float32 or [0. 255]  + uint8
Issue: Discrepancy in `nyu_depth_v2` dataset

Comment:
Oh yes! Do you want to start with the fixes? Please feel free to say no but I wanted to make sure your contributions are reflected properly in our doc and the blog :)
Issue: slice split while streaming

Comment:
Hi! Yes, that's correct. When `streaming` is `True`, only split names can be specified as `split`, and for slicing, you have to use `.skip`/`.take` instead.

E.g. 
`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train[:3]")`

rewritten with `.skip`/`.take`:
`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train").take(3)`
Issue: prebuilt dataset relies on `downloads/extracted`

Comment:
Hi! 

This issue is due to our audio/image datasets not being self-contained. This allows us to save disk space (files are written only once) but also leads to the issues like this one. We plan to make all our datasets self-contained in Datasets 3.0.

In the meantime, you can run the following map to ensure your dataset is self-contained:
```python
from datasets.table import embed_table_storage
# load_dataset ...
dset = dset.with_format("arrow")
dset.map(embed_table_storage, batched=True)
dset = dset.with_format("python")
```
Issue: prebuilt dataset relies on `downloads/extracted`

Comment:
Understood. Thank you, Mario.

Perhaps the solution could be very simple - move the extracted files into the directory of the cached dataset? Which would make it self-contained already and won't require waiting for a new major release. Unless I'm missing some back-compat nuance.

But regardless if X relies on Y - it could check if Y is still there when loading X. so not checking full consistency but just the top-level directory it relies on.
Issue: Save and resume the state of a DataLoader

Comment:
Something that'd be nice to have is "manual update of state". One of the learning from training LLMs is the ability to skip some batches whenever we notice huge spike might be handy.
Issue: Save and resume the state of a DataLoader

Comment:
Your outline spec is very sound and clear, @lhoestq - thank you!

@thomasw21, indeed that would be a wonderful extra feature. In Megatron-Deepspeed we manually drained the dataloader for the range we wanted. I wasn't very satisfied with the way we did it, since its behavior would change if you were to do multiple range skips. I think it should remember all the ranges it skipped and not just skip the last range - since otherwise the data is inconsistent (but we probably should discuss this in a separate issue not to derail this much bigger one).
Issue: ImageFolder BadZipFile: Bad offset for central directory

Comment:
Hi ! Could you share the full stack trace ? Which dataset did you try to load ?

it may be related to https://github.com/huggingface/datasets/pull/5640
Issue: ImageFolder BadZipFile: Bad offset for central directory

Comment:
The `BadZipFile` error means the ZIP file is corrupted, so I'm closing this issue as it's not directly related to `datasets`.
Issue: ImageFolder BadZipFile: Bad offset for central directory

Comment:
For others that find this issue following a `BadZipFile` error, I had the same problem because I had a file in a folder dataset `my-image.target` and the datasets library was incorrectly determining that the (PNG) file was a zip archive. When it tried to extract the file, this error occurred. 

Updating to `datasets==2.12.0` fixed the problem for me.
Issue: to_tf_dataset with a TF collator causes bizarrely persistent slowdown

Comment:
A possible hint: The function it seems to be spending a lot of time in (when iterating over the original dataset) is `_get_mp` in the PIL JPEG decoder: 
![image](https://user-images.githubusercontent.com/12866554/214057267-c889f05e-efaf-4036-b805-c5381fa62f4a.png)
Issue: to_tf_dataset with a TF collator causes bizarrely persistent slowdown

Comment:
If "mp" is multiprocessing, this might suggest some kind of negative interaction between the JPEG decoder and TF's handling of processes/threads. Note that we haven't merged the parallel `to_tf_dataset` PR yet, so it's not caused by that PR!
Issue: to_tf_dataset with a TF collator causes bizarrely persistent slowdown

Comment:
Update: MP isn't multiprocessing at all, it's an internal PIL method for loading metadata from JPEG files. No idea why that would be a bottleneck, but I'll see if a Python profiler can't figure out where the time is actually being spent.
Issue: to_tf_dataset with a TF collator causes bizarrely persistent slowdown

Comment:
After further profiling, the slowdown is in the C methods for JPEG decoding that are included as part of PIL. Because Python profilers can't inspect inside that, I don't have any further information on which lines exactly are responsible for the slowdown or why.

In the meantime, I'm going to suggest switching from `return_tensors="tf"` to `return_tensors="np"` in most of our `transformers` code - this generally works better for pre-processing. Two relevant PRs are [here](https://github.com/huggingface/transformers/pull/21266) and [here](https://github.com/huggingface/notebooks/pull/308).
Issue: info messages logged as warnings

Comment:
Looks like a duplicate of https://github.com/huggingface/datasets/issues/1948. 

I also think these should be logged as INFO messages, but let's see what @lhoestq thinks.
Issue: info messages logged as warnings

Comment:
It can be considered unexpected to see a `map` function return instantaneously. The warning is here to explain this case by mentioning that the cache was used. I don't expect first time users (only seeing warnings) to guess that the cache works this way
Issue: info messages logged as warnings

Comment:
Oh, so it's intentional? Do all Hugging Face packages use `warning` when using cache?
I guess feel free to close this issue then.
Issue: info messages logged as warnings

Comment:
Yes it's intentional for `map`. For `load_dataset` it's also intentional but for a different reason: it shows where in the cache the dataset is located, in case the user wants to clear the cache.
Issue: info messages logged as warnings

Comment:
OK I see. It's surprising to me that these are considered "something unexpected happened", the concept of cache is pretty common.

Has a user every actually complained that they ran their code once, and it took a minute while the data downloaded, then ran their code again and it ran really fast (and completed successfully) but they were so baffled by the fact that it ran quickly, _and_ didn't set the log level to INFO, _and_ hadn't read the docs (or thought about it) to know that datasets are cached, that they logged an issue asking that this information be output as a warning every time they run their code?

That seems like a very niche scenario to cater for, given that the side effect is to flood the console with irrelevant warnings for every other user every other time they run a bit of `datasets` code. And the real world impact is that people TURN OFF warnings, which is a pretty bad habit to get into.

Anyhoo, if there's no chance I'm going to change your mind, please close the issue :)
Issue: info messages logged as warnings

Comment:
I see your point and I'm not closed to switching to INFO, but I think those logs are important to make the library less opaque. I also just checked `transformers` scripts and they default to INFO which is nice. However for colab users the default is still WARNING iirc, and it counts as one of the main env where `datasets` is used.

We also use progress bars a lot in `datasets`, that are shown if the logger is at the WARNING level. But we offer a function to disable the progress bars if necessary.
Issue: info messages logged as warnings

Comment:
These kinds of messages are logged as INFO in Transformers, so we should probably be consistent with them
Issue: OneDrive Integrations  with HF Datasets 

Comment:
Hi! 

We use [`fsspec`](https://github.com/fsspec/filesystem_spec) to integrate with storage providers. You can find more info (and the usage examples) in [our docs](https://huggingface.co/docs/datasets/v2.8.0/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage).

[`gdrivefs`](https://github.com/fsspec/gdrivefs) makes it possible to use Google Drive as a storage service in Datasets, but this is not the case for OneDrive, since its[ Python SDK](https://github.com/OneDrive/onedrive-sdk-python) is not integrated with `fsspec`. Can you please request the integration with `fsspec` in their repo to address this limitation?
Issue: Can't load png dataset with 4 channel (RGBA)

Comment:
Hi! Can you please share the directory structure of your image folder and the `load_dataset` call? We decode images with Pillow, and Pillow supports RGBA PNGs, so this shouldn't be a problem.
Issue: Can't load png dataset with 4 channel (RGBA)

Comment:
> Hi! Can you please share the directory structure of your image folder and the `load_dataset` call? We decode images with Pillow, and Pillow supports RGBA PNGs, so this shouldn't be a problem.
> 
> 

I have only 1 folder that I use in the load_dataset function with the name "IMGDATA" and all my 9000 images are located in this folder.
`
from datasets import load_dataset

dataset = load_dataset("IMGDATA")
`
At the same time, using another data set with images consisting of 3 RGB channels, everything works
Issue: Can't load png dataset with 4 channel (RGBA)

Comment:
Okay, I figured out what was wrong. When uploading my dataset via Google Drive, the images broke and Pillow couldn't open them. As a result, I solved the problem by downloading the ZIP archive
Issue: Wrong statement in "Load a Dataset in Streaming mode" leads to data leakage

Comment:
Thanks for reporting, @HaoyuYang59.

Please note that these are different "dataset" objects: our docs refer to Hugging Face `datasets.Dataset` and not to TensorFlow `tf.data.Dataset`.

Our `datasets.Dataset.shuffle` method does not have a `reshuffle_each_iteration` argument. Therefore, I would say the statement in our docs is True because they refer to `datasets.Dataset.shuffle`, `datasets.Dataset.skip` and `datasets.Dataset.take`.

I think this issue is restricted to TensorFlow dataset, and this would be addressed by them in the issue you opened in their repo: https://github.com/tensorflow/tensorflow/issues/59279
Issue: Wrong statement in "Load a Dataset in Streaming mode" leads to data leakage

Comment:
Also note that you are referring to an outdated documentation page: datasets 1.10.2 version

Current datasets version is 2.8.0 and the corresponding documentation page is: https://huggingface.co/docs/datasets/stream#split-dataset
Issue: Wrong statement in "Load a Dataset in Streaming mode" leads to data leakage

Comment:
Hi @albertvillanova thanks for your reply and your explaination here. 

Sorry for the confusion as I'm not actually a user of your repo and I just happen to find the thread by Google (and didn't read carefully).

Great to know that and you made everything very clear now.

Thanks for your time and sorry for the consusion.

Wishing you a wonderful time. 

Regards
Issue: sample_dataset module not found

Comment:
working on the setfit example script

   from setfit import SetFitModel, SetFitTrainer, sample_dataset

ImportError: cannot import name 'sample_dataset' from 'setfit' (C:\Python\Python38\lib\site-packages\setfit\__init__.py)

 apart from that, I also had to hack these loads to import thses modules:
    from datasets.load import load_dataset                        
    from datasets.arrow_dataset import Dataset
    from datasets.dataset_dict import DatasetDict
Issue: sample_dataset module not found

Comment:
Hi! This issue is related to the [SetFit](https://github.com/huggingface/setfit) project, so can you please open it there?
Issue: Support latest Docker image in CI benchmarks

Comment:
Sorry, it was us:[^1] https://github.com/iterative/cml/pull/1317 & https://github.com/iterative/cml/issues/1319#issuecomment-1385599559; should be fixed with [v0.18.17](https://github.com/iterative/cml/releases/tag/v0.18.17).

[^1]: More or less, see https://github.com/yargs/yargs/issues/873.
Issue: Support latest Docker image in CI benchmarks

Comment:
Hi @0x2b3bfa0, thanks a lot for the investigation, the context about the the root cause and for fixing it!!

We are reviewing your PR to unpin the container image.
Issue: Support Apache Beam >= 2.44.0

Comment:
Some of the shard files now have 0 number of rows.

We have opened an issue in the Apache Beam repo:
- https://github.com/apache/beam/issues/25041
Issue: Load/Save FAISS index using fsspec

Comment:
Hi! Sure, feel free to submit a PR. Maybe if we want to be consistent with the existing API, it would be cleaner to directly add support for `fsspec` paths in `Dataset.load_faiss_index`/`Dataset.save_faiss_index` in the same manner as it was done in `Dataset.load_from_disk`/`Dataset.save_to_disk`.
Issue: Unable to download dataset id_clickbait

Comment:
Thanks for reporting, @ilos-vigil.

We have transferred this issue to the corresponding dataset on the Hugging Face Hub: https://huggingface.co/datasets/id_clickbait/discussions/1
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
Hi! 

`Dataset.sort` calls `df.sort_values` internally, and `df.sort_values` brings all the "sort" columns in memory, so sorting on multiple keys could be very expensive. This makes me think that maybe we can replace `df.sort_values` with `pyarrow.compute.sort_indices` - the latter can also sort on multiple keys and currently loads the data into memory; however, there is a plan to eventually implement "memory-map" friendly kernels for the Arrow compute ops (using the Acero execution engine). 

So to address this issue, you should replace `df.sort_values` with `pyarrow.compute.sort_indices` in `Dataset.sort` and adjust the signature of this function (deprecate the `kind` parameter, etc.).

PS: Feel free to ping us if you need some additional help/pointers
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
@mariosasko If I understand the code right, using `pyarrow.compute.sort_indices` would also require changes to the `select` method if it is meant to sort multiple keys. That's because `select` only accepts 1D input for `indices`, not an iterable or similar which would be required for multiple keys unless you want some looping over selects. Doesn't seem that straight-forward but I might be missing something here...
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
@MichlF No, it doesn't require modifying select because sorting on multiple keys also returns a 1D array.

It's easier to understand with an example:
```python
>>> import pyarrow as pa
>>> import pyarrow.compute as pc
>>> table = pa.table({
...     "name": ["John", "Eve", "Peter", "John"],
...     "surname": ["Johnson", "Smith", "Smith", "Doe"],
...     "age": [20, 40, 30, 50],
... })
>>> indices = pc.sort_indices(table, sort_keys=[("name", "ascending"), ("surname", "ascending")])
>>> print(indices)
[
  1,
  3,
  0,
  2
]
```
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
Thanks for clarifying.
I can prepare a PR to address this issue. This would be my first PR here so I have a few maybe silly questions but:
- What is the preferred input type of `sort_keys` for the sort method? A sequence with name, order tuples like pyarrow's `sort_indices` requires?
- What about backwards compatability: is it supposed to also accept the old way of calling sort() or should both `column` and `kind` be deprecated?
- If `sort_keys` is provided in the same format as for pyarrow's `sort_indices` - i.e. along with order for each column -, `reverse` doesn't make much sense either and should be deprecated as well I assume.
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
I think we can have the following signature:
```python
def sort(
        self,
        column_names: Union[str, Sequence[str]],
        reverse: Union[bool, Sequence[bool]] = False,
        kind="deprecated",
        null_placement: str = "last",
        keep_in_memory: bool = False,
        load_from_cache_file: bool = True,
        indices_cache_file_name: Optional[str] = None,
        writer_batch_size: Optional[int] = 1000,
        new_fingerprint: Optional[str] = None,
    ) -> "Dataset":
``` 

So we should:
* rename`column` to `column_names`. `column` is a positional argument, so it's OK to rename it (not marked as positional-only with "/", but still should be fine)
* deprecate `kind`
* keep `reverse` instead of introducing `sort_keys`, but we should allow passing a list of booleans that defines the sort order of each column from `column_names` to it (`reverse = False` would be equal to `[False] * len(column_names)` and `reverse = True` to `[True] * len(column_names)`)
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
I am pretty much done with the PR. Just one clarification: `Sequence` in `arrow_dataset.py` is a custom dataclass from `features.py` instead of the `type.hinting` class `Sequence` from Python. Do you suggest using that custom `Sequence` class somehow ? Otherwise signature currently reads instead:
```Python
    def sort(
        self,
        column_names: Union[str, List[str]],
        reverse: Union[bool, List[bool]] = False,
        kind = "deprecated",
        null_placement: str = "last",
        keep_in_memory: bool = False,
        load_from_cache_file: bool = True,
        indices_cache_file_name: Optional[str] = None,
        writer_batch_size: Optional[int] = 1000,
        new_fingerprint: Optional[str] = None,
    )
```

Also, to maintain backwards compatibility, I added conditionals for `null_placement`, because pyarrow's `null_placement` only accepts `at_start` and `at_end`, and not `last` and `first`.
If that is all good, I think I can open the PR.
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
I meant `typing.Sequence` (`datasets.Sequence` is a feature type). 

Regarding `null_placement`,  I think we can support both `at_start` and `at_end`, and  `last` and `first` (for backward compatibility; convert internally to `at_end` and `at_start` respectively).
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
> I meant typing.Sequence (datasets.Sequence is a feature type).

Sorry, I actually meant `typing.Sequence` and not `type.hinting`. However, the issue is still that `dataset.Sequence` is imported in `arrow_dataset.py` so I cannot import and use `typing.Sequence` for the `sort`'s signature without overwriting the `dataset.Sequence` import. The latter is used in the `align_labels_with_mapping` method so it's a necessary import for `arrow_dataset.py`. 
To import `typing.Sequence` as something else than `Sequence` to avoid overwriting may only be confusing and doesn't seem good practice!? The other solution is to keep `List` type hinting as in the signature I posted in my previous post but this excludes other Sequence types and may cause problems further down the line.
Please advise,
Thanks for all the clarifications!
Issue: Sort on multiple keys with datasets.Dataset.sort()

Comment:
You can avoid the name collision by renaming `typing.Sequence` to `Sequence_` when importing:
```python
from typing import Sequence as Sequence_
```
Issue: When applying `ReadInstruction` to custom load it's not DatasetDict but list of Dataset?

Comment:
Hi! You can get a `DatasetDict` if you pass a dictionary with read instructions as follows:
```python
instructions = [
    ReadInstruction(split_name="train", from_=0, to=10, unit='%', rounding='closest'),
    ReadInstruction(split_name="dev", from_=0, to=10, unit='%', rounding='closest'),
    ReadInstruction(split_name="test", from_=0, to=5, unit='%', rounding='closest')
]

dataset = load_dataset('csv', data_dir="data/", data_files={"train":"train.tsv", "dev":"dev.tsv", "test":"test.tsv"}, delimiter="\t", split={inst.split_name: inst for inst in instructions})
```
Issue: Datasets load error for saved github issues

Comment:
I can confirm that the error exists!
I'm trying to read 3 parquet files locally:
```python
from datasets import load_dataset, Features, Value, ClassLabel

review_dataset = load_dataset(
    "parquet",
    data_files={
        "train": os.path.join(sentiment_analysis_data_path, "train.parquet"),
        "validation": os.path.join(sentiment_analysis_data_path, "validation.parquet"),
        "test": os.path.join(sentiment_analysis_data_path, "test.parquet"),
    },
)
```

But you can fix it, by specifying `features` for `load_dataset()` function like this:
```python
from datasets import load_dataset, Features, Value, ClassLabel

features = Features(
    {
        "label": ClassLabel(
            num_classes=3,
            names=["negative", "neutral", "positive"],
        ),
        "text": Value(dtype="string"),
    }
)

review_dataset = load_dataset(
    "parquet",
    data_files={
        "train": os.path.join(sentiment_analysis_data_path, "train.parquet"),
        "validation": os.path.join(sentiment_analysis_data_path, "validation.parquet"),
        "test": os.path.join(sentiment_analysis_data_path, "test.parquet"),
    },
    features=features,
)

print(review_dataset)
```
Issue: Datasets load error for saved github issues

Comment:
@Extremesarova I think this is a different issue, but understand using features could be a work-around.
It seems the field `closed_at` is `null` in many cases.

I've not found a way to specify only a single feature without (succesfully) specifiying the full and quite detailed set of expected features. Using this features set gives an error the column names don't match.
`features = Features({'closed_at': Value(dtype='timestamp[s]', id=None)})`
Issue: Datasets load error for saved github issues

Comment:
Found this when searching for the same error, looks like based on #3965 it's just an issue with the data. I found that changing `df = pd.DataFrame.from_records(all_issues)` to `df = pd.DataFrame.from_records(all_issues).dropna(axis=1, how='all').drop(['milestone'], axis=1)` from the fetch_issues function fixed the issue. 
The "milestone" column seemed to be problematic (only ~50 non null rows) and dropped any columns that were all null as well just in case.
Issue: Datasets load error for saved github issues

Comment:
I have this same issue.  I saved a dataset to disk and now I can't load it.
Issue: Support case-insensitive Hub dataset name in load_dataset

Comment:
Closing as case-insensitivity should be only for URL redirection on the Hub. In the APIs, we will only support the canonical name (https://github.com/huggingface/moon-landing/pull/2399#issuecomment-1382085611)
Issue: label_column='labels' in datasets.TextClassification and 'label' or 'label_ids' in transformers.DataColator

Comment:
Hi! Thanks for pointing out this inconsistency. Changing the default value at this point is probably not worth it, considering we've started discussing the state of the task API internally - we will most likely deprecate the current one and replace it with a more robust solution that relies on the `train_eval_index` field stored in the YAML section of the dataset cards.
Issue: label_column='labels' in datasets.TextClassification and 'label' or 'label_ids' in transformers.DataColator

Comment:
The task templates API has been deprecated (will be removed in version 3.0), so I'm closing this issue.
Issue: Add ProgressBar for `to_parquet`

Comment:
Thanks for your proposal, @zanussbaum. Yes, I agree that would definitely be a nice feature to have!