SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Align task tags in dataset metadata

Comment:
Yes, @julien-c. These are some of the feedbacks:
- Most people just thank for the fix: [cahya/librivox-indonesia](https://huggingface.co/datasets/cahya/librivox-indonesia/discussions/1#6357cd8a292a050ebd705f84), [TurkuNLP/xlsum-fi](https://huggingface.co/datasets/TurkuNLP/xlsum-fi/discussions/1#6357828aa1f8ad1c31bcbe46), [coastalcph/fairlex](https://huggingface.co/datasets/coastalcph/fairlex/discussions/4#6351a527a8e595171ab1aef2)
- Why are we changing their task names? [joelito/lextreme](https://huggingface.co/datasets/joelito/lextreme/discussions/1#6351b576fe367c0d9b12041b)
  - I take note of this for the next bulk operation; besides the PR title, we should also add a description to explain the reason for the change and also maybe putting a link to some pertinent GH Issue page
- Some of them ask where to find the list of the supported task values is: [dennlinger/klexikon](https://huggingface.co/datasets/dennlinger/klexikon/discussions/3#6356b3ea80f8cb3ab777ac5c), [lmqg/qg_jaquad](https://huggingface.co/datasets/lmqg/qg_jaquad/discussions/1#635262467e4cc3135fd09f58)
  - Currently, the list is here: https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts#L85
  - Maybe we could made them more easily accessible
- Some people do not agree about current "hierarchy":
  - text-scoring: [emrecan/nli_tr_for_simcse](https://huggingface.co/datasets/emrecan/nli_tr_for_simcse/discussions/1#6357c1b128792d8cdd51e9f9) (but referring to [emrecan/nli_tr_for_simcse](https://huggingface.co/datasets/emrecan/nli_tr_for_simcse/discussions/2/files))
    - Before "text-scoring" was a task_category, with task_ids ["semantic-similarity-scoring", "sentiment-scoring"]
    - Now all three are task_ids ["text-scoring", "semantic-similarity-scoring", "sentiment-scoring"] under the task_category "text-classification"
    - People complain that their scoring tasks are not classification task
  - binary-classification: why don't we have binary-classification? We have multi-class-classification, multi-label-classification and sentiment-classification, but not binary-classification
  - symbolic-regression: [yoshitomo-matsubara/srsd-feynman_hard](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_hard/discussions/2#63614194c12a09b8a31457cc), [yoshitomo-matsubara/srsd-feynman_medium](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_medium/discussions/2#6361418aeee0d27f04379e43), [yoshitomo-matsubara/srsd-feynman_easy](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_easy/discussions/2#6361416e00905b1ffb8d0112)
    - Why don't we have symbolic-regression task?

NOTE: I'm editing this comment to add more feedback
Issue: Align task tags in dataset metadata

Comment:
As someone with feedback on the updates (which I highly appreciate seeing included here :D), a few comments from a "user perspective":  

* I think the general confusion for me was also surrounding the hierarchy; it doesn't really become super clear (even when using the tagger space) that one is a subset of the other, especially since it seems to be still possible to include fine-grained tasks without the "parent category"?
* The datasets explorer still shows tags that are no longer valid (e.g., super specific ones such as `summarization-other-paper-abstract-generation`, but also ones that should be `task_categories`, such as `summarization`). I'm assuming this will be fixed soon, but until then it can confuse people who don't understand why they suddenly can't use seemingly still valid tags anymore.
* As I mentioned to @albertvillanova, having a dedicated page in the docs with explanations (especially wrt the difference between `task_categories` and `task_ids`) would be super helpful. However, I think it would have been sufficient to just include some description in the dataset PRs where you can link to the Github/other discussion on the topic :) That way, I can check myself what changes are expected to happen.

Thanks again for the streamlining process, I personally learned a fair bit about the tagging structure in the meantime!
Best,
Dennis
Issue: Align task tags in dataset metadata

Comment:
Thanks to you both for your feedback! super useful! cc'ing @osanseviero too 🙂

> The datasets explorer still shows tags that are no longer valid

wait which explorer is that? is it https://huggingface.co/datasets/viewer/ ?
Issue: Align task tags in dataset metadata

Comment:
- Some people do not agree about current "hierarchy":
  - symbolic-regression: [yoshitomo-matsubara/srsd-feynman_hard](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_hard/discussions/2#63614194c12a09b8a31457cc), [yoshitomo-matsubara/srsd-feynman_medium](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_medium/discussions/2#6361418aeee0d27f04379e43), [yoshitomo-matsubara/srsd-feynman_easy](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_easy/discussions/2#6361416e00905b1ffb8d0112)
    - Why don't we have symbolic-regression task?
Issue: Align task tags in dataset metadata

Comment:
@albertvillanova 
Thank you for sharing our voice here!

Yes, we want `symbolic-regression` to be listed as a task. This task has been attracting attention from the machine learning/deep learning community, and unfortunately existing symbolic regression datasets are de-centralized in the community (hosted at individual platforms like author website, github, etc).
It would be great for the community if Hugging Face can support the task.
Issue: Raise ImportError instead of OSError if required extraction library is not installed

Comment:
hey @mariosasko , i made a pr for this issue. Could you please review it.
Also i found multiple `OSError` in `extract.py` file which i thought could be replaced too but wasn't sure about them.
Please do tell if that also needs to be done.
Issue: Tensor operation not functioning in dataset mapping

Comment:
Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .
Issue: Tensor operation not functioning in dataset mapping

Comment:
> Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .

Thank you.
Issue: WikiText 103 tokenizer hangs

Comment:
any updates on this? It happens to me on [OpenWikiText-20%](https://huggingface.co/datasets/Bingsu/openwebtext_20p) dataset, but not on [OpenWebText-10k](https://huggingface.co/datasets/stas/openwebtext-10k). This is really strange because I don't change anything else in my running script.

transformers version 4.18.0.dev0
datasets version 1.18.0
Issue: unexpected `cast` or `class_encode_column` result after `rename_column`

Comment:
Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...
Issue: unexpected `cast` or `class_encode_column` result after `rename_column`

Comment:
Hi, 方子东. I tried running the code with exact the same configuration (both datasets 2.5.2 and 2.6.1, python, pyarrow, pandas), but on Linux. The results seem to be the expected `{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}`.
I don't have a Mac device. I can't verify whether this is a M1 chip-specific problem.
Issue: unexpected `cast` or `class_encode_column` result after `rename_column`

Comment:
> Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...

Thank you for your attention and feel sorry to take your time. Since this is a bug of old version, I think mybe my problem is because `cast` operation directaly used cached data generated by older verion of `datasets`. I tried to deleted the cached data and I got expected result.
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
@lhoestq  I tested the script without accelerator, and I confirm this is due to datasets part as this gets similar results without accelerator.
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
Hi ! You said it works on 1 GPU but doesn't wortk without accelerator - what's the difference between running on 1 GPU and running without accelerator in your case ?
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
Hi @lhoestq 
thanks for coming back to me. Sorry for the confusion I made. I meant this works fine on 1 GPU, but on multi-gpu it is freezing. "accelerator" is not an issue as if you adapt the code without accelerator this still gets the same issue.
In order to test it. Please run "accelerate config", then use the setup for multi-gpu in one node.
After that run "accelerate launch code.py" and then you would see the freezing occurs.
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
Hi @lhoestq 
could you have the chance to reproduce the error by running the minimal example shared?
thanks
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
I think you need to do `train_dataset = train_dataset.with_format("torch")` to work with the DataLoader in a multiprocessing setup :)

The hang is probably caused by our streamign lib `fsspec` which doesn't work in multiprocessing out of the box - but we made it work with the PyTorch DataLoader when the dataset format is set to "torch"
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
Hi @lhoestq 
thanks for the response. I added  the line suggested right before calling `with accelerator.main_process_first():` in the code above and I confirm this also freezes. to reproduce it please run "accelerate launch code.py". I was wondering if you could have more suggestions for me? I do not have an idea how to fix this or debug this freezing. many thanks.
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
Maybe the `fsspec` stuff need to be clearer even before - can you try to run this function at the very beginning of your script ?
```python
import fsspec

def _set_fsspec_for_multiprocess() -> None:
    """
    Clear reference to the loop and thread.
    This is necessary otherwise HTTPFileSystem hangs in the ML training loop.
    Only required for fsspec >= 0.9.0
    See https://github.com/fsspec/gcsfs/issues/379
    """
    fsspec.asyn.iothread[0] = None
    fsspec.asyn.loop[0] = None

_set_fsspec_for_multiprocess()
```
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
Hi @lhoestq 
thank you. I tried it, I am getting `AttributeError: module 'fsspec' has no attribute 'asyn'`. which version of fsspect do you use?
I am using  
```fsspec                    2022.8.2                 pypi_0    pypi```
thank you.
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
Are you using `Pytorch 1.11`? Otherwise the script freezes because of the shuffling in this line: 
```
        return ShufflerIterDataPipe(self, buffer_size=buffer_size)
```
`ShufflerIterDataPipe` behavior must have changed for newer Pytorch versions. But this doesn't change whether you're using streaming or not in `datasets`, so probably not the same issue, but something to try.
Issue: datasets freezes with streaming mode in multiple-gpu

Comment:
> Are you using `Pytorch 1.11`? Otherwise the script freezes because of the shuffling in this line:
> 
> ```
>         return ShufflerIterDataPipe(self, buffer_size=buffer_size)
> ```
> 
> `ShufflerIterDataPipe` behavior must have changed for newer Pytorch versions. But this doesn't change whether you're using streaming or not in `datasets`, so probably not the same issue, but something to try.

I met the same issue for pytorch 1.12 and 1.13, is there a way to work around for this function for newer pytorch versions?
Issue: Progress bars have color red and never completed to 100%

Comment:
Hi @echatzikyriakidis, thanks for submitting the issue.
Which shell are you using exactly? I tried to run the command you sent, but I don't see colors at all 🧐

I tried from bash and zsh as well.
Issue: Progress bars have color red and never completed to 100%

Comment:
Got it. I [created a PR](https://github.com/huggingface/datasets/pull/5120) that fixes this issue. Turns out that the wrapping logic for the inner loop was slightly incorrect.
Issue: load_from_disk with remote filesystem fails due to a wrong temporary local folder path

Comment:
Hi Hubert! Could you please probably create a publicly available `gs://` dataset link? I think this would be easier for others to directly start to debug.
Issue: Bug with filtered indices

Comment:
Thanks a lot for the swift response! For a brief moment yesterday I thought I had gone insane 🤣On 14 Oct 2022, at 15:44, Albert Villanova del Moral ***@***.***> wrote:﻿
@PartiallyTyped, @Muennighoff: the issue is fixed.
We are planning to make a patch release today.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
Issue: map and filter not working properly in multiprocessing with the new release 2.6.0

Comment:
Thanks for reporting, @loubnabnl and for the additional information, @PartiallyTyped.

However, I'm not able to reproduce this issue, neither locally nor on Colab:
```
Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 10
})
Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 10
})
```
CC: @huggingface/datasets can anybody reproduce this?
Issue: map and filter not working properly in multiprocessing with the new release 2.6.0

Comment:
This is the minimum reproducible example. I ran this on the premium instances of colab.

```
# !pip install datasets
import datasets
from datasets import load_dataset
ds = load_dataset("copenlu/answerable_tydiqa").filter("english".__eq__, input_columns="language")
assert all(map("english".__eq__, ds["train"]["language"]))
```

In my case, the number of samples is correct, however, the samples selected when indexing are wrong.

```python
DatasetDict({
    validation: Dataset({
        features: ['question_text', 'document_title', 'language', 'annotations', 'document_plaintext', 'document_url'],
        num_rows: 990
    })
    train: Dataset({
        features: ['question_text', 'document_title', 'language', 'annotations', 'document_plaintext', 'document_url'],
        num_rows: 7389
    })
})
```

The number of rows is indeed correct, and i have checked it with a version that works.
Issue: map and filter not working properly in multiprocessing with the new release 2.6.0

Comment:
I can reproduce the issue on my mac too 
```
- `datasets` version: 2.6.0
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3
```
But not on Colab with python 3.7, maybe related to python version? (didn't manage to install python 3.9)
```
- `datasets` version: 2.6.0
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
```
Issue: map and filter not working properly in multiprocessing with the new release 2.6.0

Comment:
I think there are 2 different issues here:
- the one reported by @loubnabnl is related to multiprocessing in map and then filter; we should reproduce it first: I have tried with Python version 3.9.7 and I can't reproduce it either; maybe it is related to the version of PyArrow? To be checked.
- the issue reported by @PartiallyTyped is related just to "filter" (without multiprocessing) and I can reproduce it.
Issue: map and filter not working properly in multiprocessing with the new release 2.6.0

Comment:
Could you create another issue for the @PartiallyTyped one please ?

Regarding the OP issue, I also tried on colab or locally on py3.7 or py3.10 but didn't reproduce
Issue: map and filter not working properly in multiprocessing with the new release 2.6.0

Comment:
I managed to reproduce your issue @loubnabnl on colab by upgrading pyarrow to 9.0.0 instead of 6.0.1
Issue: map and filter not working properly in multiprocessing with the new release 2.6.0

Comment:
I managed to have a _super_ minimal reproducible example:
```python

from datasets import Dataset, concatenate_datasets

ds = concatenate_datasets([Dataset.from_dict({"a": [i]}) for i in range(10)])
ds2 = ds.map(lambda _: {}, batched=True)
assert list(ds2) == list(ds)
```
(filter uses a batched `map` under the hood)
Issue: map and filter not working properly in multiprocessing with the new release 2.6.0

Comment:
>  the one reported by @loubnabnl is related to multiprocessing in map and then filter; we should reproduce it first: I have tried with Python version 3.9.7 and I can't reproduce it either; maybe it is related to the version of PyArrow? To be checked.

So finally it was related to PyArrow version! :+1:
Issue: Map caching not working for some class methods

Comment:
The hash used for caching is computed by pickling recursively the function passed to `map`. Maybe some objects don't have the same hash across sessions.  In particular you can check the hash of your model using
```python
from datasets.fingerprint import Hasher
obj = AutoModel.from_config(config=config, add_pooling_layer=False)
print(Hasher.hash(obj))
```

You can find mode info here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument
Issue: Map caching not working for some class methods

Comment:
Indeed, the hash is changing. The `dumps` function serialize the model object in different ways because the model object is not deterministic
```python
from datasets.utils.py_utils import dumps
obj1 = AutoModel.from_config(config=config, add_pooling_layer=False)
obj2 = AutoModel.from_config(config=config, add_pooling_layer=False)

dumps(bert) == dumps(bert2). # False
```

> You can find mode info here: https://huggingface.co/docs/datasets/about_cache
> 
> You can also provide your own unique hash in map if you want, with the new_fingerprint argument


Thanks, the doc is so helpful. Indeed, we can fix the hash and get cache hit using `new_fingerprint`. Closing the issue.
Issue: Specifying an exisiting folder in download_and_prepare deletes everything in it

Comment:
Thanks for reporting, @cakiki.

I would say the deletion of the dir is an expected behavior though...
Issue: Specifying an exisiting folder in download_and_prepare deletes everything in it

Comment:
Thank you both for your feedback!

@albertvillanova I think I might have have the wrong mental model of what the function was meant to do. I thought it would be an API similar to the pandas `to_XX` write methods (Like the one @lhoestq mentions) so I just assumed it would download the dataframe to whichever folder I specififed (`"./"` in my case) so I could load it into a dask dataframe. I absolutely did not expect it to delete everything in my local directory, including the script where I called it from :smile: 

I think Quentin's proposed solution sounds like a reasonable feature!
Issue: Specifying an exisiting folder in download_and_prepare deletes everything in it

Comment:
actually there's already a `download_mode` parameter that defaults to `REUSE_DATASET_IF_EXISTS` - so I guess it's just a matter of not deleting files unrelated to the dataset, and to overwrite existing dataset files if the download mode is `REUSE_CACHE_IF_EXISTS` or `FORCE_REDOWNLOAD`
Issue: datasets doesn't support # in data paths

Comment:
`datasets` doesn't seem to urlencode the directory names here

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/file_utils.py#L109-L111

for example we should have
```python
from datasets.utils.file_utils import hf_hub_url

url = hf_hub_url("loubnabnl/bigcode_csharp", "data/c#/data_0003.jsonl")
print(url)
# Currently returns
# https://huggingface.co/datasets/loubnabnl/bigcode_csharp/resolve/main/data/c#/data_0003.jsonl
# while it should be 
# https://huggingface.co/datasets/loubnabnl/bigcode_csharp/resolve/main/data/c%23/data_0003.jsonl
```
Issue: datasets doesn't support # in data paths

Comment:
@loubnabnl The dataset you linked in the description of the bug does not work and returns a 404. Where can I find the dataset to reproduce the bug?
Issue: datasets doesn't support # in data paths

Comment:
I think you can create a dataset repository on the Hub with a dummy file containing a `#`
Issue: datasets doesn't support # in data paths

Comment:
Ah sorry it was private I just made it public, I can also help with this if needed
Issue: datasets doesn't support # in data paths

Comment:
@lhoestq Should I url encode also repo_id and revision parameters? I'm not sure what are the valid characters there.

Personally, I would be cautious and only url encode the path parameter.
Issue: datasets doesn't support # in data paths

Comment:
These are possible solutions (assuming `from urllib.parse import quote`):

1) url encode only the path parameter:
```
# src/datasets/utils/file_utils.py
def hf_hub_url(repo_id: str, path: str, revision: Optional[str] = None) -> str:
    revision = revision or config.HUB_DEFAULT_VERSION
    return config.HUB_DATASETS_URL.format(repo_id=repo_id, path=quote(path), revision=revision)
```
2) url encode all parameters:
```
# src/datasets/utils/file_utils.py
def hf_hub_url(repo_id: str, path: str, revision: Optional[str] = None) -> str:
    revision = revision or config.HUB_DEFAULT_VERSION
    return config.HUB_DATASETS_URL.format(repo_id=quote(repo_id), path=quote(path), revision=quote(revision))
```
3) url encode the whole url:
```
# src/datasets/config.py
HUB_DATASETS_PATH = "/datasets/{repo_id}/resolve/{revision}/{path}"
HUB_DATASETS_URL = HF_ENDPOINT + HUB_DATASETS_PATH
```
```
# src/datasets/utils/file_utils.py
def hf_hub_url(repo_id: str, path: str, revision: Optional[str] = None) -> str:
    revision = revision or config.HUB_DEFAULT_VERSION
    return config.HF_ENDPOINT + quote(config.HUB_DATASETS_PATH.format(repo_id=repo_id, path=path, revision=revision))
```
Issue: datasets doesn't support # in data paths

Comment:
repo_id can only contain alphanumeric characters and _- so it doesn't need to be encoded.

However I agree it's a good idea to also apply `quote` to the revision as well as in 2. !
Issue: Classes label error when loading symbolic links using imagefolder

Comment:
Hi, thanks for reporting and suggesting a fix! We still need to account for `.`/`..` in the file path, so a more robust fix would be `Path(os.path.abspath(filepath))`.
Issue: Classes label error when loading symbolic links using imagefolder

Comment:
> Hi, thanks for reporting and suggesting a fix! We still need to account for `.`/`..` in the file path, so a more robust fix would be `Path(os.path.abspath(filepath))`.

Thanks for your reply!