SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Reduce friction in tabular dataset workflow by eliminating having splits when dataset is loaded

Comment:
> We can raise an error if someone does load_dataset(...)[0] if the dataset is made of several splits,

But then how is that different to have the distinction between DatasetDict and Dataset then? Is it just that "default behaviour when there are no splits or single split, it returns directly the split when there's no ambiguity".

Also I was wondering how the concatenation could have heavy impacts when running mapping functions/filtering in batch? Typically can batch be somehow mixed?
Issue: Reduce friction in tabular dataset workflow by eliminating having splits when dataset is loaded

Comment:
> But then how is that different to have the distinction between DatasetDict and Dataset then?

Because it doesn't make sense to be able to do `example = ds[0]` or `examples = list(ds)` on a class named `DatasetDict` of type `Dict[str, Dataset]`.

> Also I was wondering how the concatenation could have heavy impacts when running mapping functions/filtering in batch? Typically can batch be somehow mixed?

No, we run each function on each split separated
Issue: Reduce friction in tabular dataset workflow by eliminating having splits when dataset is loaded

Comment:
> Because it doesn't make sense to be able to do example = ds[0] or examples = list(ds) on a class named DatasetDict of type Dict[str, Dataset].

Hum but you're still going to raise an exception in both those cases with your current change no? (actually list(ds) would return the name of the splits no?)

> No, we run each function on each split separated

Nice!
Issue: Incorrect error message when Dataset.from_sql fails and sqlalchemy not installed

Comment:
Hi! The first `Dataset.from_sql` call also outputs the "ImportError: Using URI string without sqlalchemy installed." message, but you also get "During handling of the above exception another exception occurred: ..." after which the ValueError is printed. I agree that this behavior makes it easy to miss the original error. 

I think we can improve this by not throwing the writer's ValueError if the error from a dataset script is already being handled to make debugging easier. @lhoestq @albertvillanova wdyt?
Issue: Incorrect error message when Dataset.from_sql fails and sqlalchemy not installed

Comment:
Yup ! Alternatively the error can be raised in sql.py before generating the examples ? In `_info` for example
Issue: Incorrect error message when Dataset.from_sql fails and sqlalchemy not installed

Comment:
yea @lhoestq that would probably be good. The 2nd error is useless if the 1st error is the real reason it failed.
Issue: Add notebook / other resource links to the task-specific data loading guides

Comment:
Yea this would be great! We would need an object detection tutorial notebook too if it doesn't already exist there.
Issue: Add a guide for semantic segmentation

Comment:
I am considering `torchvision` and `albumentations`. Also [works with TensorFlow](https://github.com/deep-diver/segformer-tf-transformers/blob/main/notebooks/TFSegFormer_Finetune.ipynb). 

I am assigning the issue to myself then.
Issue: An example or recommendations for creating large image datasets?

Comment:
The beam utilities allow to prepare a dataset as parquet in your cloud storage. From my perspective this CLI is not super easy to use, but we've been working on a new python API to prepare a dataset in your cloud storage:
```python
from datasets import load_dataset_builder

builder = load_dataset_builder("c4", "en")
builder.download_and_prepapre("s3://my-bucket/c4", file_format="parquet")
```

And to use Beam you can do:
```python
beam_runner = ...  # one of "SparkRunner", "DataFlowRunner", "DirectRunner", etc.
beam_options = ...

builder.download_and_prepapre(
    "s3://my-bucket/c4",
    file_format="parquet",
    beam_runner=beam_runner,
    beam_options=beam_options
)
```

Though Beam can be used ONLY if there is a dataset script based on the `BeamBasedBuilder` right now - it doesn't work on an arbitrary dataset (see [wikipedia.py](https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py) for example).
Issue: `map()` fails midway due to format incompatibility 

Comment:
You can end up with a list instead of a tensor if all the tensors inside the list can't be stacked together - can you make sure all your inputs are tensors with the same shape ?
Issue: `map()` fails midway due to format incompatibility 

Comment:
I modified my `tokenize()` function to be like so:

```py
def tokenize(batch):
    return tokenizer(batch["text"], padding="longest")
```

so that the padding always happens w.r.t to the length of the longest sequence in a batch. The issue still persists. Is there any other way?
Issue: `map()` fails midway due to format incompatibility 

Comment:
tbh I though your first implementation was fine
```python
def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True)
```

Maybe you can try to see what the erroring data looks like by adding a try/except in `get_test_accuracy` ?
Issue: `map()` fails midway due to format incompatibility 

Comment:
If you use the same batch_size for your two maps, you should get the exact same batches - therefore all containing the same shapes
Issue: Unable to download the Chinese `wikipedia`, the dumpstatus.json not found!

Comment:
Hi, @beyondguo, thanks for reporting.

You have all the information in the dataset card: https://huggingface.co/datasets/wikipedia

> Then, you can load any subset of Wikipedia per language and per date this way:
> ```python
> from datasets import load_dataset
> 
> load_dataset("wikipedia", language="sw", date="20220120", beam_runner=...) 
> ```
> where you can pass as beam_runner any Apache Beam supported runner for (distributed) data processing (see [here](https://beam.apache.org/documentation/runners/capability-matrix/)). Pass "DirectRunner" to run it on your machine.
> 
> You can find the full list of languages and dates [here](https://dumps.wikimedia.org/backup-index.html).

Note that you have to pass the language and date as keyword arguments, and the available dates depend on the language and can be found on Wikimedia website.
Issue: Unable to download the Chinese `wikipedia`, the dumpstatus.json not found!

Comment:
Also:
> Some subsets of Wikipedia have already been processed by HuggingFace, and you can load them just with:
> ```python
> load_dataset("wikipedia", "20220301.en")
> ```
> The list of pre-processed subsets is:
> - "20220301.de"
> - "20220301.en"
> - "20220301.fr"
> - "20220301.frr"
> - "20220301.it"
> - "20220301.simple"
Issue: prepare dataset for cloud storage doesn't work

Comment:
It looks like an issue with `gcsfs`, are you able to instantiate a `GCSFileSystem` manually ?
Issue: Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6

Comment:
Thanks for reporting, @Rijgersberg.

We were waiting for the release of `dill` 0.3.6, that happened 2 days ago (24 Oct 2022): https://github.com/uqfoundation/dill/releases/tag/dill-0.3.6
- See comment: https://github.com/huggingface/datasets/pull/4397#discussion_r880629543

Also `multiprocess` 0.70.14 was released 2 days ago: https://github.com/uqfoundation/multiprocess/releases/tag/multiprocess-0.70.14

We are addressing this issue to align dependencies.
Issue: Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6

Comment:
In your specific setup, I guess the compatible configuration is with `multiprocess` 0.70.13 (instead of 0.70.14).
Issue: Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6

Comment:
> @Rijgersberg this issue is fixed. It will be available in our next `datasets` release.

Any chance you have a eta?
Issue: Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6

Comment:
@Rijgersberg, please also that you can make `pip-compile` work by using the backtracking resolver (instead of the legacy one): https://pip-tools.readthedocs.io/en/latest/#a-note-on-resolvers
```
pip-compile --resolver=backtracking requirements.in
```
This resolver will automatically use `multiprocess` 0.70.13 version.
Issue: Automatically add filename for image/audio folder

Comment:
I'm fine with adding a new column with the file name personally. Not sure how breaking this is though
Issue: Automatically add filename for image/audio folder

Comment:
@patrickvonplaten do you mean just filename or full relative path inside the repo?
I think it shouldn't be breaking, at least I cannot come up with any case where it is. Maybe @mariosasko can?

also I think that the problem here and in general is that Image/AudioFolder has default configuration which implies automatic label creation if there is not metadata file. It can be changed when you load the dataset with `load_dataset` but not on it's Hub page.
Issue: Automatically add filename for image/audio folder

Comment:
> also I think that the problem here and in general Image/AudioFolder has default configuration which implies automatic label creation if there is not metadata file

Yea I agree it's often the wrong default. We can also imagine adding the builder's parameters as YAML in the repo.
Issue: Automatically add filename for image/audio folder

Comment:
@lhoestq yes I also got the idea of some YAML config! not sure of what priority it is though.
Issue: Automatically add filename for image/audio folder

Comment:
I meant just the file name (no path) that would already be super helpful IMO :-) (maybe dir+filename if there are dirs in the folder)
Issue: Automatically add filename for image/audio folder

Comment:
@patrickvonplaten one more time, to be sure I understand you.
For example, we have data structure like this:
```
├─ data/
│  └─ subdir/
│     └── cats/
│          ├── 0.jpg
│          ├── 1.jpg
│          └── 2.jpg
│     └── dogs/
│          ├── 0.jpg
│          ├── 1.jpg
│          └── 2.jpg
└── another_subdir/
      ├── 10.jpg
      ├── 11.jpg
      └── 12.jpg
```
Is it okay to provide `"data/subdir/cats/0.jpg"`, `"data/subdir/dogs/0.jpg"`, `"data/another_subdir/10.jpg"`?
I think providing just filenames might be confusing if they are not unique, as in this example.
Issue: Fix language and license tag names in all Hub datasets

Comment:
Hi @ayushthe1, thanks for your offer.

But as you can see, I self-assigned this issue.

I have already fixed 200 out of the 402 datasets. My script is still running and fixing the rest.

For example: https://huggingface.co/datasets/fhamborg/news_sentiment_newsmtsc/discussions/2/files
Issue: Consistent caching between python and jupyter

Comment:
Hi ! Maybe it's possible to have a consistent hash for a function defined in `__main__` and a function define in a notebook.

However for functions imported from another location, pickle uses the location to identify the code, so in that case we can't do much I believe.

Would it be ok for you if we only try to do this for functions in `__main__` / jupyter ?

If you'd like to contribute, you can read this part of the code and let me know if you have questions:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/py_utils.py#L617-L643

I think the key here would be to also ignore the "co_filename" of functions defined in `__main__`
Issue: Consistent caching between python and jupyter

Comment:
Seems like a good solution, I will start a PR and see if I understood the changes needed. Thanks!
Issue: Unable to download dataset using Azure Data Lake Gen 2

Comment:
Hi ! From the `adlfs` docs, there are two filesystems you can use:
> To use the Gen1 filesystem:
> - known_implementations[‘adl’] = {‘class’: ‘adlfs.AzureDatalakeFileSystem’}
> 
> To use the Gen2 filesystem:
>  - known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}

If I'm not mistaken you're using the second one - so you should use `abfs://` instead of `adl://`, and also run this at the beginning of your script:
```python
from fsspec.registry import known_implementations
known_implementations['abfs'] = {'class': 'adlfs.AzureDatalakeFileSystem'}
```
Issue: Unable to download dataset using Azure Data Lake Gen 2

Comment:
Thank you @lhoestq . Great call.
Using the default class from `known_implementations` dict solved my problem
```
known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}
```
I'm closing this issue.
Issue: default Image/AudioFolder infers labels when there is no metadata files even if there is only one dir

Comment:
Makes sense! For the last structure, we could count the path segments (delimited by "/" for URLs and `os.sep`  for local paths) to ensure all inferred labels are on the same level. Otherwise, I think it's safe to assume they are meaningless and ignore them.
Issue: Problems after upgrading to 2.6.1

Comment:
Hi! I can't reproduce the error following these steps. Can you please provide a reproducible example?
Issue: Problems after upgrading to 2.6.1

Comment:
I faced the same issue:

### Repro
```
!pip install datasets==2.6.1
import datasets as Dataset
dataset = Dataset.from_pandas(dataframe)
dataset.save_to_disk(local)

!pip install datasets==2.5.2
import datasets as Dataset
dataset = Dataset.load_from_disk(local)
```
Issue: Problems after upgrading to 2.6.1

Comment:
I bumped into the issue too. @Lokiiiiii thanks for steps. I "solved" if for now by `pip install datasets>=2.6.1` everywhere.
Issue: Problems after upgrading to 2.6.1

Comment:
Hi all, 
I experienced the same issue. 
Please note that the pull request is related to the IMDB example provided in the doc, and is a fix for that, in that context, to make sure that people can follow the doc example and have a working system. 
It does not provide a fix for Datasets itself.
Issue: Problems after upgrading to 2.6.1

Comment:
im getting the same error.
- using the base AWS HF container that uses a datasets <2.
- updating the AWS HF container to use dataset 2.4
Issue: Problems after upgrading to 2.6.1

Comment:
Same here, running on our SageMaker pipelines. It's only happening for some but not all of our saved Datasets.
Issue: Problems after upgrading to 2.6.1

Comment:
I am also receiving this error on Sagemaker but not locally, I have noticed that this occurs when the `.dataset/` folder does not contain a single file like:

`dataset.arrow`

but instead contains multiple files like:

`data-00000-of-00002.arrow`
`data-00001-of-00002.arrow`

I think that it may have something to do with this recent PR that updated the behaviour of `dataset.save_to_disk` by introducing sharding: https://github.com/huggingface/datasets/pull/5268

For now I can get around this by forcing datasets==2.8.0 on machine that creates dataset and in the huggingface instance for training (by running this at the start of training script `os.system("pip install datasets==2.8.0")`)

To ensure the dataset is a single shard when saving the dataset locally:

```python3
dataset.flatten_indices().save_to_disk('path/to/dataset', num_shards=1)
```

 and then manually changing the name afterwards from `path/to/dataset/data-00000-of-00001.arrow` to `path/to/dataset/dataset.arrow` and updating the `path/to/dataset/state.json` to reflect this name change. i.e. by changing `state.json` to this:

```javascript
{
  "_data_files": [
    {
      "filename": "dataset.arrow"
    }
  ],
  "_fingerprint": "420086f0636f8727",
  "_format_columns": null,
  "_format_kwargs": {},
  "_format_type": null,
  "_output_all_columns": false,
  "_split": null
}
```
Issue: Cannot find the rvl_cdip dataset

Comment:
Hi, @santule.

We have transferred all dataset scripts from GitHub to the Hugging Face Hub: https://huggingface.co/datasets
- Concretely, you have "rvl_cdip" here: https://huggingface.co/datasets/rvl_cdip

To be able to load them, you should update your `datasets` library:
```
pip install -U datasets
```
Issue: Allow ignoring kwargs inside fn_kwargs during dataset.map's fingerprinting

Comment:
Hi ! In the `transformers` issue the object to not hash is a `Pool` - I think you can instantiate it inside your function instead of passing it as a parameter. It's good practice that your function and all its fn_kwargs are picklable, in case you want to parallelize `map` using `num_proc>1`

For the other case `def fn(example, verbose=False):` however, I agree it would be nice to let the user specify that "verbose" needs to be ignored.

Do you think providing a decorator could help ? Maybe
```python
@datasets.hashing.register(ignore_kwargs=["verbose"])
def func(example, verbose=False):
    ...
```
Issue: Allow ignoring kwargs inside fn_kwargs during dataset.map's fingerprinting

Comment:
Hi @lhoestq! Thanks for your response.

A `Pool` shouldn't be instantiated within the function, because there's a huge overhead in doing so. The main idea is that the same `Pool` should be used across all function calls. Parallel `map` is not helpful/desired in that specific scenario, because the heavy parallel computation is done by another lib (`pyctcdecode`, called within `transformer`'s model inference code).

But yes, it makes sense to be able to leverage parallel processing by just doing `num_proc>1` when possible.

Your decorator suggestions seems like a pretty clean API to me. I didn't find a `datasets.hashing` module though. Would it be created for this specific purpose? Any downsides in just using `datasets.fingerprint`?

And would `datasets.hashing.register` just add some metadata to `func` in your approach (so it could be inspected from `fingerprint_transform`)?

And looking to the `datasets.Dataset` API, `.filter` would also benefited from this.
Issue: Allow ignoring kwargs inside fn_kwargs during dataset.map's fingerprinting

Comment:
> Would it be created for this specific purpose? Any downsides in just using datasets.fingerprint?

This can also go in datasets.fingerprint indeed - but maybe datasets.hashing tells more about what the register function does (i.e. register this function to have a custom hashing) ?

> And would datasets.hashing.register just add some metadata to func in your approach (so it could be inspected from fingerprint_transform)?

Yup that's the idea :)

> And looking to the datasets.Dataset API, .filter would also benefited from this.

Indeed !

-----

If you would like to contribute this you can assign yourself to this issue by posting #self-assign
And of course if you have questions or if I can help, feel free to ping me !
Issue: Allow ignoring kwargs inside fn_kwargs during dataset.map's fingerprinting

Comment:
> This can also go in datasets.fingerprint indeed - but maybe datasets.hashing tells more about what the register function does (i.e. register this function to have a custom hashing) ?

Sure, it makes sense.

---

I don't plan to work on it right now, so I'll let it unassigned in case somebody wants to join. I'll get back at it as soon as possible though.
Issue: Dataset order is not deterministic with ZIP archives and `iter_files`

Comment:
Thanks for reporting ! The issue doesn't come from shuffling, but from `beans` row order not being deterministic:

https://huggingface.co/datasets/beans/blob/main/beans.py uses `dl_manager.iter_files` on ZIP archives and the file order doesn't seen to be deterministic and changes across machines
Issue: Dataset order is not deterministic with ZIP archives and `iter_files`

Comment:
This is still a bug, so I'd keep this one open if you don't mind ;)
Issue: Dataset order is not deterministic with ZIP archives and `iter_files`

Comment:
Besides the linked PR, to make the loading process fully deterministic, I believe we should also sort the data files [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L276) and [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L485) (e.g. fsspec's `LocalFileSystem.glob` relies on `os.scandir`, which yields the contents in arbitrary order). My concern is the overhead of these sorts... Maybe we could introduce a new flag to `load_dataset` similar to TFDS' [`shuffle_files`](https://www.tensorflow.org/datasets/determinism#determinism_when_reading) or sort only if the number of data files is small?
Issue: Dataset order is not deterministic with ZIP archives and `iter_files`

Comment:
We already return the result sorted at the end of `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository` if I'm not mistaken
Issue: Inconsistent documentation on map remove_columns

Comment:
Thanks for reporting, @zhaowei-wang-nlp.

You are right, the documentation is confusing on the behavior of `remove_columns`. We should better explain it.
Issue: Inconsistent documentation on map remove_columns

Comment:
I'm closing this issue because as @mariosasko pointed out, it is a duplicate of:
- #2343
Issue: DownloadManager Git LFS support

Comment:
Hey ! Actually it works, just pass the right URL ;)
The URL must be the one with “/resolve/”

e.g. https://huggingface.co/datasets/imagenet-1k/resolve/main/data/test_images.tar.gz

You can even pass a relative path to the dl_manager instead, like `dl_manager.download("data/test_images.tar.gz")`
Issue: Align task tags in dataset metadata

Comment:
I have opened PRs to fix the task_ids in all datasets within a namespace as well.

Working on task_categories...
Issue: Align task tags in dataset metadata

Comment:
For future reference: this fix had some complications

When trying to open a PR to fix the task tags, an exception was thrown if:
- the metadata contained "languages" or "licenses" (instead of "language" or "license")
- the metadata contained a non-valid language: `en-US` (instead of `en`), `no` (instead of `'no'`),...
- the metadata contained a non-valid license
- either `task_categories` or `task_ids` was not an array (a dict for each config)
- the metadata contained non-valid tag names

Errors:
```
ValueError: - Error: "languages" is deprecated. Use "language" instead.
```
```
ValueError: - Error: "licenses" is deprecated. Use "license" instead.
```
```
ValueError: - Error: "language[17]" must only contain lowercase characters
```
```
ValueError: - Error: "language[0]" with value "cz, de, it" is not valid. It must be an ISO 639-1, 639-2 or 639-3 code (two/three letters), or a special value like "code", "multilingual". If you want to use BCP-47 identifiers, you can specify them in language_bcp47.
```
```
ValueError: - Error: "task_ids" must be an array
```