helmo/github-issues huggingface dataset:
Datasets method `.map` not hashing
State: closed | Author: sanchit-gandhi (CONTRIBUTOR)
Labels: name: bug

## Describe the bug
Datasets method `.map` not hashing, even with an empty no-op function

## Steps to reproduce the bug
```python
from datasets import load_dataset

# download 9MB dummy dataset
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean")

def prepare_dataset(batch):
     return(batch)

ds = ds.map(
    prepare_dataset,
    num_proc=1,
    desc="preprocess train dataset",
)
```

## Expected results
Hashed and cached dataset preprocessing

## Actual results
Does not hash properly:
```
Parameter 'function'=<function prepare_dataset at 0x7fccb68e9280> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.11.0-1028-gcp-x86_64-with-glibc2.31
- Python version: 3.9.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.2

cc @lhoestq 


Comments: Fix posted: https://github.com/huggingface/datasets/issues/4506#issuecomment-1157417219

Didn't realize it's a bug when I asked the question yesterday! Free free to post an answer if you are sure the cause has been addressed.

https://stackoverflow.com/questions/72664827/can-pickle-dill-foo-but-not-lambda-x-foox

Thank @nalzok . That works for me:

`pip install "dill<0.3.5"`
Create new sections for audio and vision in guides
State: closed | Author: stevhliu (MEMBER)
Labels: name: documentation

This PR creates separate sections in the guides for audio, vision, text, and general usage so it is easier for users to find loading, processing, or sharing guides specific to the dataset type they're working with. It'll also allow us to scale the docs to additional dataset types - like time series, tabular, etc. - while keeping our docs information architecture. 

Some other changes include:

- ~Experimented with decorating text with some CSS to highlight guides specific to each modality. Hopefully, it'll be easier for users to find and realize that these different docs exist!~ Will experiment with this in a different PR.
- Added deprecation warning for Metrics and redirect to Evaluate.
- Updated `set_format` section to recommend using the new `to_tf_dataset` function if you need to convert to a TensorFlow dataset.
- Reorganized `toctree` to nest general usage, audio, vision, and text sections under the how-to guides.
- A quick review and edit to the Load and Process docs for clarity.

Comments: _The documentation is not available anymore as the PR was closed or merged._

Ready for review!

The `toctree` is a bit longer now with the sections. I think if we keep the audio/vision/text/dataset repository sections collapsed by default, and keep the general usage expanded, it may look a little cleaner and not as overwhelming. Let me know what you think! 😄
Add tags for task_ids:summarization-* and task_categories:summarization*
State: closed | Author: hobson (CONTRIBUTOR)

yaml header at top of README.md file was edited to add task tags because I couldn't find the existing tags in the json
separate Pull Request will modify dataset_infos.json to add these tags

The Enron dataset (dataset id aeslc) is only tagged with:

    arxiv:1906.03497'
    languages:en
    pretty_name:AESLC

Using the email subject_line field as a label or target variable it possible to create models for the following task_ids (in order of relevance):

    'task_ids:summarization'
    'task_ids:summarization-other-conversations-summarization'
    "task_ids:other-other-query-based-multi-document-summarization"
    'task_ids:summarization-other-aspect-based-summarization'
    'task_ids:summarization--other-headline-generation'

The subject might also be used for the task_category "task_categories:summarization"

E-mail chains might be used for the task category "task_categories:dialogue-system"

Comments: Associated community discussion is [here](https://huggingface.co/datasets/aeslc/discussions/1).
Paper referenced in the `dataset_infos.json` is [here](https://arxiv.org/pdf/1906.03497.pdf). It mentions the  _email-subject-generation_ task, which is not a tag mentioned in any other dataset so it was not added in this pull request. The _summarization_ task is mentioned as a related task.

_The documentation is not available anymore as the PR was closed or merged._
Fix hashing for python 3.9
State: closed | Author: lhoestq (MEMBER)

In python 3.9, pickle hashes the `glob_ids` dictionary in addition to the `globs` of a function.

Therefore the test at `tests/test_fingerprint.py::RecurseDumpTest::test_recurse_dump_for_function_with_shuffled_globals` is currently failing for python 3.9

To make hashing deterministic when the globals are not in the same order, we also need to make the order of `glob_ids` deterministic.

Right now we don't have a CI to test python 3.9 but we should definitely have one. For this PR in particular I ran the tests locally using python 3.9 and they're passing now.

Fix https://github.com/huggingface/datasets/issues/4506

Comments: _The documentation is not available anymore as the PR was closed or merged._

What do you think @albertvillanova ?
Add uppercased versions of image file extensions for automatic module inference
State: closed | Author: mariosasko (COLLABORATOR)

Adds the uppercased versions of the image file extensions to the supported extensions. 

Another approach would be to call `.lower()` on extensions while resolving data files, but uppercased extensions are not something we want to encourage out of the box IMO unless they are commonly used (as they are in the vision domain)

Note that there is a slight discrepancy between the image file resolution and `imagefolder` as the latter calls `.lower()` on file extensions leading to some image file extensions being ignored by the resolution but not by the loader (e.g. `pNg`). Such extensions should also be discouraged, so I'm ignoring that case too.

Fix #4514.  

Comments: _The documentation is not available anymore as the PR was closed or merged._
Allow .JPEG as a file extension
State: closed | Author: DiGyt (NONE)
Labels: name: bug

## Describe the bug
When loading image data, HF datasets seems to recognize `.jpg` and `.jpeg` file extensions, but not e.g. .JPEG. As the naming convention .JPEG is used in important datasets such as imagenet, I would welcome if according extensions like .JPEG or .JPG would be allowed.

## Steps to reproduce the bug
```python
# use bash to create 2 sham datasets with jpeg and JPEG ext
!mkdir dataset_a
!mkdir dataset_b
!wget https://upload.wikimedia.org/wikipedia/commons/7/71/Dsc_%28179253513%29.jpeg -O example_img.jpeg
!cp example_img.jpeg ./dataset_a/
!mv example_img.jpeg ./dataset_b/example_img.JPEG

from datasets import load_dataset

# working
df1 = load_dataset("./dataset_a", ignore_verifications=True)

#not working
df2 = load_dataset("./dataset_b", ignore_verifications=True)

# show
print(df1, df2)
```

## Expected results
```
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 1
    })
}) DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 1
    })
})
```

## Actual results
```
FileNotFoundError: Unable to resolve any data file that matches '['**']' at /..PATH../dataset_b with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

I know that it can be annoying to allow seemingly arbitrary numbers of file extensions. But I think this one would be really welcome.

Comments: Hi, thanks for reporting!  I've opened a PR with the fix.

Wow, that was quick! Thank you very much 🙏
Update Google Cloud Storage documentation and add Azure Blob Storage example
State: closed | Author: alvarobartt (MEMBER)
Labels: name: documentation

While I was going through the 🤗 Datasets documentation of the Cloud storage filesystems at https://huggingface.co/docs/datasets/filesystems, I realized that the Google Cloud Storage documentation could be improved e.g. bullet point says "Load your dataset" when the actual call was to "Save your dataset", in-line code comment was mentioning "s3 bucket" instead of "gcs bucket", and some more in-line comments could be included.

Also, I think that mixing Google Cloud Storage documentation with AWS S3's one was a little bit confusing, so I moved all those to the end of the document under an h2 tab named "Other filesystems", with an h3 for "Google Cloud Storage".

Besides that, I was currently working with Azure Blob Storage and found out that the URL to [adlfs](https://github.com/fsspec/adlfs) was common for both filesystems Azure Blob Storage and Azure DataLake Storage, as well as the URL, which was updated even though the redirect was working fine, so I decided to group those under the same row in the column of supported filesystems.

And took also the change to add a small documentation entry as for Google Cloud Storage but for Azure Blob Storage, as I assume that AWS S3, GCP Cloud Storage, and Azure Blob Storage, are the most used cloud storage providers.

Let me know if you're OK with these changes, or whether you want me to roll back some of those! :hugs:

Comments: _The documentation is not available anymore as the PR was closed or merged._

Hi @stevhliu, I've kept the `>>>` before all the in-line code comments as it was done like that in the default S3 example that was already there, I assume that it's done like that just for readiness, let me know whether we should remove the `>>>` in the Python blocks before the in-line code comments or keep them.

![image](https://user-images.githubusercontent.com/36760800/174254663-b68d28d2-eae1-40f3-8695-dc4b0c3b479a.png)


Comments are ignored by doctest, so I think we can remove the `>>>` :)

Cool I'll remove those now 👍🏻

Sure @lhoestq, I just kept that structure as that was the more similar one to the one that was already there, but we can go with that approach, just let me know whether I should change the headers so as to leave all those providers in the same level (`h2`). Thanks!
Support all negative values in ClassLabel
State: closed | Author: lhoestq (MEMBER)

We usually use -1 to represent a missing label, but we should also support any negative values (some users use -100 for example). This is a regression from `datasets` 2.3

Fix https://github.com/huggingface/datasets/issues/4508

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thanks for this fix! I'm not sure what the release timeline is, but FYI #4508 is a breaking issue for transformer token classification using Trainer and PyTorch. PyTorch defaults to -100 as the ignored label for [negative log loss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html?highlight=nllloss#torch.nn.NLLLoss), so switching labels to -1 leads to index errors using Trainer defaults.

As a workaround, I'm using master branch directly (`pip install git+https://github.com/huggingface/datasets.git@master` for anyone who needs to do the same) until this gets released.

The new release `2.4` fixes the issue, feel free to update `datasets` :) 
```
pip install -U datasets
```
Add regression test for `ArrowWriter.write_batch` when batch is empty
State: closed | Author: alvarobartt (MEMBER)

As spotted by @cccntu in #4502, there's a logic bug in `ArrowWriter.write_batch` as the if-statement to handle the empty batches as detailed in the docstrings of the function ("Ignores the batch if it appears to be empty, preventing a potential schema update of unknown types."), the current if-statement is not handling properly `writer.write_batch({})` as an error is triggered.

Also, if we add a regression test in `test_arrow_writer.py::test_write_batch` before applying the fix, the test will fail as when trying to write an empty batch as follows:

```
=================================================================================== short test summary info ===================================================================================
FAILED tests/test_arrow_writer.py::test_write_batch[None-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[None-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[None-10] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-10] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-10] - ValueError: Schema and number of arrays unequal
======================================================================== 9 failed, 73 deselected, 7 warnings in 0.81s =========================================================================
```

So the batch is not ignored when empty, as `batch_examples={}` won't match the condition `if batch_examples: ...`.

Comments: _The documentation is not available anymore as the PR was closed or merged._

As mentioned by @lhoestq, the current behavior is correct and we should not expect batches with different columns, in that case, the if should fail, as the values of the batch can be empty, but not the actual `batch_examples` value.
Support skipping Parquet to Arrow conversion when using Beam
State: closed | Author: albertvillanova (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4509). All of your documentation changes will be reflected on that endpoint.

When #4724 is merged, we can just pass `file_format="parquet"` to `download_and_prepare` and it will output parquet fiels without converting to arrow

I think we can close this one
cast_storage method from datasets.features
State: closed | Author: romainremyb (NONE)
Labels: name: bug

## Describe the bug
A bug occurs when mapping a function to a dataset object. I ran the same code with the same data yesterday and it worked just fine. It works when i run locally on an old version of datasets.

## Steps to reproduce the bug
Steps are:
- load whatever datset
- write a preprocessing function such as "tokenize_and_align_labels" written in https://huggingface.co/docs/transformers/tasks/token_classification
- map the function on dataset and get "ValueError: Class label -100 less than -1" from cast_storage method from datasets.features

# Sample code to reproduce the bug
def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True, max_length=38,padding="max_length")

    labels = []
    for i, label in enumerate(examples[f"labels"]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:  # Set the special tokens to -100.
            if word_idx is None:
                label_ids.append(-100)
            elif word_idx != previous_word_idx:  # Only label the first token of a given word.
                label_ids.append(label[word_idx])
            else:
                label_ids.append(-100)
            previous_word_idx = word_idx
        labels.append(label_ids)

    tokenized_inputs["labels"] = labels
    return tokenized_inputs

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
dt = dataset.map(tokenize_and_align_labels, batched=True)

## Expected results
New dataset objects should load and do on older versions.

## Actual results
"ValueError: Class label -100 less than -1" from cast_storage method from datasets.features

## Environment info
everything works fine on older installations of datasets/transformers

Issue arises when installing datasets on google collab under python3.7
I can't manage to find the exact output you're requirering but version printed is datasets-2.3.2


Comments: Hi! We've recently added a check to the `ClassLabel` type to ensure the values are in the valid label range `-1, 0, ..., num_classes-1` (-1 is used for missing values). The error in your case happens only if the `labels` column is of type `Sequence(ClassLabel(...))` before the `map` call and can be avoided by calling `dataset = dataset.cast_column("labels", Sequence(Value("int")))` beforehand. The token-classification examples in Transformers introduce a new `labels` column, so their type is also `Sequence(Value("int"))`, which doesn't lead to an error as this type unbounded. 

I'm fine with re-adding support for all negative values for unknown/missing labels @mariosasko, wdyt ?
How to let `load_dataset` return a `Dataset` instead of `DatasetDict` in customized loading script
State: closed | Author: liyucheng09 (NONE)
Labels: name: enhancement

If the dataset does not need splits, i.e., no training and validation split, more like a table. How can I let the `load_dataset` function return a `Dataset` object directly rather than return a `DatasetDict` object with only one key-value pair.

Or I can paraphrase the question in the following way: how to skip `_split_generators` step in `DatasetBuilder` to let `as_dataset` gives a single `Dataset` rather than a list`[Dataset]`?

Many thanks for any help.

Comments: Hi @liyucheng09.

Users can pass the `split` parameter to `load_dataset`. For example, if your split name is "train",
```python
ds = load_dataset("dataset_name", split="train")
```
will return a `Dataset` instance.

@albertvillanova Thanks! I can't believe I didn't know this feature till now.
Fix double dots in data files
State: closed | Author: lhoestq (MEMBER)

As mentioned in https://github.com/huggingface/transformers/pull/17715 `data_files` can't find a file if the path contains double dots `/../`. This has been introduced in https://github.com/huggingface/datasets/pull/4412, by trying to ignore hidden files and directories (i.e. if they start with a dot)

I fixed this and added a test

cc @sgugger @ydshieh 

Comments: _The documentation is not available anymore as the PR was closed or merged._

The CI fails are unrelated to this PR (apparently something related to `seqeval` on windows) - merging :)
Refactor and add metadata to fever dataset
State: closed | Author: albertvillanova (MEMBER)

Related to: #4452 and #3792.

Comments: _The documentation is not available anymore as the PR was closed or merged._

But this is somehow fever v3 dataset (see this link https://fever.ai/ under the dropdown menu called Datasets). Our fever dataset already contains v1 and v2 configs. Then, I added this as if v3 config (but named feverous instead of v3 to align with the original naming by data owners).

In any case, if you really think this should be a new dataset, then I would propose to create it on the Hub instead, as "fever/feverous".

> In any case, if you really think this should be a new dataset, then I would propose to create it on the Hub instead, as "fever/feverous".

Yea makes sense ! thanks :) let's push more datasets on the hub rather than on github from now on

I have added "feverous" dataset to the Hub: https://huggingface.co/datasets/fever/feverous

I change the name of this PR accordingly, as now it only:
- Refactors code and include for both Fever v1.0 and v2.0 specific:
  - Descriptions
  - Citations
  - Homepages
- Updates documentation card aligned with above:
  - It was missing v2.0 description and citation.
- Update metadata JSON
Add `concatenate_datasets` for iterable datasets
State: closed | Author: lhoestq (MEMBER)

`concatenate_datasets` currently only supports lists of `datasets.Dataset`, not lists of `datasets.IterableDataset` like `interleave_datasets`

Fix https://github.com/huggingface/datasets/issues/2564

I also moved `_interleave_map_style_datasets` from combine.py to arrow_dataset.py, since the logic depends a lot on the `Dataset` object internals

And I moved `concatenate_datasets` from arrow_dataset.py to combine.py to have it with `interleave_datasets` (though it's also copied in arrow_dataset module for backward compatibility for now)

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thanks ! I addressed your comments :)

> There is a slight difference in concatenate_datasets between the version for map-style datasets and the one for iterable datasets

Indeed, here is what I did to fix this:

- axis 0: fill missing columns with None.
(I first iterate over the input datasets to infer their columns from the first examples, then I set the features of the resulting dataset to be the merged features)
This is consistent with non-streaming concatenation

- axis 1: **fill the missing rows with None**, for consistency with axis 0
(but let me know what you think, I can still revert this behavior and raise an error when one of the dataset runs out of examples)
We might have to align the non-streaming concatenation with this behavior though, for consistency. What do you think ?

Added more comments as suggested, and some typing

While factorizing _apply_features_types for both IterableDataset and TypedExamplesIterable, I  fixed a missing `token_per_repo_id` that was not passed to TypedExamplesIteable

Let me know what you think now @mariosasko
fix ETT m1/m2 test/val dataset
State: closed | Author: kashif (CONTRIBUTOR)

https://huggingface.co/datasets/ett/discussions/1

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thansk for the fix ! Can you regenerate the datasets_infos.json please ? This way it will update the expected number of examples in the test and val splits

ah yes!