SurAyush/github_issues huggingface dataset:
Issue: Allow .JPEG as a file extension
State: closed
Author: DiGyt

## Describe the bug
When loading image data, HF datasets seems to recognize `.jpg` and `.jpeg` file extensions, but not e.g. .JPEG. As the naming convention .JPEG is used in important datasets such as imagenet, I would welcome if according extensions like .JPEG or .JPG would be allowed.

## Steps to reproduce the bug
```python
# use bash to create 2 sham datasets with jpeg and JPEG ext
!mkdir dataset_a
!mkdir dataset_b
!wget https://upload.wikimedia.org/wikipedia/commons/7/71/Dsc_%28179253513%29.jpeg -O example_img.jpeg
!cp example_img.jpeg ./dataset_a/
!mv example_img.jpeg ./dataset_b/example_img.JPEG

from datasets import load_dataset

# working
df1 = load_dataset("./dataset_a", ignore_verifications=True)

#not working
df2 = load_dataset("./dataset_b", ignore_verifications=True)

# show
print(df1, df2)
```

## Expected results
```
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 1
    })
}) DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 1
    })
})
```

## Actual results
```
FileNotFoundError: Unable to resolve any data file that matches '['**']' at /..PATH../dataset_b with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

I know that it can be annoying to allow seemingly arbitrary numbers of file extensions. But I think this one would be really welcome.

Comments:
Hi, thanks for reporting!  I've opened a PR with the fix.

Wow, that was quick! Thank you very much 🙏
Issue: Update Google Cloud Storage documentation and add Azure Blob Storage example
State: closed
Author: alvarobartt

While I was going through the 🤗 Datasets documentation of the Cloud storage filesystems at https://huggingface.co/docs/datasets/filesystems, I realized that the Google Cloud Storage documentation could be improved e.g. bullet point says "Load your dataset" when the actual call was to "Save your dataset", in-line code comment was mentioning "s3 bucket" instead of "gcs bucket", and some more in-line comments could be included.

Also, I think that mixing Google Cloud Storage documentation with AWS S3's one was a little bit confusing, so I moved all those to the end of the document under an h2 tab named "Other filesystems", with an h3 for "Google Cloud Storage".

Besides that, I was currently working with Azure Blob Storage and found out that the URL to [adlfs](https://github.com/fsspec/adlfs) was common for both filesystems Azure Blob Storage and Azure DataLake Storage, as well as the URL, which was updated even though the redirect was working fine, so I decided to group those under the same row in the column of supported filesystems.

And took also the change to add a small documentation entry as for Google Cloud Storage but for Azure Blob Storage, as I assume that AWS S3, GCP Cloud Storage, and Azure Blob Storage, are the most used cloud storage providers.

Let me know if you're OK with these changes, or whether you want me to roll back some of those! :hugs:

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @stevhliu, I've kept the `>>>` before all the in-line code comments as it was done like that in the default S3 example that was already there, I assume that it's done like that just for readiness, let me know whether we should remove the `>>>` in the Python blocks before the in-line code comments or keep them.

![image](https://user-images.githubusercontent.com/36760800/174254663-b68d28d2-eae1-40f3-8695-dc4b0c3b479a.png)


Comments are ignored by doctest, so I think we can remove the `>>>` :)

Cool I'll remove those now 👍🏻

Sure @lhoestq, I just kept that structure as that was the more similar one to the one that was already there, but we can go with that approach, just let me know whether I should change the headers so as to leave all those providers in the same level (`h2`). Thanks!
Issue: Support all negative values in ClassLabel
State: closed
Author: lhoestq

We usually use -1 to represent a missing label, but we should also support any negative values (some users use -100 for example). This is a regression from `datasets` 2.3

Fix https://github.com/huggingface/datasets/issues/4508

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for this fix! I'm not sure what the release timeline is, but FYI #4508 is a breaking issue for transformer token classification using Trainer and PyTorch. PyTorch defaults to -100 as the ignored label for [negative log loss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html?highlight=nllloss#torch.nn.NLLLoss), so switching labels to -1 leads to index errors using Trainer defaults.

As a workaround, I'm using master branch directly (`pip install git+https://github.com/huggingface/datasets.git@master` for anyone who needs to do the same) until this gets released.

The new release `2.4` fixes the issue, feel free to update `datasets` :) 
```
pip install -U datasets
```
Issue: Add regression test for `ArrowWriter.write_batch` when batch is empty
State: closed
Author: alvarobartt

As spotted by @cccntu in #4502, there's a logic bug in `ArrowWriter.write_batch` as the if-statement to handle the empty batches as detailed in the docstrings of the function ("Ignores the batch if it appears to be empty, preventing a potential schema update of unknown types."), the current if-statement is not handling properly `writer.write_batch({})` as an error is triggered.

Also, if we add a regression test in `test_arrow_writer.py::test_write_batch` before applying the fix, the test will fail as when trying to write an empty batch as follows:

```
=================================================================================== short test summary info ===================================================================================
FAILED tests/test_arrow_writer.py::test_write_batch[None-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[None-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[None-10] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-10] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-10] - ValueError: Schema and number of arrays unequal
======================================================================== 9 failed, 73 deselected, 7 warnings in 0.81s =========================================================================
```

So the batch is not ignored when empty, as `batch_examples={}` won't match the condition `if batch_examples: ...`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As mentioned by @lhoestq, the current behavior is correct and we should not expect batches with different columns, in that case, the if should fail, as the values of the batch can be empty, but not the actual `batch_examples` value.
Issue: Support skipping Parquet to Arrow conversion when using Beam
State: closed
Author: albertvillanova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4509). All of your documentation changes will be reflected on that endpoint.

When #4724 is merged, we can just pass `file_format="parquet"` to `download_and_prepare` and it will output parquet fiels without converting to arrow

I think we can close this one
Issue: cast_storage method from datasets.features
State: closed
Author: romainremyb

## Describe the bug
A bug occurs when mapping a function to a dataset object. I ran the same code with the same data yesterday and it worked just fine. It works when i run locally on an old version of datasets.

## Steps to reproduce the bug
Steps are:
- load whatever datset
- write a preprocessing function such as "tokenize_and_align_labels" written in https://huggingface.co/docs/transformers/tasks/token_classification
- map the function on dataset and get "ValueError: Class label -100 less than -1" from cast_storage method from datasets.features

# Sample code to reproduce the bug
def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True, max_length=38,padding="max_length")

    labels = []
    for i, label in enumerate(examples[f"labels"]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:  # Set the special tokens to -100.
            if word_idx is None:
                label_ids.append(-100)
            elif word_idx != previous_word_idx:  # Only label the first token of a given word.
                label_ids.append(label[word_idx])
            else:
                label_ids.append(-100)
            previous_word_idx = word_idx
        labels.append(label_ids)

    tokenized_inputs["labels"] = labels
    return tokenized_inputs

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
dt = dataset.map(tokenize_and_align_labels, batched=True)

## Expected results
New dataset objects should load and do on older versions.

## Actual results
"ValueError: Class label -100 less than -1" from cast_storage method from datasets.features

## Environment info
everything works fine on older installations of datasets/transformers

Issue arises when installing datasets on google collab under python3.7
I can't manage to find the exact output you're requirering but version printed is datasets-2.3.2


Comments:
Hi! We've recently added a check to the `ClassLabel` type to ensure the values are in the valid label range `-1, 0, ..., num_classes-1` (-1 is used for missing values). The error in your case happens only if the `labels` column is of type `Sequence(ClassLabel(...))` before the `map` call and can be avoided by calling `dataset = dataset.cast_column("labels", Sequence(Value("int")))` beforehand. The token-classification examples in Transformers introduce a new `labels` column, so their type is also `Sequence(Value("int"))`, which doesn't lead to an error as this type unbounded. 

I'm fine with re-adding support for all negative values for unknown/missing labels @mariosasko, wdyt ?
Issue: How to let `load_dataset` return a `Dataset` instead of `DatasetDict` in customized loading script
State: closed
Author: liyucheng09

If the dataset does not need splits, i.e., no training and validation split, more like a table. How can I let the `load_dataset` function return a `Dataset` object directly rather than return a `DatasetDict` object with only one key-value pair.

Or I can paraphrase the question in the following way: how to skip `_split_generators` step in `DatasetBuilder` to let `as_dataset` gives a single `Dataset` rather than a list`[Dataset]`?

Many thanks for any help.

Comments:
Hi @liyucheng09.

Users can pass the `split` parameter to `load_dataset`. For example, if your split name is "train",
```python
ds = load_dataset("dataset_name", split="train")
```
will return a `Dataset` instance.

@albertvillanova Thanks! I can't believe I didn't know this feature till now.
Issue: Fix double dots in data files
State: closed
Author: lhoestq

As mentioned in https://github.com/huggingface/transformers/pull/17715 `data_files` can't find a file if the path contains double dots `/../`. This has been introduced in https://github.com/huggingface/datasets/pull/4412, by trying to ignore hidden files and directories (i.e. if they start with a dot)

I fixed this and added a test

cc @sgugger @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The CI fails are unrelated to this PR (apparently something related to `seqeval` on windows) - merging :)
Issue: Refactor and add metadata to fever dataset
State: closed
Author: albertvillanova

Related to: #4452 and #3792.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

But this is somehow fever v3 dataset (see this link https://fever.ai/ under the dropdown menu called Datasets). Our fever dataset already contains v1 and v2 configs. Then, I added this as if v3 config (but named feverous instead of v3 to align with the original naming by data owners).

In any case, if you really think this should be a new dataset, then I would propose to create it on the Hub instead, as "fever/feverous".

> In any case, if you really think this should be a new dataset, then I would propose to create it on the Hub instead, as "fever/feverous".

Yea makes sense ! thanks :) let's push more datasets on the hub rather than on github from now on

I have added "feverous" dataset to the Hub: https://huggingface.co/datasets/fever/feverous

I change the name of this PR accordingly, as now it only:
- Refactors code and include for both Fever v1.0 and v2.0 specific:
  - Descriptions
  - Citations
  - Homepages
- Updates documentation card aligned with above:
  - It was missing v2.0 description and citation.
- Update metadata JSON
Issue: Add `concatenate_datasets` for iterable datasets
State: closed
Author: lhoestq

`concatenate_datasets` currently only supports lists of `datasets.Dataset`, not lists of `datasets.IterableDataset` like `interleave_datasets`

Fix https://github.com/huggingface/datasets/issues/2564

I also moved `_interleave_map_style_datasets` from combine.py to arrow_dataset.py, since the logic depends a lot on the `Dataset` object internals

And I moved `concatenate_datasets` from arrow_dataset.py to combine.py to have it with `interleave_datasets` (though it's also copied in arrow_dataset module for backward compatibility for now)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks ! I addressed your comments :)

> There is a slight difference in concatenate_datasets between the version for map-style datasets and the one for iterable datasets

Indeed, here is what I did to fix this:

- axis 0: fill missing columns with None.
(I first iterate over the input datasets to infer their columns from the first examples, then I set the features of the resulting dataset to be the merged features)
This is consistent with non-streaming concatenation

- axis 1: **fill the missing rows with None**, for consistency with axis 0
(but let me know what you think, I can still revert this behavior and raise an error when one of the dataset runs out of examples)
We might have to align the non-streaming concatenation with this behavior though, for consistency. What do you think ?

Added more comments as suggested, and some typing

While factorizing _apply_features_types for both IterableDataset and TypedExamplesIterable, I  fixed a missing `token_per_repo_id` that was not passed to TypedExamplesIteable

Let me know what you think now @mariosasko
Issue: fix ETT m1/m2 test/val dataset
State: closed
Author: kashif

https://huggingface.co/datasets/ett/discussions/1

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thansk for the fix ! Can you regenerate the datasets_infos.json please ? This way it will update the expected number of examples in the test and val splits

ah yes!
Issue: WER and CER > 1
State: closed
Author: sadrasabouri

## Describe the bug
It seems that in some cases in which the `prediction` is longer than the `reference` we may have word/character error rate higher than 1 which is a bit odd.

If it's a real bug I think I can solve it with a PR changing [this](https://github.com/huggingface/datasets/blob/master/metrics/wer/wer.py#L105) line to
```python
return min(incorrect / total, 1.0)
```

## Steps to reproduce the bug
```python
from datasets import load_metric
wer = load_metric("wer")
wer_value = wer.compute(predictions=["Hi World vka"], references=["Hello"])
print(wer_value)
```

## Expected results
```
1.0
```

## Actual results
```
3.0
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

Comments:
WER can have values bigger than 1.0, this is expected when there are too many insertions

From [wikipedia](https://en.wikipedia.org/wiki/Word_error_rate):
> Note that since N is the number of words in the reference, the word error rate can be larger than 1.0
Issue: Re-add download_manager module in utils
State: closed
Author: lhoestq

https://github.com/huggingface/datasets/pull/4384 moved `datasets.utils.download_manager` to `datasets.download.download_manager`

This breaks `evaluate` which imports `DownloadMode` from `datasets.utils.download_manager` 

This PR re-adds `datasets.utils.download_manager` without circular imports.

We could also show a message that says that accessing it is deprecated, but I think we can do this in a subsequent PR, and just focus on doing a patch release for now

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the fix.

I'm wondering how this fixes backward compatibility...

Executing this code:
```python
from datasets.utils.download_manager import DownloadMode
```
we will have
```python
DownloadMode = None
```

If afterwards we use something like:
```python
if download_mode == DownloadMode.FORCE_REDOWNLOAD
```
that will raise an exception.

It works fine on my side:
```python
>>> from datasets.utils.download_manager import DownloadMode
>>> DownloadMode is not None
True
```

As reported in https://github.com/huggingface/evaluate/pull/143
```python
from datasets.utils import DownloadConfig
```
is also missing, I'm re-adding it

Took the liberty of merging this one, to do a patch release soon. If we think of a better approach we can improve it later
Issue: Replace `assertEqual` with `assertTupleEqual` in unit tests for verbosity
State: closed
Author: alvarobartt

As detailed in #4419 and as suggested by @mariosasko, we could replace the `assertEqual` assertions with `assertTupleEqual` when the assertion is between Tuples, in order to make the tests more verbose.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

FYI I used the following regex to look for the `assertEqual` statements where the assertion was being done over a Tuple: `self.assertEqual(.*, \(.*,)(\)\))$`, hope this is useful!
Issue: Fix patching module that doesn't exist
State: closed
Author: lhoestq

Reported in https://github.com/huggingface/huggingface_hub/runs/6894703718?check_suite_focus=true

When trying to patch `scipy.io.loadmat`:

```python
ModuleNotFoundError: No module named 'scipy'
```

Instead it shouldn't raise an error and do nothing

Bug introduced by #4375

Fix https://github.com/huggingface/datasets/issues/4494

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Patching fails for modules that are not installed or don't exist
State: closed
Author: lhoestq

Reported in https://github.com/huggingface/huggingface_hub/runs/6894703718?check_suite_focus=true

When trying to patch `scipy.io.loadmat`:

```python
ModuleNotFoundError: No module named 'scipy'
```

Instead it shouldn't raise an error and do nothing

We use patching to extend such functions to support remote URLs and work in streaming mode

Comments:
Issue: Add `@transmit_format` in `flatten`
State: closed
Author: alvarobartt

As suggested by @mariosasko in https://github.com/huggingface/datasets/pull/4411, we should include the `@transmit_format` decorator to `flatten`, `rename_column`, and `rename_columns` so as to ensure that the value of `_format_columns` in an `ArrowDataset` is properly updated.

**Edit**: according to @mariosasko comment below, the decorator `@transmit_format` doesn't handle column renaming, so it's done manually for those instead.

Comments:
@mariosasko please let me know whether we need to include some sort of tests to make sure that the decorator is working as expected. Thanks! 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4493). All of your documentation changes will be reflected on that endpoint.

Hi, thanks for working on this! Yes, please add (simple) tests so we can avoid any unexpected behavior in the future.

`@transmit_format` doesn't handle column renaming, so I removed it from `rename_column` and `rename_columns` and added a comment to explain this.

Oops, I thought this PR was already merged and deleted from the source repository, I'll be creating a new branch out of `main` so as to re-create this PR... My bad :weary:
Issue: Dataset Viewer issue for Pavithree/test
State: closed
Author: Pavithree

### Link

https://huggingface.co/datasets/Pavithree/test

### Description

I have extracted the subset of original eli5 dataset found at hugging face. However, while loading the dataset It throws ArrowNotImplementedError: Unsupported cast from string to null using function cast_null error. Is there anything missing from my end? Kindly help.

### Owner

_No response_

Comments:
This issue can be resolved according to this post https://stackoverflow.com/questions/70566660/parquet-with-null-columns-on-pyarrow. It looks like first data entry in the json file must not have any null values as pyarrow uses this first file to infer schema for entire dataset.
Issue: Use `torch.nested_tensor` for arrays of varying length in torch formatter
State: open
Author: mariosasko

Use `torch.nested_tensor` for arrays of varying length in `TorchFormatter`.

The PyTorch API of nested tensors is in the prototype stage, so wait for it to become more mature.

Comments:
What's the current behavior?

Currently, we return a list of Torch tensors if their shapes don't match. If they do, we consolidate them into a single Torch tensor.