helmo/github-issues huggingface dataset:
WER and CER > 1
State: closed | Author: sadrasabouri (NONE)
Labels: name: bug

## Describe the bug
It seems that in some cases in which the `prediction` is longer than the `reference` we may have word/character error rate higher than 1 which is a bit odd.

If it's a real bug I think I can solve it with a PR changing [this](https://github.com/huggingface/datasets/blob/master/metrics/wer/wer.py#L105) line to
```python
return min(incorrect / total, 1.0)
```

## Steps to reproduce the bug
```python
from datasets import load_metric
wer = load_metric("wer")
wer_value = wer.compute(predictions=["Hi World vka"], references=["Hello"])
print(wer_value)
```

## Expected results
```
1.0
```

## Actual results
```
3.0
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

Comments: WER can have values bigger than 1.0, this is expected when there are too many insertions

From [wikipedia](https://en.wikipedia.org/wiki/Word_error_rate):
> Note that since N is the number of words in the reference, the word error rate can be larger than 1.0
Re-add download_manager module in utils
State: closed | Author: lhoestq (MEMBER)

https://github.com/huggingface/datasets/pull/4384 moved `datasets.utils.download_manager` to `datasets.download.download_manager`

This breaks `evaluate` which imports `DownloadMode` from `datasets.utils.download_manager` 

This PR re-adds `datasets.utils.download_manager` without circular imports.

We could also show a message that says that accessing it is deprecated, but I think we can do this in a subsequent PR, and just focus on doing a patch release for now

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thanks for the fix.

I'm wondering how this fixes backward compatibility...

Executing this code:
```python
from datasets.utils.download_manager import DownloadMode
```
we will have
```python
DownloadMode = None
```

If afterwards we use something like:
```python
if download_mode == DownloadMode.FORCE_REDOWNLOAD
```
that will raise an exception.

It works fine on my side:
```python
>>> from datasets.utils.download_manager import DownloadMode
>>> DownloadMode is not None
True
```

As reported in https://github.com/huggingface/evaluate/pull/143
```python
from datasets.utils import DownloadConfig
```
is also missing, I'm re-adding it

Took the liberty of merging this one, to do a patch release soon. If we think of a better approach we can improve it later
Replace `assertEqual` with `assertTupleEqual` in unit tests for verbosity
State: closed | Author: alvarobartt (MEMBER)

As detailed in #4419 and as suggested by @mariosasko, we could replace the `assertEqual` assertions with `assertTupleEqual` when the assertion is between Tuples, in order to make the tests more verbose.

Comments: _The documentation is not available anymore as the PR was closed or merged._

FYI I used the following regex to look for the `assertEqual` statements where the assertion was being done over a Tuple: `self.assertEqual(.*, \(.*,)(\)\))$`, hope this is useful!
Fix patching module that doesn't exist
State: closed | Author: lhoestq (MEMBER)

Reported in https://github.com/huggingface/huggingface_hub/runs/6894703718?check_suite_focus=true

When trying to patch `scipy.io.loadmat`:

```python
ModuleNotFoundError: No module named 'scipy'
```

Instead it shouldn't raise an error and do nothing

Bug introduced by #4375

Fix https://github.com/huggingface/datasets/issues/4494

Comments: _The documentation is not available anymore as the PR was closed or merged._
Patching fails for modules that are not installed or don't exist
State: closed | Author: lhoestq (MEMBER)

Reported in https://github.com/huggingface/huggingface_hub/runs/6894703718?check_suite_focus=true

When trying to patch `scipy.io.loadmat`:

```python
ModuleNotFoundError: No module named 'scipy'
```

Instead it shouldn't raise an error and do nothing

We use patching to extend such functions to support remote URLs and work in streaming mode
Add `@transmit_format` in `flatten`
State: closed | Author: alvarobartt (MEMBER)

As suggested by @mariosasko in https://github.com/huggingface/datasets/pull/4411, we should include the `@transmit_format` decorator to `flatten`, `rename_column`, and `rename_columns` so as to ensure that the value of `_format_columns` in an `ArrowDataset` is properly updated.

**Edit**: according to @mariosasko comment below, the decorator `@transmit_format` doesn't handle column renaming, so it's done manually for those instead.

Comments: @mariosasko please let me know whether we need to include some sort of tests to make sure that the decorator is working as expected. Thanks! 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4493). All of your documentation changes will be reflected on that endpoint.

Hi, thanks for working on this! Yes, please add (simple) tests so we can avoid any unexpected behavior in the future.

`@transmit_format` doesn't handle column renaming, so I removed it from `rename_column` and `rename_columns` and added a comment to explain this.

Oops, I thought this PR was already merged and deleted from the source repository, I'll be creating a new branch out of `main` so as to re-create this PR... My bad :weary:
Dataset Viewer issue for Pavithree/test
State: closed | Author: Pavithree (NONE)
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/Pavithree/test

### Description

I have extracted the subset of original eli5 dataset found at hugging face. However, while loading the dataset It throws ArrowNotImplementedError: Unsupported cast from string to null using function cast_null error. Is there anything missing from my end? Kindly help.

### Owner

_No response_

Comments: This issue can be resolved according to this post https://stackoverflow.com/questions/70566660/parquet-with-null-columns-on-pyarrow. It looks like first data entry in the json file must not have any null values as pyarrow uses this first file to infer schema for entire dataset.
Use `torch.nested_tensor` for arrays of varying length in torch formatter
State: open | Author: mariosasko (COLLABORATOR)
Labels: name: enhancement

Use `torch.nested_tensor` for arrays of varying length in `TorchFormatter`.

The PyTorch API of nested tensors is in the prototype stage, so wait for it to become more mature.

Comments: What's the current behavior?

Currently, we return a list of Torch tensors if their shapes don't match. If they do, we consolidate them into a single Torch tensor.
Add SV-Ident dataset
State: closed | Author: e-tornike (NONE)


Comments: Hi @e-tornike, thanks a lot for adding this interesting dataset.

Recently at Hugging Face, we have decided to give priority to adding datasets directly on the Hub. Would you mind to transfer your loading script to the Hub? You could create a dedicated org namespace, so that your dataset would be accessible using `vadis/sv_ident` or `sdproc/sv_ident` or `coling/sv_ident` (as you prefer).

You have an example here: https://huggingface.co/datasets/projecte-aina/catalan_textual_corpus

Additionally, please feel free to ping us if you need assistance/help in creating this dataset.

You could put the link to your Hub dataset here in this Issue discussion page, so that we can follow the progress. :)

Hi @albertvillanova, thanks for the feedback! Uploading via the Hub is a lot easier! 

I've uploaded the dataset here: https://huggingface.co/datasets/vadis/sv-ident, but it's reporting a "Status400Error". Is there any way to see the logs of the dataset script and what is causing the error?

Hi @e-tornike, good job at https://huggingface.co/datasets/vadis/sv-ident.

Normally, you can run locally the loading of the dataset by passing `streaming=True` (as the previewer does):
```python
ds = load_dataset("path/to/sv_ident.py, split="train", streaming=True)
item = next(iter(ds))
item
```

Let me have a look and open a discussion on your Hub repo! ;)

I've opened an Issue: 
- #4527
Update PASS dataset version
State: closed | Author: mariosasko (COLLABORATOR)

Update the PASS dataset to version v3 (the newest one) from the [version history](https://github.com/yukimasano/PASS/blob/main/version_history.txt).

PS: The older versions are not exposed as configs in the script because v1 was removed from Zenodo, and the same thing will probably happen to v2.

Comments: _The documentation is not available anymore as the PR was closed or merged._
Add CCAgT dataset
State: closed | Author: johnnv1 (NONE)

As described in #4075 

I could not generate the dummy data. Also, on the data repository isn't provided the split IDs, but I copy the functions that provide the correct data split. In summary, to have a better distribution, the data in this dataset should be separated based on the amount of NORs in each image.

Comments: _The documentation is not available anymore as the PR was closed or merged._

Hi! Excellent job @johnnv1! There were typos/missing words in the card, so I took the liberty to rewrite some parts to make them easier to understand. Let me know if you are ok with the changes. Also, feel free to add some info under the `Who are the annotators?` section.

Additionally, I fixed the issue with streaming and renamed the `digits` feature to `objects`.

@lhoestq Are you ok with skipping the dummy data test here as it's tricky to generate it due to the splits separation logic?

I think I can also add instance segmentation: by exposing the segment of each instance, so it will be similar with object detection:

- `instances`: a dictionary containing bounding boxes, segments, and labels of the cell objects 
  - `bbox`: a list of bounding boxes
  - `segment`: a list of segments in format of `[polygon]`, where each polygon is `[x0, y0, ..., xn, yn]`
  - `label`: a list of integers representing the category

Do you think it would be ok?

Don't you think it makes sense to keep the same category IDs for all approaches? 

Now we have:
 - nucleus category ID equals 0 for object detection and instance segmentation
 - background category ID equals 0 (on the masks) for semantic segmentation

I find it weird to have a dummy label in object detection just to align the mapping with semantic segmentation. Instead, let's explain in the card (under Data Fields -> annotation) what the pixel values mean (background + object detection labels)

Ok, I can do that in the next few days. I will create a `lapix` organization, and I will add this dataset and also #4565

So, I think we can close this PR? I have already moved these files there.

The link of CCAgT dataset is: https://huggingface.co/datasets/lapix/CCAgT

🤗 

Woohoo awesome !

closing this PR :)
Fix cast to null
State: closed | Author: lhoestq (MEMBER)

It currently fails with `ArrowNotImplementedError` instead of `TypeError` when one tries to cast integer to null type.

Because if this, type inference breaks when one replaces null values with integers in `map` (it first tries to cast to the previous type before inferring the new type).

Fix https://github.com/huggingface/datasets/issues/4483

Comments: _The documentation is not available anymore as the PR was closed or merged._
Better ImportError message when a dataset script dependency is missing
State: closed | Author: lhoestq (MEMBER)

When a depenency is missing for a dataset script, an ImportError message is shown, with a tip to install the missing dependencies. This message is not ideal at the moment: it may show duplicate dependencies, and is not very readable.

I improved it from
```
ImportError: To be able to use bigbench, you need to install the following dependencies['bigbench', 'bigbench', 'bigbench', 'bigbench'] using 'pip install "bigbench @ https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz" bigbench bigbench bigbench' for instance'
```
to
```
ImportError: To be able to use bigbench, you need to install the following dependency: bigbench.
Please install it using 'pip install "bigbench @ https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz"' for instance'
```

Comments: _The documentation is not available anymore as the PR was closed or merged._

Discussed offline with @mariosasko, merging :)

Fwiw, i think this same issue is occurring on the datasets website page, where preview isn't available due to the `bigbench` import error

For the preview of BigBench datasets, we're just waiting for bigbench to have a stable version on PyPI, instead of the one hosted on GCS ;)
Test that TensorFlow is not imported on startup
State: closed | Author: lhoestq (MEMBER)

TF takes some time to be imported, and also uses some GPU memory.

I just added a test to make sure that in the future it's never imported by default when
```python
import datasets
```
is called.

Right now this fails because `huggingface_hub` does import tensorflow (though this is fixed now on their `main` branch)

I'll mark this PR as ready for review once `huggingface_hub` has a new release

Comments: _The documentation is not available anymore as the PR was closed or merged._

Should we close this PR?

I'm closing this PR. Feel free to reopen it if necessary.
Fix iwslt2017
State: closed | Author: lhoestq (MEMBER)

The files were moved to google drive, I hosted them on the Hub instead (ok according to the license)

I also updated the `datasets_infos.json`

Comments: _The documentation is not available anymore as the PR was closed or merged._

CI fails are just abut missing tags in the dataset card, merging !

FYI, 

The checksums have not been edited from the changes of .tgz to .zip files, and as a result a `ExpectedMoreDownloadedFiles` error occurs. Updating them in the `dataset_infos.json` should fix the error. 

Thanks for reporting and sorry for the delay, I opened https://huggingface.co/datasets/iwslt2017/discussions/2 to fix this
Include entity positions as feature in ReCoRD
State: closed | Author: richarddwang (CONTRIBUTOR)

https://huggingface.co/datasets/super_glue/viewer/record/validation

TLDR: We need to record entity positions, which are included in the source data but excluded by the loading script, to enable efficient and effective training for ReCoRD.

Currently, the loading script ignores the entity positions ("entity_start", "entity_end") and only records entity text. This might be because the training method of the official baseline is to make n training instance from a datapoint by replacing \"\@ placeholder\" in query with each entity individually.

But it increases the already heavy computation by multiple folds. So DeBERTa uses a method that take entity embeddings by their positions in the passage, and thus makes one training instance from one data point. It is way more efficient and proved effective for the ReCoRD task.

Can anybody help me with the dataset card rendering error? Maybe @lhoestq ?

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thanks for the reply @lhoestq !

I have sucessed on `datasets-cli test ./datasets/super_glue --name record --save_infos`,
But as you can see, the check ran into `FAILED tests/test_dataset_cards.py::test_changed_dataset_card[super_glue] - V...`.
How can we solve it?

That would be neat! Let me implement it.
Dataset Viewer issue for fgrezes/WIESP2022-NER
State: closed | Author: AshTayade (NONE)

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments: https://huggingface.co/datasets/fgrezes/WIESP2022-NER

The error:

```
Message:       Couldn't find a dataset script at /src/services/worker/fgrezes/WIESP2022-NER/WIESP2022-NER.py or any data file in the same directory. Couldn't find 'fgrezes/WIESP2022-NER' on the Hugging Face Hub either: FileNotFoundError: Unable to resolve any data file that matches ['**test*', '**eval*'] in dataset repository fgrezes/WIESP2022-NER with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

I understand the issue is not related to the dataset viewer in itself, but with the autodetection of the data files without a loading script in the datasets library. cc @lhoestq @albertvillanova @mariosasko 

Apparently it finds `scoring-scripts/compute_seqeval.py` which matches `**eval*`, a regex that detects a test split. We should probably improve the regex because it's not supposed to catch this kind of files. It must also only check for files with supported extensions: txt, csv, png etc.
`to_pandas` doesn't take into account format.
State: closed | Author: Dref360 (CONTRIBUTOR)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**

I have a large dataset that I need to convert part of to pandas to do some further analysis. Calling `to_pandas` directly on it is expensive. So I thought I could simply select the columns that I want and then call `to_pandas`.

**Describe the solution you'd like**

```python
from datasets import Dataset

ds = Dataset.from_dict({'a': [1,2,3], 'b': [5,6,7], 'c': [8,9,10]})
pandas_df = ds.with_format(columns=['a', 'b']).to_pandas()

# I would expect `pandas_df` to only include a,b as column.
```


**Describe alternatives you've considered**
I could remove all columns that I don't want? But I don't know all of them in advance. 

**Additional context**
I can probably make a PR with some pointers.


Comments: Thanks for opening a discussion :)

Note that you can use `.remove_columns(...)` to keep only the ones you're interested in before calling `.to_pandas()`

Yes I can do that thank you!

Do you think that conceptually my example should work? If not, I'm happy to close this issue. 

If yes, I can start working on it.

Hi! Instead of `with_format(columns=['a', 'b']).to_pandas()`, use `with_format("pandas", columns=["a", "b"])` for easy conversion of the parts of the dataset to pandas via indexing/slicing.

The full code:
```python
from datasets import Dataset

ds = Dataset.from_dict({'a': [1,2,3], 'b': [5,6,7], 'c': [8,9,10]})
pandas_df = ds.with_format("pandas", columns=['a', 'b'])[:]
```

Ahhhh Thank you!

closing then :)
Improve error message for missing pacakges from inside dataset script
State: closed | Author: mariosasko (COLLABORATOR)

Improve the error message for missing packages from inside a dataset script:

With this change, the error message for missing packages for `bigbench` looks as follows:
```
ImportError: To be able to use bigbench, you need to install the following dependencies:
        - 'bigbench' using 'pip install "bigbench @ https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz"'
```

And this is how it looked before:
```
ImportError: To be able to use bigbench, you need to install the following dependencies['bigbench', 'bigbench', 'bigbench', 'bigbench'] using 'pip install "bigbench @ https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz" bigbench bigbench bigbench' for instance'
```

Comments: _The documentation is not available anymore as the PR was closed or merged._

I opened a PR before I noticed yours ^^' You can find it here: https://github.com/huggingface/datasets/pull/4484

The only comment I have regarding your message is that it possibly shows several `pip install` commands, whereas one can run one single `pip install` command with the list of missing dependencies, which is maybe simpler.

Let me know which one your prefer

Closing in favor of #4484.
[Docs] How to use with PyTorch page
State: closed | Author: lhoestq (MEMBER)
Labels: name: documentation

Currently the docs about PyTorch are scattered around different pages, and we were missing a place to explain more in depth how to use and optimize a dataset for PyTorch. This PR is related to #4457 which is the TF counterpart :)

cc @Rocketknight1 we can try to align both documentations contents now I think

cc @stevhliu let me know what you think !

Comments: _The documentation is not available anymore as the PR was closed or merged._
Add SST-2 dataset
State: closed | Author: albertvillanova (MEMBER)

Add SST-2 dataset.

Currently it is part of GLUE benchmark.

This PR adds it as a standalone dataset.

CC: @julien-c 

Comments: _The documentation is not available anymore as the PR was closed or merged._

on the hub this dataset is referenced as `sst-2` not `sst2` – is there a canonical orthography? If not, could we name it `sst-2`?

@julien-c, we normally do not use hyphens for dataset names: whenever the original dataset name contains a hyphen, we usually:
- either suppress it: CoNLL-2000 (`conll2000`), CORD-19 (`cord19`)
- or replace it with underscore: CC-News (`cc_news`), SQuAD-es (`squad_es`)

There are some exceptions though... (I wonder why)

I think, the reason is there was a 1-to-1 relation with the corresponding Python module name.

I personally find confusing not having a rule and using both hyphens and underscores indistinctly: you never know which is the right orthography.

Whichever the decision we make, I would prefer to be applied consistently.

Also note that we already implemented this dataset as part of GLUE: https://github.com/huggingface/datasets/blob/master/datasets/glue/glue.py#L163
- dataset name: `glue`
- config name: `sst2`

On the other hand, let's see how other libraries name it:
- torchtext: `SST2` https://pytorch.org/text/stable/datasets.html#sst2
- OpenAI CLIP: `rendered-sst2` https://github.com/openai/CLIP/blob/main/data/rendered-sst2.md
- Kaggle: `SST2` https://www.kaggle.com/datasets/atulanandjha/stanford-sentiment-treebank-v2-sst2/version/22
- TensorFlow Datasets: `glue/sst2` https://www.tensorflow.org/datasets/catalog/glue#gluesst2

Ok, another option is to open PRs against the models in https://huggingface.co/models?datasets=sst-2 to change their dataset reference to `sst2`

(BTW some models refer to `sst2` already – but they're less popular: https://huggingface.co/models?datasets=sst2)

OK, I'm taking care of the subsequent PRs on models to align with this dataset name.
CI error with repo lhoestq/_dummy
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

## Describe the bug
CI is failing because of repo "lhoestq/_dummy". See: https://app.circleci.com/pipelines/github/huggingface/datasets/12461/workflows/1b040b45-9578-4ab9-8c44-c643c4eb8691/jobs/74269
```
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/api/datasets/lhoestq/_dummy?full=true
```

The repo seems to no longer exist: https://huggingface.co/api/datasets/lhoestq/_dummy
```
error: "Repository not found"
```

CC: @lhoestq 

Comments: fixed by https://github.com/huggingface/datasets/pull/4472