Minimartzz/github-issues huggingface dataset:
Align tqdm control with cache control
State: closed

Currently, once disabled with `disable_progress_bar`, progress bars cannot be re-enabled again. To overcome this limitation, this PR introduces the `set_progress_bar_enabled` function that accepts a boolean indicating whether to display progress bars. The goal is to provide a similar API to the existing cache control API. Following the Zen of Python (😄), there should be one and preferably only one obvious way to do it, so I'm also deprecating the aforementioned `disable_progress_bar` function. Additionally, I justify the deprecation with the fact that this function has never been in the docs.

Moreover, similar API changes have recently been introduced to [`tfds`](https://github.com/tensorflow/datasets/blob/a1e8b98f45b0214082b546cc967c67c43fffda55/tensorflow_datasets/core/utils/tqdm_utils.py#L98-L112).

Considering the popularity of the [comment](https://github.com/huggingface/datasets/issues/1627#issuecomment-751383559) I made a while ago, this API (`set_progress_bar_enabled` and `is_progress_bar_enabled`) should be mentioned in the docs, but I'm not sure where to put it exactly. Maybe we can replace the `logging_methods` page under `package_reference` with `utility_methods` and then introduce two subsections on that page: `Logging methods` and `tqdm control`.

Additionally, this PR:
* adds the `disable_tqdm` keyword arg of `Dataset._map_single` to the `ignore_kwargs` list to ignore it when computing the fingerprint (forgot to add it in #2696)
* deletes the unused components in `tqdm_utils.py`, which seem to be inherited from `tfds`
* disables the tqdm output in the test suite. As I see it, this output doesn't seem informative, but let me know if this is not a good idea

Comments:
Could you add this function to the documentation please ?

You can add it in `main_classes.rst`, and maybe add a `Tip` section in the `map` section in the `process.rst`
Add `remove_columns` to `IterableDataset`
State: closed

Fixes #2944

WIP
* Not tested yet.
* We might want to allow batched remove for efficiency.

@lhoestq Do you think it should have `batched=` and `batch_size=`?

Comments:
Thanks ! That looks all good :)

I don't think that batching would help. Indeed we're dealing with python iterators that yield elements one by one, so batched `map` needs to accumulate a batch, apply the function, and then yield examples from the batch.

Though once we have parallel processing in `map`, we can reconsider it

Also feel free to check the CI failure - apparently the import of `Union` is missing

Thanks for the review and explaining that! 
On top of what you said, I think `remove_columns` is very unlikely to be a bottleneck, so it doesn't matter anyways.

Thank you for reviewing! @mariosasko 

I wonder how the checking would work. Is there any checking present in `IterableDataset ` now? What if `.remove_columns()` is applied after some arbitrary `.map()`?

> I wonder how the checking would work. Is there any checking present in IterableDataset now? What if .remove_columns() is applied after some arbitrary .map()?

That's the challenge here indeed ^^ In this case it's not trivial to know the names of the columns. Feel free to open an issue so we can discuss this
Use standard open-domain validation split in nq_open
State: closed

The nq_open dataset originally drew the validation set from this file:
https://github.com/google-research-datasets/natural-questions/blob/master/nq_open/NQ-open.efficientqa.dev.1.1.sample.jsonl
However, that's the dev set used specifically and only for the efficientqa competition, and it's not the same dev set as is used in every open-domain question answering paper (including the Lee et al paper that introduced the open-domain variant of NQ, cited at the top of the dataset file). This PR changes nq_open to use the standard validation split and bumps the version to 2.0.0 since this is a breaking change.

Comments:
I had to run datasets-cli with --ignore_verifications the first time since it was complaining about a missing file, but now it runs without that flag fine. I moved dummy_data.zip to the new folder, but also had to modify the filename of the test file in the zip (should I not have done that?). Finally, I added the pretty name tag.

Great, thanks for the help.
Properly install ruamel-yaml for windows CI
State: closed


Comments:
@lhoestq I would say this does not "properly" install `ruamel-yaml`, but the contrary, you overwrite the previous version without desinstalling it first.

According to `pip` docs:
>  This can break your system if the existing package is of a different version or was installed with a different package manager!

Note that our case fulfills both conditions:
- the installing version (`0.17.16`) is different from the existing one (`0.15.87`)
- you are installing using `pip` (`setuptools`), whereas the exisitng version was installed using `distutils`

That is why I did not fix the issue this way, made a hotfix pinning `huggingface_hub` (#3025), while looking for a permanent solution for the issue.

Yea I did this because we need the latest version of `huggingface_hub` for #2986 and because I didn't want to ssh to the windows worker x)
Maybe it can be fixed by installing it with conda - let me try

Oh yea it may not work since it was first installed with distutils
Resolve data_files by split name
State: closed

This issue is about discussing the default behavior when someone loads a dataset that consists in data files. For example:
```python
load_dataset("lhoestq/demo1")
```
should return two splits "train" and "test" since the dataset repostiory is like
```
data/
├── train.csv
└── test.csv
```
Currently it returns only one split "train" which contains the data of both files


I started playing with this idea on this branch btw: `resolve-data_files-by-split-name`
Basically the idea is that if you named you data files after split names then the default pattern is
```python
{
    "train": ["*train*"],
    "test": ["*test*"],
    "validation": ["*dev*", "valid"],
}
```
otherwise it's
```python
{
    "train": ["*"]
}
```

Let me know what you think !
cc @albertvillanova @LysandreJik @vblagoje 

Comments:
Awesome @lhoestq I like the proposal and it works great on my JSON community dataset. Here is the [log](https://gist.github.com/vblagoje/714babc325bcbdd5de579fd8e1648892). 

From my discussion with @borisdayma it would be more general the files match if their paths contains the split name - not only if the filename contains the split name. For example for a dataset like this:
```
train/
└── data.csv
test/
└── data.csv
```

But IMO the default should be 
```
data/
├── train.csv
└── test.csv
```
because it allows people to have other directories if they have different subsets of their data (different configurations, not splits)

I just created a PR for this at https://github.com/huggingface/datasets/pull/3221, let me know what you think :)
Windows test suite fails
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug

There is an error during installation of tests dependencies for Windows: https://app.circleci.com/pipelines/github/huggingface/datasets/7981/workflows/9b6a0114-2b8e-4069-94e5-e844dbbdba4e/jobs/49206

```
ERROR: Cannot uninstall 'ruamel-yaml'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
```



Comments:
MeDAL dataset: Add further description and update download URL
State: closed

Added more details in the following sections:
* Dataset Structure
* Data Instances
* Data Splits
* Source Data
* Annotations
* Discussions of Biases
* LIcensing Information

Comments:
@lhoestq I'm a bit confused by the error message. I haven't touched the YAML code at all - do you have any insight on that?

I just added the missing `pretty_name` tag in the YAML - sorry about that ;)

Thanks! Seems like it did the trick since the tests are passing. Let me know if there's anything else I can do in this PR!

It's all good thank you :)

merging !
Add a metric for the MATH dataset (competition_math).
State: closed

This metric computes accuracy for the MATH dataset (https://arxiv.org/abs/2103.03874) after canonicalizing the prediction and the reference (e.g., converting "1/2" to "\\\\frac{1}{2}").

Comments:
I believe the only failed test related to this PR is tests/test_metric_common.py::LocalMetricTest::test_load_metric_competition_math. It gives the following error:

ImportError: To be able to use this dataset, you need to install the following dependencies['math_equivalence'] using 'pip install git+https://github.com/hendrycks/math.git' for instance'

It fails along with (these fail with ImportError as well):
test_load_metric_bertscore
test_load_metric_bleurt
test_load_metric_comet
test_load_metric_coval

Let me know if there is anything I need to change.

Hi ! The script looks all good thanks :)

To fix the CI you just need to merge `master` into your branch
```
git fetch upstream/master
git merge upstream/master
```

Then you also need to add `math_equivalence` to the list of git packages installed for the tests in `additional-tests-requirements.txt`
https://github.com/huggingface/datasets/blob/ba831e4bcd175ae3d52afbf7d12c4f625bf541b0/additional-tests-requirements.txt#L1-L3

I ran:

git fetch upstream
git merge upstream/master

And I also added  math_equivalence to the list of git packages installed for the tests in additional-tests-requirements.txt

tests/test_metric_common.py fails with the same errors as before. tests/test_dataset_cards.py also fails, but it doesn't look related to this PR (it's an issue datasets/ami/README.md).

@lhoestq Anything else I can do? I re-merged again and am getting the same test failures as described in the previous comment.
Fix filter leaking
State: closed

If filter is called after using a first transform `shuffle`, `select`, `shard`, `train_test_split`, or `filter`, then it could not work as expected and return examples from before the first transform. This is because the indices mapping was not taken into account when saving the indices to keep when doing the filtering

Affected versions: 1.12.0 and 1.12.1

This should fix #3010

Comments:
Support multiple zipped CSV data files
State: open
Labels: description: New feature or request
name: enhancement

As requested by @lewtun, support loading multiple zipped CSV data files.

```python
from datasets import load_dataset

url = "https://domain.org/filename.zip"
data_files = {"train": "train_filename.csv", "test": "test_filename.csv"}
dataset = load_dataset("csv", data_dir=url, data_files=data_files)
```


Comments:
@lhoestq I would like to draw your attention to the proposed API by @lewtun, using `data_dir` to pass the ZIP URL.

I'm not totally convinced with this... What do you think?

Maybe we could discuss other approaches...

One brainstorming idea: what about using URL chaining with the hop operator in `data_files`?

`data_dir` is currently exclusively used for manually downloaded data.

Maybe we can have an API that only uses data_files as you are suggesting, using URL chaining ?
```python
from datasets import load_dataset

url = "https://domain.org/filename.zip"
data_files = {"train": "zip://train_filename.csv::" + url, "test": "zip://test_filename.csv::" + url}
dataset = load_dataset("csv", data_files=data_files)
```

URL chaining is used by `fsspec` to get access to files in nested filesystems of any kind. Since `fsspec` is being used by `pandas`, `dask` and also extensively by `datasets` I think it would be nice to use it here too

URL chaining sounds super nice to me! And it's also a nice way to leverage the same concepts we currently have in the docs around `fsspec` :)
Improve `get_dataset_infos`?
State: closed
Labels: description: Further information is requested
name: question

description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

Using the dedicated function `get_dataset_infos` on a dataset that has no dataset-info.json file returns an empty info:

```
>>> from datasets import get_dataset_infos
>>> get_dataset_infos('wit')
{}
```

While it's totally possible to get it (regenerate it) with:

```
>>> from datasets import load_dataset_builder
>>> builder = load_dataset_builder('wit')
>>> builder.info
DatasetInfo(description='Wikipedia-based Image Text (WIT) Dataset is a large multimodal multilingual dataset. WIT is composed of a curated set\n of 37.6 million entity rich image-text examples with 11.5 million unique images across 108 Wikipedia languages. Its\n size enables WIT to be used as a pretraining dataset for multimodal machine learning models.\n', citation='@article{srinivasan2021wit,\n  title={WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning},\n  author={Srinivasan, Krishna and Raman, Karthik and Chen, Jiecao and Bendersky, Michael and Najork, Marc},\n  journal={arXiv preprint arXiv:2103.01913},\n  year={2021}\n}\n', homepage='https://github.com/google-research-datasets/wit', license='', features={'b64_bytes': Value(dtype='string', id=None), 'embedding': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None), 'image_url': Value(dtype='string', id=None), 'metadata_url': Value(dtype='string', id=None), 'original_height': Value(dtype='int32', id=None), 'original_width': Value(dtype='int32', id=None), 'mime_type': Value(dtype='string', id=None), 'caption_attribution_description': Value(dtype='string', id=None), 'wit_features': Sequence(feature={'language': Value(dtype='string', id=None), 'page_url': Value(dtype='string', id=None), 'attribution_passes_lang_id': Value(dtype='string', id=None), 'caption_alt_text_description': Value(dtype='string', id=None), 'caption_reference_description': Value(dtype='string', id=None), 'caption_title_and_reference_description': Value(dtype='string', id=None), 'context_page_description': Value(dtype='string', id=None), 'context_section_description': Value(dtype='string', id=None), 'hierarchical_section_title': Value(dtype='string', id=None), 'is_main_image': Value(dtype='string', id=None), 'page_changed_recently': Value(dtype='string', id=None), 'page_title': Value(dtype='string', id=None), 'section_title': Value(dtype='string', id=None)}, length=-1, id=None)}, post_processed=None, supervised_keys=None, task_templates=None, builder_name='wit', config_name='default', version=0.0.0, splits=None, download_checksums=None, download_size=None, post_processing_size=None, dataset_size=None, size_in_bytes=None)
```

Should we test if info is empty, and in that case regenerate it? Or always generate it?

Comments:
To keeps things simple maybe we should use `load_dataset_builder` in `get_dataset_infos`.
`load_dataset_builder` instantiates a builder and runs the _infos() method in order to give you the most up-to-date infos, even if the dataset_infos.json is outdated or missing.
load_dataset_builder should error if "name" does not exist?
State: open
Labels: description: Something isn't working
name: bug

description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

```
import datasets as ds
builder = ds.load_dataset_builder('sent_comp', name="doesnotexist")
builder.info.config_name
```

returns

```
'doesnotexist'
```

Shouldn't it raise an error instead?

For this dataset, the only valid values for `name` should be: `"default"` or `None` (ie. argument not passed)

Comments:
Yes I think it should raise an error. Currently it looks like it instantiates a custom configuration with the name given by the user:
https://github.com/huggingface/datasets/blob/ba27ce33bf568374cf23a07669fdd875b5718bc2/src/datasets/builder.py#L391-L397
DatasetDict.filter and Dataset.filter crashes with any "fn_kwargs" argument
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
The ".filter" method of DatasetDict or Dataset objects fails when passing any "fn_kwargs" argument

## Steps to reproduce the bug
```python
import datasets

example_dataset = datasets.Dataset.from_dict({"a": {1, 2, 3, 4}})

def filter_value(example, value):
    return example['a'] == value

filtered = example_dataset.filter(filter_value, fn_kwargs={'value': 3})
```

## Expected results
`filtered` is a dataset containing {"a": {3}}

## Actual results

> Traceback (most recent call last):
>   File "C:\Users\qsemi\Documents\git\nlp_experiments\gpt_celebrity\src\test_faulty_filter.py", line 8, in <module>
>     filtered = example_dataset.filter(filter_value, fn_kwargs={'value': 3})
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 185, in wrapper
>     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\fingerprint.py", line 398, in wrapper
>     out = func(self, *args, **kwargs)
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 2169, in filter
>     indices = self.map(
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 1686, in map
>     return self._map_single(
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 185, in wrapper
>     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\fingerprint.py", line 398, in wrapper
>     out = func(self, *args, **kwargs)
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 2048, in _map_single
>     batch = apply_function_on_filtered_inputs(
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 1939, in apply_function_on_filtered_inputs
>     function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
> TypeError: get_indices_from_mask_function() got an unexpected keyword argument 'value'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.9.7
- PyArrow version: 5.0.0


Comments:
Hi @DrMatters, thanks for reporting.

This issue was fixed 14 days ago: #2950.

Currently, the fix is only in the master branch and will be made available in our next library release.

In the meantime, you can incorporate the fix by installing datasets from the master branch:
```shell
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```shell
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```

Thanks, sorry for bothering
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English.
State: closed

Inspired by the recent widespread use of the GLUE multi-task benchmark NLP dataset (Wang et al., 2018), the subsequent more difficult SuperGLUE (Wang et al., 2019), other previous multi-task NLP benchmarks (Conneau and Kiela, 2018; McCann et al., 2018), and similar initiatives in other domains (Peng et al., 2019), we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a benchmark dataset to evaluate the performance of NLP methods in legal tasks. LexGLUE is based on seven existing legal NLP datasets, selected using criteria largely from SuperGLUE.

As in GLUE and SuperGLUE (Wang et al., 2019b,a), one of our goals is to push towards generic (or ‘foundation’) models that can cope with multiple NLP tasks, in our case legal NLP tasks possibly with limited task-specific fine-tuning. Another goal is to provide a convenient and informative entry point for NLP researchers and practitioners wishing to explore or develop methods for legalNLP. Having these goals in mind, the datasets we include in LexGLUE and the tasks they address have been simplified in several ways to make it easier for newcomers and generic models to address all tasks.

LexGLUE benchmark is accompanied by experimental infrastructure that relies on Hugging Face Transformers library and resides at: https://github.com/coastalcph/lex-glue.

Comments:
Please wait until Tuesday. Arxiv pre-print is pending. 🤗 

Hi @lhoestq, I updated the README with the Arxiv publication info and now the tests are not passing.

It seems that the error is completely irrelevant to my code:

```
 Attempting uninstall: ruamel.yaml
    Found existing installation: ruamel-yaml 0.15.87
ERROR: Cannot uninstall 'ruamel-yaml'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
```

Hi ! Thanks for adding this one. Amazing work :o 

Yea you can ignore the `ruamel-yaml` error, it's unrelated to your PR and fixed on `master`

Hi @lhoestq, 

- I fetched and merged the upstream master to get the `ruamel-yaml` fix.
- I updated the README.md file including more information for the inputs and labels, while I also turned all tables in HTML format hoping that they will render nicely in the dataset card in the HF website.
- I also simplified the CaseHOLD input, excl. the unused `question` field and the `context` replicas, as suggested.
Remove a reference to the open Arrow file when deleting a TF dataset created with to_tf_dataset
State: closed

This [comment](https://github.com/huggingface/datasets/issues/2934#issuecomment-922970919) explains the issue. This PR fixes that with a `weakref` callback, and additionally:
* renames `TensorflowDatasetMixIn` to `TensorflowDatasetMixin` for consistency
* correctly indents `TensorflowDatasetMixin`'s docstring
* replaces `tf.data.AUTOTUNE` with `tf.data.experimental.AUTOTUNE` (we support TF>=2.2 according to the [setup.py](https://github.com/huggingface/datasets/blob/fc46bba66ba4f432cc10501c16a677112e13984c/setup.py#L188) and `AUTOTUNE` has been moved to the experimental part of `tf.data` in 1.X if I'm not mistaken)

Fixes #2934

Comments:
@lhoestq The test passes even without the try/except block!

Hey, I'm a little late because I was caught up in the course work, but I double-checked this and it looks great. Thanks for fixing!
Fix cast to Python scalar in Matthews Correlation metric
State: closed

This PR is motivated by issue #2964.

The Matthews Correlation metric relies on sklearn's `matthews_corrcoef` function to compute the result. This function returns either `float` or `np.float64` (see the [source](https://github.com/scikit-learn/scikit-learn/blob/844b4be24d20fc42cc13b957374c718956a0db39/sklearn/metrics/_classification.py#L906-L909)). Obviously, calling `.item()` on the float value will fail, so I'm fixing this with the built-in `float()` function, which covers both cases. Surprisingly, on my machine, casting `np.float64` to a Python scalar with `float()` is even faster than with the `.item()` method.

Comments:
Fix json loader when conversion not implemented
State: closed

Sometimes the arrow json parser fails if the `block_size` is too small and returns an `ArrowNotImplementedError: JSON conversion to struct...` error.

By increasing the block size it makes it work again.

Hopefully it should help with https://github.com/huggingface/datasets/issues/2799

I tried with the file mentioned in the issue and it worked for me
cc @lewtun can you try again from this branch ?

Comments:
And we're already at PR number 3,000 ! ^^

Thank you so much for fixing this @lhoestq 😍 ! I just tested the branch out and it works like a charm!