helmo/github-issues huggingface dataset:
Streaming not supported in Theivaprakasham/wildreceipt
State: closed | Author: NitishkKarra (NONE)
Labels: name: streaming

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments: Thanks for reporting @NitishkKarra.

The root source of the issue is that streaming mode is not supported out-of-the-box for that dataset, because it contains a TAR file.

We have opened a discussion in the corresponding Hub dataset page, pointing out this issue: https://huggingface.co/datasets/Theivaprakasham/wildreceipt/discussions/1

I'm closing this issue here, so this discussion is transferred there instead.
Training hangs at the end of epoch, with set_transform/with_transform+multiple workers
State: open | Author: memray (NONE)
Labels: name: bug

## Describe the bug
I use load_dataset() (I tried with [wiki](https://huggingface.co/datasets/wikipedia) and my own json data) and use set_transform/with_transform for preprocessing. But it hangs at the end of the 1st epoch if dataloader_num_workers>=1. No problem with single worker. 


## Steps to reproduce the bug
```python
train_dataset = datasets.load_dataset("wikipedia", "20220301.en",
                                      split='train', 
                                      cache_dir=model_args.cache_dir,
                                      streaming=False)
train_dataset.set_transform(psg_parse_fn)
train_dataloader = DataLoader(
            train_dataset,
            batch_size=args.train_batch_size,
            sampler=DistributedSampler(train_dataset),
            collate_fn=data_collator,
            drop_last=args.dataloader_drop_last,
            num_workers=args.dataloader_num_workers,
        )
```

## Expected results


## Actual results
It simply hangs. The ending step is num_example/batch_size (one epoch).

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Linux-5.4.170+-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.1
Document loading from relative path
State: closed | Author: stevhliu (MEMBER)
Labels: name: documentation

This PR describes loading a dataset from the Hub by specifying a relative path in `data_dir` or `data_files` in `load_dataset` (see #4757).

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thanks for the feedback!

I agree that adding it to `load_hub.mdx` is probably a bit too specific, especially for beginners reading the tutorials. Since this clarification is closely related to loading from the Hub (the only difference being the presence/absence of a loading script), I think it makes the most sense to keep it somewhere in `loading.mdx`. What do you think about adding a Warning in Loading >>> Hugging Face Hub that explains the difference between relative/absolute paths when there is a script?

What about updating the section about "manual download" ? I think it goes there no ?

https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download

Updated the manual download section :)

Thanks ! Pinging @albertvillanova to review this change, and then I think we're good to merge
More rigorous shape inference in to_tf_dataset
State: closed | Author: Rocketknight1 (MEMBER)

`tf.data` needs to know the shape of tensors emitted from a `tf.data.Dataset`. Although `None` dimensions are possible, overusing them can cause problems - Keras uses the dataset tensor spec at compile-time, and so saying that a dimension is `None` when it's actually constant can hurt performance, or even cause training to fail for dimensions that are needed to determine the shape of weight tensors!

The compromise I used here was to sample several batches from the underlying dataset and apply the `collate_fn` to them, and then to see which dimensions were "empirically variable". There's an obvious problem here, though - if you sample 10 batches and they all have the same shape on a certain dimension, there's still a small chance that the 11th batch will be different, and Keras will throw an error  if a dataset tries to emit a tensor whose shape doesn't match the spec.

I encountered this bug in practice once or twice for datasets that were mostly-but-not-totally constant on a given dimension, and I still don't have a perfect solution, but this PR should greatly reduce the risk. It samples many more batches, and also samples very small batches (size 2) - this increases the variability, making it more likely that a few outlier samples will be detected.

Ideally, of course, we'd determine the full output shape analytically, but that's surprisingly tricky when the `collate_fn` can be any arbitrary Python code!

Comments: _The documentation is not available anymore as the PR was closed or merged._
Improve features resolution in streaming
State: closed | Author: lhoestq (MEMBER)

`IterableDataset._resolve_features` was returning the features sorted alphabetically by column name, which is not consistent with non-streaming. I changed this and used the order of columns from the data themselves. It was causing some inconsistencies in the dataset viewer as well.

I also fixed `interleave_datasets` that was not filling missing columns with None, because it was not using the columns from `IterableDataset._resolve_features`

cc @severo 

Comments: _The documentation is not available anymore as the PR was closed or merged._

Just took your comment into account @mariosasko , let me know if it's good for you now :)
Dataset Viewer issue for Toygar/turkish-offensive-language-detection
State: closed | Author: tanyelai (NONE)
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/Toygar/turkish-offensive-language-detection

### Description

Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.

Hi, I provided train.csv, test.csv and valid.csv files. However, viewer says dataset does not exist. 
Should I need to do anything else?

### Owner

Yes

Comments: I refreshed the dataset viewer manually, it's fixed now. Sorry for the inconvenience.
<img width="1557" alt="Capture d’écran 2022-07-28 à 09 17 39" src="https://user-images.githubusercontent.com/1676121/181514666-92d7f8e1-ddc1-4769-84f3-f1edfdb902e8.png">
Document better when relative paths are transformed to URLs
State: closed | Author: albertvillanova (MEMBER)
Labels: name: documentation

As discussed with @ydshieh, when passing a relative path as `data_dir` to `load_dataset` of a dataset hosted on the Hub, the relative path is transformed to the corresponding URL of the Hub dataset.

Currently, we mention this in our docs here: [Create a dataset loading script > Download data files and organize splits](https://huggingface.co/docs/datasets/v2.4.0/en/dataset_script#download-data-files-and-organize-splits)
> If the data files live in the same folder or repository of the dataset script, you can just pass the relative paths to the files instead of URLs.

Maybe we should document better how relative paths are handled, not only when creating a dataset loading script, but also when passing to `load_dataset`:
- `data_dir`
- `data_files`

CC: @stevhliu
Remove "unkown" language tags
State: closed | Author: lhoestq (MEMBER)

Following https://github.com/huggingface/datasets/pull/4753 there was still a "unknown" langauge tag in `wikipedia` so the job at https://github.com/huggingface/datasets/runs/7542567336?check_suite_focus=true failed for wikipedia

Comments: _The documentation is not available anymore as the PR was closed or merged._
Add `language_bcp47` tag
State: closed | Author: lhoestq (MEMBER)

Following (internal) https://github.com/huggingface/moon-landing/pull/3509, we need to move the bcp47 tags to `language_bcp47` and keep the `language` tag for iso 639 1-2-3 codes. In particular I made sure that all the tags in `languages` are not longer than 3 characters. I moved the rest to `language_bcp47` and fixed some of them.

After this PR is merged I think we can simplify the language validation from the DatasetMetadata class (and keep it bare-bone just for the tagging app)

PS: the CI is failing because of missing content in dataset cards that are unrelated to this PR

Comments: _The documentation is not available anymore as the PR was closed or merged._
Easily create loading script for benchmark comprising multiple huggingface datasets
State: closed | Author: JoelNiklaus (CONTRIBUTOR)

Hi,

I would like to create a loading script for a benchmark comprising multiple huggingface datasets.
The function _split_generators needs to return the files for the respective dataset. However, the files are not always in the same location for each dataset. I want to just make a wrapper dataset that provides a single interface to all the underlying datasets. 
I thought about downloading the files with the load_dataset function and then providing the link to the cached file. But this seems a bit inelegant to me. What approach would you propose to do this?

Please let me know if you have any questions.

Cheers,
Joel

Comments: Hi ! I think the simplest is to copy paste the `_split_generators` code from the other datasets and do a bunch of if-else, as in the glue dataset: https://huggingface.co/datasets/glue/blob/main/glue.py#L467

Ok, I see. Thank you
Shard parquet in `download_and_prepare`
State: closed | Author: lhoestq (MEMBER)

Following https://github.com/huggingface/datasets/pull/4724 (needs to be merged first)

It's good practice to shard parquet files to enable parallelism with spark/dask/etc.

I added the `max_shard_size` parameter to `download_and_prepare` (default to 500MB for parquet, and None for arrow).

```python
from datasets import *

output_dir = "./output_dir"  # also supports "s3://..."
builder = load_dataset_builder("squad")
builder.download_and_prepare(output_dir, file_format="parquet", max_shard_size="5MB")
```

### Implementation details

The examples are written to a parquet file until `ParquetWriter._num_bytes > max_shard_size`. When this happens, a new writer is instantiated to start writing the next shard. At the end, all the shards are renamed to include the total number of shards in their names: `{builder.name}-{split}-{shard_id:05d}-of-{num_shards:05d}.parquet`

I also added the `MAX_SHARD_SIZE` config variable (default to 500MB)

TODO:
- [x] docstrings
- [x] docs
- [x] tests

cc @severo 

Comments: _The documentation is not available anymore as the PR was closed or merged._

This is ready for review cc @mariosasko :) please let me know what you think !
Dataset Viewer issue for yanekyuk/wikikey
State: closed | Author: ai-ashok (NONE)
Labels: name: dataset-viewer

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments: The dataset is empty, as far as I can tell: there are no files in the repository at https://huggingface.co/datasets/yanekyuk/wikikey/tree/main

Maybe the viewer can display a better message for empty datasets

OK. Closing as it's not an error. We will work on making the error message a lot clearer.
Allow `list_datasets` to include private datasets
State: closed | Author: ola13 (NONE)
Labels: name: enhancement

I am working with a large collection of private datasets, it would be convenient for me to be able to list them.

I would envision extending the convention of using `use_auth_token` keyword argument to `list_datasets` function, then calling:

```
list_datasets(use_auth_token="my_token")
```

would return the list of all datasets I have permissions to view, including private ones. The only current alternative I see is to use the hub website to manually obtain the list of dataset names - this is in the context of BigScience where respective private spaces contain hundreds of datasets, so not very convenient to list manually.

Comments: Thanks for opening this issue :)

If it can help, I think you can already use `huggingface_hub` to achieve this:
```python
>>> from huggingface_hub import HfApi
>>> [ds_info.id for ds_info in HfApi().list_datasets(use_auth_token=token) if ds_info.private]
['bigscience/xxxx', 'bigscience-catalogue-data/xxxxxxx', ... ]
```

---------

Though the latest versions of `huggingface_hub` that contain this feature are not available on python 3.6, so maybe we should first drop support for python 3.6 (see #4460) to update `list_datasets` in `datasets` as well (or we would have to copy/paste some `huggingface_hub` code)

Great, thanks @lhoestq the workaround works! I think it would be intuitive to have the support directly in `datasets` but it makes sense to wait given that the workaround exists :)

i also think that going forward we should replace more and more implementations inside datasets with the corresponding ones from `huggingface_hub` (same as we're doing in `transformers`)

`datasets.list_datasets` is now deprecated in favor of `huggingface_hub.list_datasets` (returns private datasets when `token` is present), so I'm closing this issue.
Remove instructions to generate dummy data from our docs
State: closed | Author: albertvillanova (MEMBER)
Labels: name: documentation

In our docs, we indicate to generate the dummy data: https://huggingface.co/docs/datasets/dataset_script#testing-data-and-checksum-metadata

However:
- dummy data makes sense only for datasets in our GitHub repo: so that we can test their loading with our CI
- for datasets on the Hub:
  - they do not pass any CI test requiring dummy data
  - there are no instructions on how they can test their dataset locally using the dummy data
  - the generation of the dummy data assumes our GitHub directory structure:
    - the dummy data will be generated under `./datasets/<dataset_name>/dummy` even if locally there is no `./datasets` directory (which is the usual case). See issue:
      - #4742 

CC: @stevhliu 

Comments: Note that for me personally, conceptually all the dummy data (even for "canonical" datasets) should be superseded by `datasets-server`, which performs some kind of CI/CD of datasets (including the canonical ones)

I totally agree: next step should be rethinking if dummy data makes sense for canonical datasets (once we have datasets-server) and eventually remove it.

But for now, we could at least start by removing the indication to generate dummy data from our docs.
Dummy data nowhere to be found
State: closed | Author: BramVanroy (CONTRIBUTOR)
Labels: name: bug

## Describe the bug
To finalize my dataset, I wanted to create dummy data as per the guide and I ran 

```shell
 datasets-cli dummy_data datasets/hebban-reviews --auto_generate
```

where hebban-reviews is [this repo](https://huggingface.co/datasets/BramVanroy/hebban-reviews). And even though the scripts runs and shows a message at the end that it succeeded, I cannot find the dummy data anywhere. Where is it?

## Expected results

To see the dummy data in the datasets' folder or in the folder where I ran the command.

## Actual results

I see the following message but I cannot find the dummy data anywhere.

```
Dummy data generation done and dummy data test succeeded for config 'filtered''.
Automatic dummy data generation succeeded for all configs of '.\datasets\hebban-reviews\'
```

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 8.0.0
- Pandas version: 1.4.3



Comments: Hi @BramVanroy, thanks for reporting.

First of all, please note that you do not need the dummy data: this was the case when we were adding datasets to the `datasets` library (on this GitHub repo), so that we could test the correct loading of all datasets with our CI. However, this is no longer the case for datasets on the Hub.
- We should definitely update our docs.

Second, the dummy data is generated locally:
- in your case, the dummy data will be generated inside the directory:  `./datasets/hebban-reviews/dummy`
- please note the preceding `./datasets` directory: the reason for this is that the command to generate the dummy data was specifically created for our `datasets` library, and therefore assumes our directory structure: commands are run from the root directory of our GitHub repo, and datasets scripts are under `./datasets` 


 

I have opened an Issue to update the instructions on dummy data generation:
- #4744

Dummy data generation is deprecated now, so I think we can close this issue.
Fix multiprocessing in map_nested 
State: closed | Author: albertvillanova (MEMBER)

As previously discussed:

Before, multiprocessing was not used in `map_nested` if `num_proc` was greater than or equal to `len(iterable)`.
- Multiprocessing was not used e.g. when passing `num_proc=20` but having 19 files to download
- As by default, `DownloadManager` sets `num_proc=16`, before multiprocessing was only used when `len(iterable)>16` by default

Now, if `num_proc` is greater than or equal to ``len(iterable)``, `num_proc` is set to ``len(iterable)`` and multiprocessing is used.
- We pass the variable `parallel_min_length=16`, so that multiprocessing is only used if at least 16 files to be downloaded
- ~As by default, `DownloadManager` sets `num_proc=16`, now multiprocessing is used when `len(iterable)>1` by default~

See discussion below.

~After having had to fix some tests (87602ac), I am wondering:~
- ~do we want to have multiprocessing by default?~
  - ~please note that `DownloadManager.download` sets `num_proc=16` by default~
- ~or would it be better to ask the user to set it explicitly if they want multiprocessing (and default to `num_proc=1`)?~

Fix #4636.

CC: @nateraw 

Comments: _The documentation is not available anymore as the PR was closed or merged._

@lhoestq as a workaround to preserve previous behavior, the parameter `multiprocessing_min_length=16` is passed from `download` to `map_nested`, so that multiprocessing is only used if at least 16 files to be downloaded.

Note that there is a small breaking change (I think previously it was unintended behavior, so that I have fixed it):
- Before (with default `num_proc=16`) if there were 16 files to be downloaded, multiprocessing was not used
- Now (with default `num_proc=16`) if there are 16 files to be downloaded, multiprocessing is used

Thanks for the workaround !
Deprecate metrics
State: closed | Author: albertvillanova (MEMBER)

Deprecate metrics:
- deprecate public functions: `load_metric`, `list_metrics` and `inspect_metric`: docstring and warning
- test deprecation warnings are issues
- deprecate metrics in all docs
- remove mentions to metrics in docs and README
- deprecate internal functions/classes

Maybe we should also stop testing metrics?

Comments: _The documentation is not available anymore as the PR was closed or merged._

I mark this as Draft because the deprecated version number needs being updated after the latest release.

Perhaps now is the time to also update the `inspect_metric` from `evaluate` with the changes introduced in https://github.com/huggingface/datasets/pull/4433 (cc @lvwerra) 

What do you think of including what changes users have to do to switch to `evaluate` in the warning message ?
(basically replace `datasets.load_metric` by `evaluate.load`)

I think it can help users migrate to `evaluate` and silence the warnings
Use CI unit/integration tests
State: closed | Author: albertvillanova (MEMBER)

This PR:
- Implements separate unit/integration tests
- A fail in integration tests does not cancel the rest of the jobs
  - We should implement more robust integration tests: work in progress in a subsequent PR
- For the moment, test involving network requests are marked as integration: to be evolved

Comments: _The documentation is not available anymore as the PR was closed or merged._

I think this PR can be merged. Willing to see it in action.

CC: @lhoestq
Download error on scene_parse_150
State: closed | Author: juliensimon (NONE)
Labels: name: bug

```
from datasets import load_dataset
dataset = load_dataset("scene_parse_150", "scene_parsing")

FileNotFoundError: Couldn't find file at http://data.csail.mit.edu/places/ADEchallenge/ADEChallengeData2016.zip
```


Comments: Hi! The server with the data seems to be down. I've reported this issue (https://github.com/CSAILVision/sceneparsing/issues/34) in the dataset repo. 

The URL seems to work now, and therefore the script as well.
Dataset Viewer issue for deepklarity/huggingface-spaces-dataset
State: closed | Author: dk-crazydiv (NONE)
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/deepklarity/huggingface-spaces-dataset/viewer/deepklarity--huggingface-spaces-dataset/train

### Description

Hi Team, 
I'm getting the following error on a uploaded dataset. I'm getting the same status for a couple of hours now. The dataset size is `<1MB` and the format is csv, so I'm not sure if it's supposed to take this much time or not. 
```
Status code:   400
Exception:     Status400Error
Message:       The split is being processed. Retry later.
```

Is there any explicit step to be taken to get the viewer to work? 

### Owner

Yes

Comments: Thanks for reporting. You're right, workers were under-provisioned due to a manual error, and the job queue was full. It's fixed now.
Package rouge-score cannot be imported
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

## Describe the bug
After the today release of `rouge_score-0.0.7` it seems no longer importable. Our CI fails: https://github.com/huggingface/datasets/runs/7463218591?check_suite_focus=true
```
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_bigbench
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_builder_configs_bigbench
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_bigbench
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_rouge
```
with errors:
```
>   from rouge_score import rouge_scorer
E   ModuleNotFoundError: No module named 'rouge_score'
```
```
E           ImportError: To be able to use rouge, you need to install the following dependency: rouge_score.
E           Please install it using 'pip install rouge_score' for instance'
```


Comments: We have added a comment on an existing issue opened in their repo: https://github.com/google-research/google-research/issues/1212#issuecomment-1192267130
- https://github.com/google-research/google-research/issues/1212
rouge metric
State: closed | Author: asking28 (NONE)
Labels: name: bug

## Describe the bug
A clear and concise description of what the bug is.
Loading  Rouge metric gives error after latest rouge-score==0.0.7 release.
Downgrading rougemetric==0.0.4 works fine.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
A clear and concise description of the expected results.
from rouge_score import rouge_scorer, scoring 
should run

## Actual results
Specify the actual results or traceback.
File "/root/.cache/huggingface/modules/datasets_modules/metrics/rouge/0ffdb60f436bdb8884d5e4d608d53dbe108e82dac4f494a66f80ef3f647c104f/rouge.py", line 21, in <module>
    from rouge_score import rouge_scorer, scoring
ImportError: cannot import name 'rouge_scorer' from 'rouge_score' (unknown location)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Linux
- Python version:3.9
- PyArrow version:


Comments: Fixed by:
- #4735