noamaanMulla-03/datasets-issues huggingface dataset:
Add splits to MBPP dataset

This PR addresses https://github.com/huggingface/datasets/issues/4795


```
(env) cwarny@Cedrics-Air datasets % RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_mbpp
================================================================================================ test session starts =================================================================================================
platform darwin -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0
rootdir: /Users/cwarny/datasets, configfile: setup.cfg
collected 1 item                                                                                                                                                                                                     

tests/test_dataset_common.py .                                                                                                                                                                                 [100%]

================================================================================================= 1 passed in 1.12s ==================================================================================================
(env) cwarny@Cedrics-Air datasets % RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_mbpp               
================================================================================================ test session starts =================================================================================================
platform darwin -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0
rootdir: /Users/cwarny/datasets, configfile: setup.cfg
collected 1 item                                                                                                                                                                                                     

tests/test_dataset_common.py .                                                                                                                                                                                 [100%]

================================================================================================= 1 passed in 0.35s ==================================================================================================

```

_The documentation is not available anymore as the PR was closed or merged._

Hi @cwarny ! Thanks for adding the correct splits :)

You can fix the CI error by running `make style` - this should reformat the dataset script

done
Dataset Viewer issue for wmt16

### Link

https://huggingface.co/datasets/wmt16

### Description

[Reported](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/12#62cb83f14c7f35284e796f9c) by a user of AutoTrain Evaluate. AFAIK this dataset was working 1-2 weeks ago, and I'm not sure how to interpret this error.

```
Status code:   400
Exception:     NotImplementedError
Message:       This is a abstract method
```

Thanks!

### Owner

No

Labels: name: dataset-viewer

Thanks for reporting, @lewtun.

~We can't load the dataset locally, so I think this is an issue with the loading script (not the viewer).~

 We are investigating...

Recently, there was a merged PR related to this dataset:
- #4554

We are looking at this...

Indeed, the above mentioned PR fixed the loading script (it was not working before).

I'm forcing the refresh of the Viewer.

Please note that the above mentioned PR also made an enhancement in the `datasets` library, required by this loading script. This enhancement will only be available to the Viewer once we make our next release.

OK, it's working now.

https://huggingface.co/datasets/wmt16/viewer/ro-en/test

<img width="1434" alt="Capture d’écran 2022-09-08 à 10 15 55" src="https://user-images.githubusercontent.com/1676121/189071665-17d2d149-9b22-42bf-93ac-1a966c3f637a.png">


Thank you @severo !!
Preserve non-`input_colums` in `Dataset.map` if `input_columns` are specified

Currently, if the `input_columns` list in `Dataset.map` is specified, the columns not in that list are dropped after the `map` transform.

This makes the behavior inconsistent with `IterableDataset.map`.
 
(It seems this issue was introduced by mistake in https://github.com/huggingface/datasets/pull/2246) 

Fix https://github.com/huggingface/datasets/issues/4858


_The documentation is not available anymore as the PR was closed or merged._
[not really a bug] `identical_ok` is deprecated in huggingface-hub's `upload_file`

In the huggingface-hub dependency, the `identical_ok` argument has no effect in `upload_file` (and it will be removed soon)

See

https://github.com/huggingface/huggingface_hub/blob/43499582b19df1ed081a5b2bd7a364e9cacdc91d/src/huggingface_hub/hf_api.py#L2164-L2169

It's used here:

https://github.com/huggingface/datasets/blob/fcfcc951a73efbc677f9def9a8707d0af93d5890/src/datasets/dataset_dict.py#L1373-L1381

https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4354-L4362

https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4197-L4213

We should remove it.

Maybe the third code sample has an unexpected behavior since it uses the non-default value `identical_ok = False`, but the argument is ignored.

Labels: name: help wanted

name: question

Resolved via https://github.com/huggingface/datasets/pull/4937.
Dataset should have a `.name` field

**Is your feature request related to a problem? Please describe.**
If building pipelines that can evaluate on more than one dataset, it would be nice to be able to log results of things like `Evaluating on {dataset.name}` or `results for {dataset.name} are: {results}`

Without some way of concisely identifying a dataset from the dataset object, tools which might run on more than one dataset must be passed the dataset object _and_ the name/id of the dataset being used. 

**Describe the solution you'd like**
The DatasetInfo class should have a `name` field which is the name of a dataset. then for a given dataset if it evolves in time the `version` can be updated but its different versions of the same dataset with a unique `name`. The name could then all be accessed by `dataset.name`

**Describe alternatives you've considered**
For my own purposes I am considering making `NamedDataset[Dataset]` where the subclass just has a .name field. 

**Additional context**
My guess is that most usecases are not working with more than one dataset in a given pipeline so a name is not really needed. This has surprised me though as one of the advantages of a standard dataset interface is to be able to build pipelines which can be passed in a dataset and separate responsibilities of the dataset loading from the train or eval pipeline.


Labels: name: enhancement

Hi! You can already use `dset.builder_name` and `dset.config_name` for that purpose. And when it comes to versioning, it's better to use `dset._fingerprint` than the `version` attribute as the former represents a deterministic hash that encodes all the mutable ops executed on a dataset, and the latter stays the same unless it's manually updated after each op.

@mariosasko  Can we make ._fingerprint not private? seems a critical component for tracking how a model was generated to ensure reproducibility.
Shard parquet in `download_and_prepare`

Following https://github.com/huggingface/datasets/pull/4724 (needs to be merged first)

It's good practice to shard parquet files to enable parallelism with spark/dask/etc.

I added the `max_shard_size` parameter to `download_and_prepare` (default to 500MB for parquet, and None for arrow).

```python
from datasets import *

output_dir = "./output_dir"  # also supports "s3://..."
builder = load_dataset_builder("squad")
builder.download_and_prepare(output_dir, file_format="parquet", max_shard_size="5MB")
```

### Implementation details

The examples are written to a parquet file until `ParquetWriter._num_bytes > max_shard_size`. When this happens, a new writer is instantiated to start writing the next shard. At the end, all the shards are renamed to include the total number of shards in their names: `{builder.name}-{split}-{shard_id:05d}-of-{num_shards:05d}.parquet`

I also added the `MAX_SHARD_SIZE` config variable (default to 500MB)

TODO:
- [x] docstrings
- [x] docs
- [x] tests

cc @severo 


_The documentation is not available anymore as the PR was closed or merged._

This is ready for review cc @mariosasko :) please let me know what you think !
Providing dataset size

**Is your feature request related to a problem? Please describe.**
Especially for big datasets like [LAION](https://huggingface.co/datasets/laion/laion2B-en/), it's hard to know exactly the downloaded size (because there are many files and you don't have their exact size when downloaded).

**Describe the solution you'd like**
Auto-populating the downloaded dataset size on the dataset page would be really useful, including that of each split (when there are some).

**Describe alternatives you've considered**
People should be adding this to dataset cards, but I don't think that is systematically the case :slightly_smiling_face: 

**Additional context**
Mentioned to @lhoestq 


Labels: name: enhancement

Hi @sashavor, thanks for your suggestion.

Until now we have the CLI command 
```
datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
```
that generates the `dataset_infos.json` with the size of the downloaded dataset, among other information.

We are currently in the middle of removing those JSON files and putting their information directly in the header of the `README.md` (as YAML tags). Normally, the CLI command should continue working but saving its output to the dataset card instead. See:
- #4926

Additionally, the download size can be inferred by doing HEAD requests to the files to be downloaded. And for files hosted on the hub you can even get the file sizes using the Hub API

Amazing @albertvillanova ! I think just having that information visible in the dataset info (without having to do any requests/additional coding) would be really useful :hugs:
ConnectionError in IndicGLUE dataset

While I am trying to load IndicGLUE dataset (https://huggingface.co/datasets/indic_glue) it is giving me with the error:
```
ConnectionError: Couldn't reach https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/wikiann-ner.tar.gz (error 403)

Labels: name: dataset bug

@sahoodib, thanks for reporting.

Indeed, none of the data links appearing in the IndicGLUE website are working, e.g.: https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/soham-articles.tar.gz
```
<Error>
<Code>UserProjectAccountProblem</Code>
<Message>User project billing account not in good standing.</Message>
<Details>
The billing account for the owning project is disabled in state delinquent
</Details>
</Error>
```

We have contacted the data owners to inform them about their issue and ask them if they plan to fix it.

Yesterday I resent a reminder email with more AI4Bharat-related people in the loop.

I also opened an issue in their repos:
- https://github.com/AI4Bharat/indicnlp_corpus/issues/14
- https://github.com/AI4Bharat/ai4bharat.org/issues/71

We have received a reply from the authors reporting they have updated the URLs of their data files and opened a PR. See:
- #4978
docs: ✏️ add links to the Datasets API

I added some links to the Datasets API in the docs. See https://github.com/huggingface/datasets-server/pull/566 for a companion PR in the datasets-server. The idea is to improve the discovery of the API through the docs.

I'm a bit shy about pasting a lot of links to the API in the docs, so it's minimal for now. I'm interested in ideas to integrate the API better in these docs without being too much. cc @lhoestq @julien-c @albertvillanova @stevhliu.


_The documentation is not available anymore as the PR was closed or merged._

OK, thanks @lhoestq. I'll close this PR, and come back to it with @stevhliu once we work on https://github.com/huggingface/datasets-server/issues/568
Add `Dataset.from_generator`

Add `Dataset.from_generator` to the API to allow creating datasets from data larger than RAM. The implementation relies on a packaged module not exposed in `load_dataset` to tie this method with `datasets`' caching mechanism.

Closes https://github.com/huggingface/datasets/issues/4417


I restarted the builder PR job just in case

_The documentation is not available anymore as the PR was closed or merged._

CI is now green. https://github.com/huggingface/doc-builder/pull/296 explains why it failed.
Make `pyarrow` optional

**Is your feature request related to a problem? Please describe.**
Is `pyarrow` really needed for every dataset?

**Describe the solution you'd like**
It is made optional.

**Describe alternatives you've considered**
Likely, no.


Labels: name: enhancement

The whole datasets library is pretty much a wrapper to pyarrow (just take a look at some of the source for a Dataset) https://github.com/huggingface/datasets/blob/51aef08ad7053c0bfe8f9a961207b26df15850d3/src/datasets/arrow_dataset.py#L639 

I think removing the pyarrow dependency would involve a complete rewrite / a different library with minimal functionality (datasets-lite ?)

Thanks for the proposal, @KOLANICH. And also thanks for your answer, @dconathan.

Indeed, we are using `pyarrow` as the backend for our datasets, in order to cache them and also allow memory-mapping (using datasets larger than your RAM memory).

One way to avoid using `pyarrow` could be loading the datasets in streaming mode, by passing `streaming=True` to `load_dataset`. This way you basically get a generator for the dataset; nothing is downloaded, nor cached. 

Thanks for the info. Could `datasets` then be made optional for `transformers` instead? I used `transformers` only to deal with pretrained models to deploy them (convert to ONNX, and then I use TVM), so I don't really need `pyarrow` and `datasets` by now.
[doc] Fix broken snippet that had too many quotes

Hello!

### Pull request overview
* Fix broken snippet in https://huggingface.co/docs/datasets/main/en/process that has too many quotes

### Details
The snippet in question can be found here: https://huggingface.co/docs/datasets/main/en/process#map
This screenshot shows the issue, there is a quote too many, causing the snippet to be colored incorrectly:
![image](https://user-images.githubusercontent.com/37621491/190640627-f7587362-0e44-4464-a5d1-a0b98df6986f.png)

The change speaks for itself.

Thank you for the detailed documentation, by the way. 

- Tom Aarsen



_The documentation is not available anymore as the PR was closed or merged._

Spent the day familiarising myself with the huggingface line of products, and happened to run into some small issues here and there. Magically, I've found exactly one small issue in `transformers`, one in `accelerate` and now one in `datasets`, hah!

As for this PR, the issue seems solved according to the [new PR documentation](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4986/en/process#map):
![image](https://user-images.githubusercontent.com/37621491/190646405-6afa06fa-9eac-48f6-ab30-2677944fb7b6.png)
datasets.set_progress_bar_enabled(False) not working in datasets v2

## Describe the bug

datasets.set_progress_bar_enabled(False) not working in datasets v2

## Steps to reproduce the bug
```python
datasets.set_progress_bar_enabled(False)
```

## Expected results
datasets not using any progress bar

## Actual results

AttributeError: module 'datasets' has no attribute 'set_progress_bar_enabled

## Environment info

datasets version 2


Labels: name: bug

Now, I can't find any reference to set_progress_bar_enabled in the code.

I think it have been deleted

Hi @virilo,

Please note that since `datasets` version 2.0.0, we have aligned with `transformers` the management of the progress bar (among other things):
- #3897

Now, you should update your code to use `datasets.logging.disable_progress_bar`.

You have more info in our docs: [Logging methods](https://huggingface.co/docs/datasets/package_reference/logging_methods)

One important thing for beginner like me is: from datasets.utils.logging import disable_progress_bar
Do not forget the 'utils' or you will waste a long time like me....
datasets generate large arrow file

Checking the large file in disk, and found the large cache file in the cifar10 data directory:

![image](https://user-images.githubusercontent.com/18533904/186830449-ba96cdeb-0fe8-4543-994d-2abe7145933f.png)

As we know, the size of cifar10 dataset is ~130MB, but the cache file has almost 30GB size, there may be some problems here.

Labels: name: bug

Hi ! The cache files are the results of all the transforms you applied to the dataset using `map` for example.
Did you run a transform that could potentially blow up the size of the dataset ?

@lhoestq,
I don't remember, but I can't imagine what kind of transform may generate data that grow over 200 times in size. 
I think maybe it doesn' matter, it's just cache after all.
Running add_column() seems to corrupt existing sequence-type column info

I have a dataset that contains a column ("foo") that is a sequence type of length 4. So when I run .to_pandas() on it, the resulting dataframe correctly contains 4 columns - foo_0, foo_1, foo_2, foo_3. So the 1st row of the dataframe might look like:

ds = load_dataset(...)
df = ds.to_pandas()

df:
foo_0 | foo_1 | foo_2 | foo_3
0.0 | 1.0 | 2.0 | 3.0

If I run .add_column("new_col", data) on the dataset, and then .to_pandas() on the resulting new dataset, the resulting dataframe contains only 2 columns - foo, new_col. The values in column foo are lists of length 4, the 4 elements that should have been split into separate columns. Dataframe 1st row would be:

ds = load_dataset(...)
new_ds = ds.add_column("new_col", data)
df = new_ds.to_pandas()

df:
foo | new_col
[0.0, 1.0, 2.0, 3.0] | new_val

I've explored the 2 datasets in a debugger and haven't noticed any changes to any attributes related to the foo column, but I can't determine why the dataframes are so different.

Labels: name: bug

Nevermind, I was incorrect.
Fix map batched with torch output

Reported in https://discuss.huggingface.co/t/typeerror-when-applying-map-after-set-format-type-torch/23067/2

Currently it fails  if one uses batched `map` and the map function returns a torch tensor.

I fixed it for torch, tf, jax and pandas series.


_The documentation is not available anymore as the PR was closed or merged._
More fine-grained taxonomy of error types

**Is your feature request related to a problem? Please describe.**

Exceptions like `FileNotFoundError` can be raised by different parts of the code, and it's hard to detect which one did

**Describe the solution you'd like**

Give a specific exception type for every group of similar errors

**Describe alternatives you've considered**

Rely on the error message, using regex


Labels: name: enhancement

name: dataset-viewer

related: #4995
load_dataset_builder should error if "name" does not exist?

```
import datasets as ds
builder = ds.load_dataset_builder('sent_comp', name="doesnotexist")
builder.info.config_name
```

returns

```
'doesnotexist'
```

Shouldn't it raise an error instead?

For this dataset, the only valid values for `name` should be: `"default"` or `None` (ie. argument not passed)

Labels: name: bug

name: dataset-viewer

Yes I think it should raise an error. Currently it looks like it instantiates a custom configuration with the name given by the user:
https://github.com/huggingface/datasets/blob/ba27ce33bf568374cf23a07669fdd875b5718bc2/src/datasets/builder.py#L391-L397
Don't add a tag on the Hub on release

Datasets with no namespace on the Hub have tags to redirect to the version of datasets where they come from.
I’m about to remove them all because I think it looks bad/unexpected in the UI and it’s not actually useful

Therefore I'm also disabling tagging.

Note that the CI job will be completely removed in https://github.com/huggingface/datasets/pull/4974 anyway


_The documentation is not available anymore as the PR was closed or merged._
Dataset without script does not support regular JSON data file

### Link

https://huggingface.co/datasets/julien-c/label-studio-my-dogs

### Description

<img width="1115" alt="image" src="https://user-images.githubusercontent.com/326577/189422048-7e9c390f-bea7-4521-a232-43f049ccbd1f.png">


### Owner

Yes


Hi @julien-c,

Out of the box, we only support JSON lines (NDJSON) data files, but your data file is a regular JSON file. The reason is we use `pyarrow.json.read_json` and this only supports line-delimited JSON.
Add support for parsing JSON files in array form

Support parsing JSON files in the array form (top-level object is an array). For simplicity, `json.load` is used for decoding. This means the entire file is loaded into memory. If requested, we can optimize this by introducing a param similar to `lines` in [`pandas.read_json`](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html), which, if set to `True`, would allow us to read in chunks.

Fixes https://github.com/huggingface/datasets/issues/4963



_The documentation is not available anymore as the PR was closed or merged._
fix: avoid casting tuples after Dataset.map

This PR updates features.py to avoid casting tuples to lists when reading the results of Dataset.map as suggested by @lhoestq [here](https://github.com/huggingface/datasets/issues/4676#issuecomment-1187371367) in https://github.com/huggingface/datasets/issues/4676.



_The documentation is not available anymore as the PR was closed or merged._
vivos (Vietnamese speech corpus) dataset not accessible

## Describe the bug
VIVOS data is not accessible anymore, neither of these links work (at least from France):
* https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (data)
* https://ailab.hcmus.edu.vn/vivos (dataset page) 

Therefore `load_dataset` doesn't work.

## Steps to reproduce the bug
```python
ds = load_dataset("vivos")
```

## Expected results
dataset loaded

## Actual results
```
ConnectionError: Couldn't reach https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='ailab.hcmus.edu.vn', port=443): Max retries exceeded with url: /assets/vivos.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d8a27d190>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))")))
```

Will try to contact the authors, as we wanted to use Vivos as an example in documentation on how to create scripts for audio datasets (https://github.com/huggingface/datasets/pull/4872), because it's small and straightforward and uses tar archives. 

Labels: name: dataset bug

If you need an example of a small audio datasets, I just created few hours ago a speech dataset with only 300MB of compressed audio files https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia. It works also with streaming (@albertvillanova helped me adding this functionality) :-)

@cahya-wirawan omg this is awesome!! thank you! 

We have contacted the authors to ask them.
Dataset Viewer issue for loubnabnl/humaneval-x

### Link

https://huggingface.co/datasets/loubnabnl/humaneval-x/viewer/

### Description

The dataset has subsets but the viewer gets stuck in the default subset even when I select another one (the data loading of the subsets works fine)

### Owner

Yes

Labels: name: dataset-viewer

It's a bug! Thanks for reporting, I'm looking at it

Fixed.