Minimartzz/github-issues huggingface dataset:
docs: ✏️ add links to the Datasets API
State: closed

I added some links to the Datasets API in the docs. See https://github.com/huggingface/datasets-server/pull/566 for a companion PR in the datasets-server. The idea is to improve the discovery of the API through the docs.

I'm a bit shy about pasting a lot of links to the API in the docs, so it's minimal for now. I'm interested in ideas to integrate the API better in these docs without being too much. cc @lhoestq @julien-c @albertvillanova @stevhliu.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

OK, thanks @lhoestq. I'll close this PR, and come back to it with @stevhliu once we work on https://github.com/huggingface/datasets-server/issues/568
Make `pyarrow` optional
State: closed
Labels: description: New feature or request
name: enhancement

**Is your feature request related to a problem? Please describe.**
Is `pyarrow` really needed for every dataset?

**Describe the solution you'd like**
It is made optional.

**Describe alternatives you've considered**
Likely, no.


Comments:
The whole datasets library is pretty much a wrapper to pyarrow (just take a look at some of the source for a Dataset) https://github.com/huggingface/datasets/blob/51aef08ad7053c0bfe8f9a961207b26df15850d3/src/datasets/arrow_dataset.py#L639 

I think removing the pyarrow dependency would involve a complete rewrite / a different library with minimal functionality (datasets-lite ?)

Thanks for the proposal, @KOLANICH. And also thanks for your answer, @dconathan.

Indeed, we are using `pyarrow` as the backend for our datasets, in order to cache them and also allow memory-mapping (using datasets larger than your RAM memory).

One way to avoid using `pyarrow` could be loading the datasets in streaming mode, by passing `streaming=True` to `load_dataset`. This way you basically get a generator for the dataset; nothing is downloaded, nor cached. 

Thanks for the info. Could `datasets` then be made optional for `transformers` instead? I used `transformers` only to deal with pretrained models to deploy them (convert to ONNX, and then I use TVM), so I don't really need `pyarrow` and `datasets` by now.
Fix missing tags in dataset cards
State: closed

Fix missing tags in dataset cards:
- amazon_us_reviews
- art
- discofuse
- indic_glue
- ubuntu_dialogs_corpus

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Providing dataset size
State: open
Labels: description: New feature or request
name: enhancement

**Is your feature request related to a problem? Please describe.**
Especially for big datasets like [LAION](https://huggingface.co/datasets/laion/laion2B-en/), it's hard to know exactly the downloaded size (because there are many files and you don't have their exact size when downloaded).

**Describe the solution you'd like**
Auto-populating the downloaded dataset size on the dataset page would be really useful, including that of each split (when there are some).

**Describe alternatives you've considered**
People should be adding this to dataset cards, but I don't think that is systematically the case :slightly_smiling_face: 

**Additional context**
Mentioned to @lhoestq 


Comments:
Hi @sashavor, thanks for your suggestion.

Until now we have the CLI command 
```
datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
```
that generates the `dataset_infos.json` with the size of the downloaded dataset, among other information.

We are currently in the middle of removing those JSON files and putting their information directly in the header of the `README.md` (as YAML tags). Normally, the CLI command should continue working but saving its output to the dataset card instead. See:
- #4926

Additionally, the download size can be inferred by doing HEAD requests to the files to be downloaded. And for files hosted on the hub you can even get the file sizes using the Hub API

Amazing @albertvillanova ! I think just having that information visible in the dataset info (without having to do any requests/additional coding) would be really useful :hugs:
Hope to adapt Python3.9 as soon as possible
State: open
Labels: description: New feature or request
name: enhancement

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.


Comments:
Hi! `datasets` should work in Python 3.9. What kind of issue have you encountered?

There is this related issue already: https://github.com/huggingface/datasets/issues/4113
And I guess we need a CI job for 3.9 ^^

Perhaps we should report this issue in the `filelock` repo?
Add `fn_kwargs` param to `IterableDataset.map`
State: closed

Add the `fn_kwargs` parameter to `IterableDataset.map`.

("Resolves" https://discuss.huggingface.co/t/how-to-use-large-image-text-datasets-in-hugging-face-hub-without-downloading-for-free/22780/3)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for adding this fix! 

Would it be possible to get `fn_kwargs` added to `IterableDatasetDict.map` as well? It looks like a very similar problem, and hopefully shouldn't be a huge change. 


Hi @brianhill11! https://github.com/huggingface/datasets/pull/5810 adds this (opened a couple of days ago). It should be merged soon.

That's fantastic news, thanks @mariosasko ! I'll give it a shot once the changes are merged in.
[GH->HF] Part 2: Remove all dataset scripts from github
State: closed

Now that all the datasets live on the Hub we can remove the /datasets directory that contains all the dataset scripts of this repository

- [x] Needs https://github.com/huggingface/datasets/pull/4973 to be merged first
- [x] and PR to be enabled on the Hub for non-namespaced datasets

Comments:
_The documentation is not available anymore as the PR was closed or merged._

So this means metrics will be deleted from this repo in favor of the "evaluate" library? Maybe you guys could just redirect metrics to that library.

We are deprecating the metrics in `datasets` indeed and suggest users to switch to `evaluate` (via a warning message)

We'll keep the current metrics as they are for now, but they'll be completely removed at one point

I guess this is ready to merge ?

It should break nothing except one rare case:

If someone is using an old version of `datasets` to try to load a recent dataset. Indeed in that case it fetches the `main` branch on github to see if it exists. But since we're removing all the datasets, forward fetching won't work anymore.

e.g. if someone uses "imagenet-1k" with a version of `datasets` that didn't have it at that time. I checked on kibana and one single user would be affected with 4k downloads/months. It should still work for them though thanks to the `datasets` cache

But if they delete their cache, the workaround is... 🥁 update `datasets` 😅

Let's merge this on monday if we can, to make sure contributors who wanted to merge their dataset PRs here could do it

Alright, merging !
[GH->HF] Load datasets from the Hub
State: closed

Currently datasets with no namespace (e.g. squad, glue) are loaded from github.

In this PR I changed this logic to use the Hugging Face Hub instead.

This is the first step in removing all the dataset scripts in this repository

related to discussions in https://github.com/huggingface/datasets/pull/4059 (I should have continued from this PR actually)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Duplicate of:
- #4059
Fix map batched with torch output
State: closed

Reported in https://discuss.huggingface.co/t/typeerror-when-applying-map-after-set-format-type-torch/23067/2

Currently it fails  if one uses batched `map` and the map function returns a torch tensor.

I fixed it for torch, tf, jax and pandas series.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Preserve non-`input_colums` in `Dataset.map` if `input_columns` are specified
State: closed

Currently, if the `input_columns` list in `Dataset.map` is specified, the columns not in that list are dropped after the `map` transform.

This makes the behavior inconsistent with `IterableDataset.map`.
 
(It seems this issue was introduced by mistake in https://github.com/huggingface/datasets/pull/2246) 

Fix https://github.com/huggingface/datasets/issues/4858

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix data URL and metadata of vivos dataset
State: closed

After contacting the authors of the VIVOS dataset to report that their data server is down, we have received a reply from Hieu-Thi Luong that their data is now hosted on Zenodo: https://doi.org/10.5281/zenodo.7068130

This PR updates their data URL and some metadata (homepage, citation and license).

Fix #4936.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Dataset without script does not support regular JSON data file
State: closed

### Link

https://huggingface.co/datasets/julien-c/label-studio-my-dogs

### Description

<img width="1115" alt="image" src="https://user-images.githubusercontent.com/326577/189422048-7e9c390f-bea7-4521-a232-43f049ccbd1f.png">


### Owner

Yes

Comments:
Hi @julien-c,

Out of the box, we only support JSON lines (NDJSON) data files, but your data file is a regular JSON file. The reason is we use `pyarrow.json.read_json` and this only supports line-delimited JSON.
Update setup.py
State: closed

exclude broken version of fsspec. See the [related issue](https://github.com/huggingface/datasets/issues/4961)

Comments:
Before addressing this PR, we should be sure about the issue. See my comment in:
- https://github.com/huggingface/datasets/issues/4961#issuecomment-1243376247

Once we know 2022.8.2 works, I'm closing this PR, as the corresponding issue.
Fix data URLs of compguesswhat dataset
State: closed

After we informed the `compguesswhat` dataset authors about an error with their data URLs, they have updated them:
- https://github.com/CompGuessWhat/compguesswhat.github.io/issues/1

This PR updates their data URLs in our loading script.

Related to:
- #3191

Comments:
_The documentation is not available anymore as the PR was closed or merged._
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.4.0/datasets/jsonl/jsonl.py
State: closed
Labels: description: Something isn't working
name: bug

Hi,
When I use load_dataset from local jsonl files, below error happens, and I type the link into the browser prompting me `404: Not Found`. I download the other `.py` files using the same method and it works. It seems that the server is missing the appropriate file, or it is a problem with the code version.

```
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b08342004c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")))

```



Comments:
I have solved this problem... The extension of the file should be `.json` not `.jsonl`
Add `Dataset.from_generator`
State: closed

Add `Dataset.from_generator` to the API to allow creating datasets from data larger than RAM. The implementation relies on a packaged module not exposed in `load_dataset` to tie this method with `datasets`' caching mechanism.

Closes https://github.com/huggingface/datasets/issues/4417

Comments:
I restarted the builder PR job just in case

_The documentation is not available anymore as the PR was closed or merged._

CI is now green. https://github.com/huggingface/doc-builder/pull/296 explains why it failed.
Raise a more precise error when the URL is unreachable in streaming mode
State: open
Labels: description: New feature or request
name: enhancement

See for example:

- https://github.com/huggingface/datasets/issues/3191
- https://github.com/huggingface/datasets/issues/3186

It would help provide clearer information on the Hub and help the dataset maintainer solve the issue by themselves quicker. Currently:

- https://huggingface.co/datasets/compguesswhat

  <img width="1029" alt="Capture d’écran 2022-09-08 à 15 51 37" src="https://user-images.githubusercontent.com/1676121/189139946-6deffb91-f21b-4281-8825-a98026c69740.png">

- https://huggingface.co/datasets/nli_tr

  <img width="1032" alt="Capture d’écran 2022-09-08 à 15 51 44" src="https://user-images.githubusercontent.com/1676121/189139963-d26490ed-ad23-48ea-9cfc-1ab9c4d08d0c.png">


cc @albertvillanova 

Comments:
CI test of TensorFlow is failing
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
The following CI test fails: https://github.com/huggingface/datasets/runs/8246722693?check_suite_focus=true
```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - AssertionError:
```

Details:
```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw0] linux -- Python 3.7.13 /opt/hostedtoolcache/Python/3.7.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        def gen_random_output():
            model = layers.Dense(2)
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
        with temp_seed(42, set_tensorflow=True):
            out1 = gen_random_output()
        with temp_seed(42, set_tensorflow=True):
            out2 = gen_random_output()
        out3 = gen_random_output()
    
>       np.testing.assert_equal(out1, out2)
E       AssertionError: 
E       Arrays are not equal
E       
E       Mismatched elements: 2 / 2 (100%)
E       Max absolute difference: 0.84619296
E       Max relative difference: 16.083529
E        x: array([[-0.793581,  0.333286]], dtype=float32)
E        y: array([[0.052612, 0.539708]], dtype=float32)

tests/test_py_utils.py:149: AssertionError
```




Comments:
Add test-datasets CI job
State: closed

To avoid having too many conflicts in the datasets and metrics dependencies I split the CI into test and test-catalog

test does the test of the core of the `datasets` lib, while test-catalog tests the datasets scripts and metrics scripts

This also makes `pip install -e .[dev]` much smaller for developers

WDYT @albertvillanova ?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Closing this one since the dataset scripts will be removed in https://github.com/huggingface/datasets/pull/4974
Update Enwik8 broken link and information
State: closed

The current enwik8 dataset link give a 502 bad gateway error which can be view on https://huggingface.co/datasets/enwik8 (click the dropdown to see the dataset preview, it will show the error). This corrects the links, and json metadata as well as adds a little bit more information about enwik8.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update enwik8 fixing the broken link
State: closed

The current enwik8 dataset link give a 502 bad gateway error which can be view on https://huggingface.co/datasets/enwik8 (click the dropdown to see the dataset preview, it will show the error). This corrects the links, and json metadata as well as adds a little bit more information about enwik8.

Comments:
Closing pull request to following contributing guidelines of making a new branch and will make a new pull request
Introduce regex check when pushing as well
State: closed

Closes https://github.com/huggingface/datasets/issues/4945 by adding a regex check when pushing to hub.

Let me know if this is helpful and if it's the fix you would have in mind for the issue and I'm happy to contribute tests.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Let me take over this PR if you don't mind
larger dataset, larger GPU memory in the training phase? Is that correct?
State: closed
Labels: description: Something isn't working
name: bug

    from datasets import set_caching_enabled
    set_caching_enabled(False)
    for ds_name in ["squad","newsqa","nqopen","narrativeqa"]:
            train_ds = load_from_disk("../../../dall/downstream/processedproqa/{}-train.hf".format(ds_name))

        break
    train_ds = concatenate_datasets([train_ds,train_ds,train_ds,train_ds]) #operation 1


   trainer = QuestionAnsweringTrainer( #huggingface trainer
        model=model,
        args=training_args,
        train_dataset=train_ds,
        eval_dataset= None,
        eval_examples=None,
        answer_column_name=answer_column,
        dataset_name="squad",
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
    )

with operation 1, the GPU memory increases from 16G to 23G

Comments:
does the trainer save it in GPU? sooo curious... how to fix it

It's my bad. didn't limit the input length
Add splits to MBPP dataset
State: closed

This PR addresses https://github.com/huggingface/datasets/issues/4795

Comments:
```
(env) cwarny@Cedrics-Air datasets % RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_mbpp
================================================================================================ test session starts =================================================================================================
platform darwin -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0
rootdir: /Users/cwarny/datasets, configfile: setup.cfg
collected 1 item                                                                                                                                                                                                     

tests/test_dataset_common.py .                                                                                                                                                                                 [100%]

================================================================================================= 1 passed in 1.12s ==================================================================================================
(env) cwarny@Cedrics-Air datasets % RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_mbpp               
================================================================================================ test session starts =================================================================================================
platform darwin -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0
rootdir: /Users/cwarny/datasets, configfile: setup.cfg
collected 1 item                                                                                                                                                                                                     

tests/test_dataset_common.py .                                                                                                                                                                                 [100%]

================================================================================================= 1 passed in 0.35s ==================================================================================================

```

_The documentation is not available anymore as the PR was closed or merged._

Hi @cwarny ! Thanks for adding the correct splits :)

You can fix the CI error by running `make style` - this should reformat the dataset script

done
Trec Dataset has incorrect labels
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
Both coarse and fine labels seem to be out of line.


## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = "trec"
raw_datasets = load_dataset(dataset)
df = pd.DataFrame(raw_datasets["test"])
df.head()
```

## Expected results
text (string) | coarse_label (class label) | fine_label (class label)
-- | -- | --
How far is it from Denver to Aspen ? | 5  	(NUM) | 40  	(NUM:dist)
What county is Modesto , California in ? | 4  	(LOC) | 32  	(LOC:city)
Who was Galileo ? | 3  	(HUM) | 31  	(HUM:desc)
What is an atom ? | 2  	(DESC) | 24  	(DESC:def)
When did Hawaii become a state ? | 5  	(NUM) | 39  	(NUM:date)

## Actual results
  index | label-coarse  |label-fine                  |                    text
-- |-- | -- | --
0    |         4     |     40     | How far is it from Denver to Aspen ?
1      |       5        |  21  | What county is Modesto , California in ?
2        |     3        |  12        |                 Who was Galileo ?
3         |    0       |    7         |                What is an atom ?
4         |    4       |    8       |   When did Hawaii become a state ?

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-1086-azure-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3



Comments:
Thanks for reporting, @wmpauli. 

Indeed we recently fixed this issue:
- #4801 

The fix will be accessible after our next library release. In the meantime, you can have it by passing `revision="main"` to `load_dataset`.
Remove deprecated identical_ok
State: closed

`huggingface-hub` says that the `identical_ok` argument of `HfApi.upload_file` is now deprecated, and will be removed soon. It even has no effect at the moment when it's passed:

```python
Args:
...
    identical_ok (`bool`, *optional*, defaults to `True`):
        Deprecated: will be removed in 0.11.0.
        Changing this value has no effect.
...
```

There was only one occurence of `identical_ok=False` but it's maybe not worth adding a check ti verify if the files were the same.

cc @mariosasko 

Comments:
_The documentation is not available anymore as the PR was closed or merged._