SurAyush/github_issues huggingface dataset:
Issue: Dataset Viewer issue for Jean-Baptiste/wikiner_fr
State: closed
Author: severo

### Link

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr

### Description

```
Error code:   StreamingRowsError
Exception:    FileNotFoundError
Message:      [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
Traceback:    Traceback (most recent call last):
               File "/src/services/worker/src/worker/responses/first_rows.py", line 337, in get_first_rows_response
                 rows = get_rows(dataset, config, split, streaming=True, rows_max_number=rows_max_number, hf_token=hf_token)
               File "/src/services/worker/src/worker/utils.py", line 123, in decorator
                 return func(*args, **kwargs)
               File "/src/services/worker/src/worker/responses/first_rows.py", line 77, in get_rows
                 rows_plus_one = list(itertools.islice(ds, rows_max_number + 1))
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 718, in __iter__
                 for key, example in self._iter():
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 708, in _iter
                 yield from ex_iterable
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 112, in __iter__
                 yield from self.generate_examples_fn(**self.kwargs)
               File "/tmp/modules-cache/datasets_modules/datasets/Jean-Baptiste--wikiner_fr/683a580ba6ec769d508f7dfc603a651667b0ed3817b1ae5bfd45f97cc024923f/wikiner_fr.py", line 165, in _generate_examples
                 dataset = Dataset.load_from_disk(filepath)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1210, in load_from_disk
                 with open(Path(dataset_path, config.DATASET_STATE_JSON_FILENAME).as_posix(), encoding="utf-8") as state_file:
             FileNotFoundError: [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
```

Is it an error with the dataset script, or the data itself, @huggingface/datasets?

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr/tree/main 

### Owner

No

Comments:
The script uses `Dataset.load_from_disk`, which as you can expect, doesn't work in streaming mode.

It would probably be more practical to load the dataset locally using `Dataset.load_from_disk` first and then `push_to_hub` to upload it in Parquet on the Hub

I've transferred this issue to the Hub repo: https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr/discussions/3

I'm closing this.
Issue: Get a specific Exception when the dataset has no data
State: closed
Author: severo

In the dataset viewer on the Hub (https://huggingface.co/datasets/glue/viewer), we would like (https://github.com/huggingface/moon-landing/issues/3882) to show a specific message when the repository lacks any data files.

In that case, instead of showing a complex traceback, we want to show a call to action to help the user upload data.

To do that, it would be very helpful to know for sure that the repository is missing any (supported) data files.

It could be done by raising a custom exception, for example, `NoDataError`.

Comments:
Issue: delete the hardcoded license list in `datasets`
State: closed
Author: julien-c

> Feel free to delete the license list in `datasets` [...]
>
> Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238401662_

> [...], in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now?

_Originally posted by @julien-c in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238390659_

Comments:
Issue: fix: avoid casting tuples after Dataset.map
State: closed
Author: szmoro

This PR updates features.py to avoid casting tuples to lists when reading the results of Dataset.map as suggested by @lhoestq [here](https://github.com/huggingface/datasets/issues/4676#issuecomment-1187371367) in https://github.com/huggingface/datasets/issues/4676.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Fix missing tags in dataset cards
State: closed
Author: albertvillanova

Fix missing tags in dataset cards:
- aeslc
- empathetic_dialogues
- event2Mind
- gap
- iwslt2017
- newsgroup
- qa4mre
- scicite

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931
- #4979

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Running add_column() seems to corrupt existing sequence-type column info
State: closed
Author: derek-rocheleau

I have a dataset that contains a column ("foo") that is a sequence type of length 4. So when I run .to_pandas() on it, the resulting dataframe correctly contains 4 columns - foo_0, foo_1, foo_2, foo_3. So the 1st row of the dataframe might look like:

ds = load_dataset(...)
df = ds.to_pandas()

df:
foo_0 | foo_1 | foo_2 | foo_3
0.0 | 1.0 | 2.0 | 3.0

If I run .add_column("new_col", data) on the dataset, and then .to_pandas() on the resulting new dataset, the resulting dataframe contains only 2 columns - foo, new_col. The values in column foo are lists of length 4, the 4 elements that should have been split into separate columns. Dataframe 1st row would be:

ds = load_dataset(...)
new_ds = ds.add_column("new_col", data)
df = new_ds.to_pandas()

df:
foo | new_col
[0.0, 1.0, 2.0, 3.0] | new_val

I've explored the 2 datasets in a debugger and haven't noticed any changes to any attributes related to the foo column, but I can't determine why the dataframes are so different.

Comments:
Nevermind, I was incorrect.
Issue: Add `IterableDataset.from_generator` to the API
State: closed
Author: mariosasko

We've just added `Dataset.from_generator` to the API. It would also be cool to add `IterableDataset.from_generator` to support creating an iterable dataset from a generator.

cc @lhoestq 

Comments:
#take

Thanks @hamid-vakilzadeh ! Let us know if you have some questions or if we can help

Thank you! I certainly will reach out if I need any help.
Issue: Embed image/audio data in dl_and_prepare parquet
State: closed
Author: lhoestq

Embed the bytes of the image or audio files in the Parquet files directly, instead of having a "path" that points to a local file.

Indeed Parquet files are often used to share data or to be used by workers that may not have access to the local files.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: [doc] Fix broken snippet that had too many quotes
State: closed
Author: tomaarsen

Hello!

### Pull request overview
* Fix broken snippet in https://huggingface.co/docs/datasets/main/en/process that has too many quotes

### Details
The snippet in question can be found here: https://huggingface.co/docs/datasets/main/en/process#map
This screenshot shows the issue, there is a quote too many, causing the snippet to be colored incorrectly:
![image](https://user-images.githubusercontent.com/37621491/190640627-f7587362-0e44-4464-a5d1-a0b98df6986f.png)

The change speaks for itself.

Thank you for the detailed documentation, by the way. 

- Tom Aarsen


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Spent the day familiarising myself with the huggingface line of products, and happened to run into some small issues here and there. Magically, I've found exactly one small issue in `transformers`, one in `accelerate` and now one in `datasets`, hah!

As for this PR, the issue seems solved according to the [new PR documentation](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4986/en/process#map):
![image](https://user-images.githubusercontent.com/37621491/190646405-6afa06fa-9eac-48f6-ab30-2677944fb7b6.png)
Issue: Prefer split patterns from directories over split patterns from filenames
State: closed
Author: polinaeterna

related to https://github.com/huggingface/datasets/issues/4895


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can we merge this one since the issue this PR fixes was reported for the second time? I also think we don't need a test for this simple change.

@mariosasko sure! could you please approve it? 

Hi there @polinaeterna @mariosasko! I have installed 5.2.3.dev0, which should have this fix. Unfortunately, I am still getting the error:
`ValueError: Unknown split "validation". Should be one of ['train'].` When I call `load_dataset("csv", data_files=files, split=split)`

Any help would be greatly appreciated!
Issue: docs: ✏️ add links to the Datasets API
State: closed
Author: severo

I added some links to the Datasets API in the docs. See https://github.com/huggingface/datasets-server/pull/566 for a companion PR in the datasets-server. The idea is to improve the discovery of the API through the docs.

I'm a bit shy about pasting a lot of links to the API in the docs, so it's minimal for now. I'm interested in ideas to integrate the API better in these docs without being too much. cc @lhoestq @julien-c @albertvillanova @stevhliu.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

OK, thanks @lhoestq. I'll close this PR, and come back to it with @stevhliu once we work on https://github.com/huggingface/datasets-server/issues/568
Issue: Make `pyarrow` optional
State: closed
Author: KOLANICH

**Is your feature request related to a problem? Please describe.**
Is `pyarrow` really needed for every dataset?

**Describe the solution you'd like**
It is made optional.

**Describe alternatives you've considered**
Likely, no.


Comments:
The whole datasets library is pretty much a wrapper to pyarrow (just take a look at some of the source for a Dataset) https://github.com/huggingface/datasets/blob/51aef08ad7053c0bfe8f9a961207b26df15850d3/src/datasets/arrow_dataset.py#L639 

I think removing the pyarrow dependency would involve a complete rewrite / a different library with minimal functionality (datasets-lite ?)

Thanks for the proposal, @KOLANICH. And also thanks for your answer, @dconathan.

Indeed, we are using `pyarrow` as the backend for our datasets, in order to cache them and also allow memory-mapping (using datasets larger than your RAM memory).

One way to avoid using `pyarrow` could be loading the datasets in streaming mode, by passing `streaming=True` to `load_dataset`. This way you basically get a generator for the dataset; nothing is downloaded, nor cached. 

Thanks for the info. Could `datasets` then be made optional for `transformers` instead? I used `transformers` only to deal with pretrained models to deploy them (convert to ONNX, and then I use TVM), so I don't really need `pyarrow` and `datasets` by now.
Issue: Fix missing tags in dataset cards
State: closed
Author: albertvillanova

Fix missing tags in dataset cards:
- amazon_us_reviews
- art
- discofuse
- indic_glue
- ubuntu_dialogs_corpus

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Providing dataset size
State: open
Author: sashavor

**Is your feature request related to a problem? Please describe.**
Especially for big datasets like [LAION](https://huggingface.co/datasets/laion/laion2B-en/), it's hard to know exactly the downloaded size (because there are many files and you don't have their exact size when downloaded).

**Describe the solution you'd like**
Auto-populating the downloaded dataset size on the dataset page would be really useful, including that of each split (when there are some).

**Describe alternatives you've considered**
People should be adding this to dataset cards, but I don't think that is systematically the case :slightly_smiling_face: 

**Additional context**
Mentioned to @lhoestq 


Comments:
Hi @sashavor, thanks for your suggestion.

Until now we have the CLI command 
```
datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
```
that generates the `dataset_infos.json` with the size of the downloaded dataset, among other information.

We are currently in the middle of removing those JSON files and putting their information directly in the header of the `README.md` (as YAML tags). Normally, the CLI command should continue working but saving its output to the dataset card instead. See:
- #4926

Additionally, the download size can be inferred by doing HEAD requests to the files to be downloaded. And for files hosted on the hub you can even get the file sizes using the Hub API

Amazing @albertvillanova ! I think just having that information visible in the dataset info (without having to do any requests/additional coding) would be really useful :hugs:
Issue: Hope to adapt Python3.9 as soon as possible
State: open
Author: RedHeartSecretMan

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.


Comments:
Hi! `datasets` should work in Python 3.9. What kind of issue have you encountered?

There is this related issue already: https://github.com/huggingface/datasets/issues/4113
And I guess we need a CI job for 3.9 ^^

Perhaps we should report this issue in the `filelock` repo?
Issue: Add `fn_kwargs` param to `IterableDataset.map`
State: closed
Author: mariosasko

Add the `fn_kwargs` parameter to `IterableDataset.map`.

("Resolves" https://discuss.huggingface.co/t/how-to-use-large-image-text-datasets-in-hugging-face-hub-without-downloading-for-free/22780/3)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for adding this fix! 

Would it be possible to get `fn_kwargs` added to `IterableDatasetDict.map` as well? It looks like a very similar problem, and hopefully shouldn't be a huge change. 


Hi @brianhill11! https://github.com/huggingface/datasets/pull/5810 adds this (opened a couple of days ago). It should be merged soon.

That's fantastic news, thanks @mariosasko ! I'll give it a shot once the changes are merged in.
Issue: [GH->HF] Part 2: Remove all dataset scripts from github
State: closed
Author: lhoestq

Now that all the datasets live on the Hub we can remove the /datasets directory that contains all the dataset scripts of this repository

- [x] Needs https://github.com/huggingface/datasets/pull/4973 to be merged first
- [x] and PR to be enabled on the Hub for non-namespaced datasets

Comments:
_The documentation is not available anymore as the PR was closed or merged._

So this means metrics will be deleted from this repo in favor of the "evaluate" library? Maybe you guys could just redirect metrics to that library.

We are deprecating the metrics in `datasets` indeed and suggest users to switch to `evaluate` (via a warning message)

We'll keep the current metrics as they are for now, but they'll be completely removed at one point

I guess this is ready to merge ?

It should break nothing except one rare case:

If someone is using an old version of `datasets` to try to load a recent dataset. Indeed in that case it fetches the `main` branch on github to see if it exists. But since we're removing all the datasets, forward fetching won't work anymore.

e.g. if someone uses "imagenet-1k" with a version of `datasets` that didn't have it at that time. I checked on kibana and one single user would be affected with 4k downloads/months. It should still work for them though thanks to the `datasets` cache

But if they delete their cache, the workaround is... 🥁 update `datasets` 😅

Let's merge this on monday if we can, to make sure contributors who wanted to merge their dataset PRs here could do it

Alright, merging !
Issue: [GH->HF] Load datasets from the Hub
State: closed
Author: lhoestq

Currently datasets with no namespace (e.g. squad, glue) are loaded from github.

In this PR I changed this logic to use the Hugging Face Hub instead.

This is the first step in removing all the dataset scripts in this repository

related to discussions in https://github.com/huggingface/datasets/pull/4059 (I should have continued from this PR actually)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Duplicate of:
- #4059
Issue: Fix map batched with torch output
State: closed
Author: lhoestq

Reported in https://discuss.huggingface.co/t/typeerror-when-applying-map-after-set-format-type-torch/23067/2

Currently it fails  if one uses batched `map` and the map function returns a torch tensor.

I fixed it for torch, tf, jax and pandas series.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Preserve non-`input_colums` in `Dataset.map` if `input_columns` are specified
State: closed
Author: mariosasko

Currently, if the `input_columns` list in `Dataset.map` is specified, the columns not in that list are dropped after the `map` transform.

This makes the behavior inconsistent with `IterableDataset.map`.
 
(It seems this issue was introduced by mistake in https://github.com/huggingface/datasets/pull/2246) 

Fix https://github.com/huggingface/datasets/issues/4858

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Fix data URL and metadata of vivos dataset
State: closed
Author: albertvillanova

After contacting the authors of the VIVOS dataset to report that their data server is down, we have received a reply from Hieu-Thi Luong that their data is now hosted on Zenodo: https://doi.org/10.5281/zenodo.7068130

This PR updates their data URL and some metadata (homepage, citation and license).

Fix #4936.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Dataset without script does not support regular JSON data file
State: closed
Author: julien-c

### Link

https://huggingface.co/datasets/julien-c/label-studio-my-dogs

### Description

<img width="1115" alt="image" src="https://user-images.githubusercontent.com/326577/189422048-7e9c390f-bea7-4521-a232-43f049ccbd1f.png">


### Owner

Yes

Comments:
Hi @julien-c,

Out of the box, we only support JSON lines (NDJSON) data files, but your data file is a regular JSON file. The reason is we use `pyarrow.json.read_json` and this only supports line-delimited JSON.
Issue: Update setup.py
State: closed
Author: DCNemesis

exclude broken version of fsspec. See the [related issue](https://github.com/huggingface/datasets/issues/4961)

Comments:
Before addressing this PR, we should be sure about the issue. See my comment in:
- https://github.com/huggingface/datasets/issues/4961#issuecomment-1243376247

Once we know 2022.8.2 works, I'm closing this PR, as the corresponding issue.
Issue: Fix data URLs of compguesswhat dataset
State: closed
Author: albertvillanova

After we informed the `compguesswhat` dataset authors about an error with their data URLs, they have updated them:
- https://github.com/CompGuessWhat/compguesswhat.github.io/issues/1

This PR updates their data URLs in our loading script.

Related to:
- #3191

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.4.0/datasets/jsonl/jsonl.py
State: closed
Author: hasakikiki

Hi,
When I use load_dataset from local jsonl files, below error happens, and I type the link into the browser prompting me `404: Not Found`. I download the other `.py` files using the same method and it works. It seems that the server is missing the appropriate file, or it is a problem with the code version.

```
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b08342004c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")))

```



Comments:
I have solved this problem... The extension of the file should be `.json` not `.jsonl`
Issue: Add `Dataset.from_generator`
State: closed
Author: mariosasko

Add `Dataset.from_generator` to the API to allow creating datasets from data larger than RAM. The implementation relies on a packaged module not exposed in `load_dataset` to tie this method with `datasets`' caching mechanism.

Closes https://github.com/huggingface/datasets/issues/4417

Comments:
I restarted the builder PR job just in case

_The documentation is not available anymore as the PR was closed or merged._

CI is now green. https://github.com/huggingface/doc-builder/pull/296 explains why it failed.
Issue: Raise a more precise error when the URL is unreachable in streaming mode
State: open
Author: severo

See for example:

- https://github.com/huggingface/datasets/issues/3191
- https://github.com/huggingface/datasets/issues/3186

It would help provide clearer information on the Hub and help the dataset maintainer solve the issue by themselves quicker. Currently:

- https://huggingface.co/datasets/compguesswhat

  <img width="1029" alt="Capture d’écran 2022-09-08 à 15 51 37" src="https://user-images.githubusercontent.com/1676121/189139946-6deffb91-f21b-4281-8825-a98026c69740.png">

- https://huggingface.co/datasets/nli_tr

  <img width="1032" alt="Capture d’écran 2022-09-08 à 15 51 44" src="https://user-images.githubusercontent.com/1676121/189139963-d26490ed-ad23-48ea-9cfc-1ab9c4d08d0c.png">


cc @albertvillanova 

Comments:
Issue: CI test of TensorFlow is failing
State: closed
Author: albertvillanova

## Describe the bug
The following CI test fails: https://github.com/huggingface/datasets/runs/8246722693?check_suite_focus=true
```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - AssertionError:
```

Details:
```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw0] linux -- Python 3.7.13 /opt/hostedtoolcache/Python/3.7.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        def gen_random_output():
            model = layers.Dense(2)
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
        with temp_seed(42, set_tensorflow=True):
            out1 = gen_random_output()
        with temp_seed(42, set_tensorflow=True):
            out2 = gen_random_output()
        out3 = gen_random_output()
    
>       np.testing.assert_equal(out1, out2)
E       AssertionError: 
E       Arrays are not equal
E       
E       Mismatched elements: 2 / 2 (100%)
E       Max absolute difference: 0.84619296
E       Max relative difference: 16.083529
E        x: array([[-0.793581,  0.333286]], dtype=float32)
E        y: array([[0.052612, 0.539708]], dtype=float32)

tests/test_py_utils.py:149: AssertionError
```




Comments: