helmo/github-issues huggingface dataset:
ImageFolder raises an error with parameters drop_metadata=True and drop_labels=False when metadata.jsonl is present
State: closed | Author: polinaeterna (CONTRIBUTOR)
Labels: name: bug

## Describe the bug

If you pass `drop_metadata=True` and `drop_labels=False` when a `data_dir` contains at least one `matadata.jsonl` file, you will get a KeyError. This is probably not a very useful case but we shouldn't get an error anyway. Asking users to move metadata files manually outside `data_dir` or pass features manually (when there is a tool that can infer them automatically) don't look like a good idea to me either.

## Steps to reproduce the bug
### Clone an example dataset from the Hub
```bash
git clone https://huggingface.co/datasets/nateraw/test-imagefolder-metadata
```
### Try to load it
```python
from datasets import load_dataset
ds = load_dataset("test-imagefolder-metadata", drop_metadata=True, drop_labels=False)
```
or even just
```python
ds = load_dataset("test-imagefolder-metadata", drop_metadata=True)
```
as `drop_labels=False` is a default value.

## Expected results
A DatasetDict object with two features: `"image"` and `"label"`.

## Actual results
```
Traceback (most recent call last):
  File "/home/polina/workspace/datasets/debug.py", line 18, in <module>
    ds = load_dataset(
  File "/home/polina/workspace/datasets/src/datasets/load.py", line 1732, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 1227, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 1218, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1596, in encode_example
    return encode_nested_example(self, example)
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1165, in encode_nested_example
    {
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1165, in <dictcomp>
    {
  File "/home/polina/workspace/datasets/src/datasets/utils/py_utils.py", line 249, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/home/polina/workspace/datasets/src/datasets/utils/py_utils.py", line 249, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
KeyError: 'label'
```

## Environment info
`datasets` master branch 

- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.14.0-1042-oem-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.1
- Pandas version: 1.4.1
Data type is not recognized when using datetime.time
State: closed | Author: severo (COLLABORATOR)
Labels: name: bug

## Describe the bug

Creating a dataset from a pandas dataframe with `datetime.time` format generates an error.

## Steps to reproduce the bug
```python
import pandas as pd
from datetime import time
from datasets import Dataset
df = pd.DataFrame({"feature_name": [time(1, 1, 1)]})
dataset = Dataset.from_pandas(df)
```

## Expected results

The dataset should be created.

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 823, in from_pandas
    return cls(table, info=info, split=split)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 679, in __init__
    inferred_features = Features.from_arrow_schema(arrow_table.schema)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1551, in from_arrow_schema
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1551, in <dictcomp>
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1315, in generate_from_arrow_type
    return Value(dtype=_arrow_to_datasets_dtype(pa_type))
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 83, in _arrow_to_datasets_dtype
    return f"time64[{arrow_type.unit}]"
AttributeError: 'pyarrow.lib.DataType' object has no attribute 'unit'
```

## Environment info

- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.13.0-1031-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.4.2

Comments: cc @mariosasko 

Hi, thanks for reporting! I'm investigating the issue.
np arrays get turned into native lists
State: open | Author: ZhaofengWu (NONE)
Labels: name: bug

## Describe the bug
When attaching an `np.array` field, it seems that it automatically gets turned into a list (see below). Why is this happening? Could it lose precision? Is there a way to make sure this doesn't happen?

## Steps to reproduce the bug
```python
>>> import datasets, numpy as np
>>> dataset = datasets.load_dataset("glue", "mrpc")["validation"]
Reusing dataset glue (...)
100%|███████████████████████████████████████████████| 3/3 [00:00<00:00, 1360.61it/s]
>>> dataset2 = dataset.map(lambda x: {"tmp": np.array([0.5])}, batched=False)
100%|██████████████████████████████████████████| 408/408 [00:00<00:00, 10819.97ex/s]
>>> dataset2[0]["tmp"]
[0.5]
>>> type(dataset2[0]["tmp"])
<class 'list'>
```

## Expected results
`dataset2[0]["tmp"]` should be an `np.ndarray`.

## Actual results
It's a list.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: mac, though I'm pretty sure it happens on a linux machine too
- Python version: 3.9.7
- PyArrow version: 6.0.1


Comments: If you add the line `dataset2.set_format('np')` before calling `dataset2[0]['tmp']` it should return `np.ndarray`.
I believe internally it will not store it as a list, it is only returning a list when you index it.

```
In [1]: import datasets, numpy as np
In [2]: dataset = datasets.load_dataset("glue", "mrpc")["validation"]
In [3]: dataset2 = dataset.map(lambda x: {"tmp": np.array([0.5])}, batched=False)
In [4]: dataset2[0]["tmp"]
Out[4]: [0.5]

In [5]: dataset2.set_format('np')

In [6]: dataset2[0]["tmp"]
Out[6]: array([0.5])
```

I see, thanks! Any idea if the default numpy → list conversion might cause precision loss?

I'm not super familiar with our datasets works internally, but I think your `np` array will be stored in a `pyarrow` format, and then you take a view of this as a python array. In which case, I think the precision should be preserved.
contribute data loading for object detection datasets with yolo data format
State: open | Author: faizankshaikh (NONE)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
At the moment, HF datasets loads [image classification datasets](https://huggingface.co/docs/datasets/image_process) out-of-the-box. There could be a data loader for loading standard object detection datasets ([original discussion here](https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/discussions/2))

**Describe the solution you'd like**
I wrote a [custom script](https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/blob/main/detect_chess_pieces.py) to load dataset which has YOLO data format. 

**Describe alternatives you've considered**
The script can either be a standalone dataset builder, or a modified version of `ImageFolder`

**Additional context**
I would be happy to contribute to this, but I would do it at a very slow pace (maybe a month or two) as I have my exams approaching 😄 


Comments: Hi! The `imagefolder` script is already quite complex, so a standalone script sounds better. Also, I suggest we create an org on the Hub (e.g. `hf-loaders`) and store such scripts there for easier maintenance rather than having them as packaged modules (IMO only very generic loaders should be packaged). WDYT @lhoestq @albertvillanova @polinaeterna?

@mariosasko sounds good to me!


Thank you for the suggestion @mariosasko . I agree with the point, but I have a few doubts

1. How would the user access the script if it's not a part of the core codebase?
2. Could you direct me as to what will be the tasks I have to do to contribute to the code? As per my understanding, it would be like
    1. Create a new org "hf-loaders" and add you (and more HF people) to the org
    2. Add data loader script as a (model?)
    3. Test it with a dataset on HF hub
3. We should maybe brainstorm as to which public datasets have this format (YOLO type) and are the most important ones to test the script with. We can even add the datasets on HF Hub alongside the script

1. Like this: `load_dataset("hf-loaders/yolo", data_files=...)`
2. The steps would be:
    1. Create a new org `hf-community-loaders` (IMO a better name than "hf-loaders") and add me (as an admin)
    2. Create a new dataset repo `yolo` and add the loading script to it (`yolo.py`)
    3. Open a discussion to request our review
4. I like this idea. Another option is to add snippets that describe how to load such datasets using the `yolo` loader.
Fix `embed_storage` on features inside lists/sequences
State: closed | Author: mariosasko (COLLABORATOR)

Add a dedicated function for embed_storage to always preserve the embedded/casted arrays (and to have more control over `embed_storage` in general).

Fix #4591 

~~(Waiting for #4608 to be merged to mark this PR as ready for review - required for fixing `xgetsize` in private repos)~~ Done!

Comments: _The documentation is not available anymore as the PR was closed or merged._
Ensure ConcatenationTable.cast uses target_schema metadata
State: closed | Author: dtuit (CONTRIBUTOR)

Currently, `ConcatenationTable.cast` does not use target_schema metadata when casting subtables. This causes an issue when using cast_column and the underlying table is a ConcatenationTable.

Code example of where issue arrises:
```
from datasets import Dataset, Image

column1 = [0, 1]
image_paths = ['/images/image1.jpg', '/images/image2.jpg']

ds = Dataset.from_dict({"column1": column1})
ds = ds.add_column("image", image_paths)
ds.cast_column("image", Image()) # Fails here
```
Output
```
...
TypeError: Couldn't cast array of type
string
to
{'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='string', id=None)}
```


Comments: Hi @lhoestq, Thanks for the detailed comment. I've tested the suggested approach and can confirm it works for the testcase outlined above! The PR is updated with the changes.

_The documentation is not available anymore as the PR was closed or merged._
Release 2.3.0 broke custom iterable datasets
State: closed | Author: aapot (NONE)
Labels: name: bug

## Describe the bug
Trying to iterate examples from custom iterable dataset fails to bug introduced in `torch_iterable_dataset.py` since the release of 2.3.0. 

## Steps to reproduce the bug
```python
next(iter(custom_iterable_dataset))
```

## Expected results
`next(iter(custom_iterable_dataset))` should return examples from the dataset

## Actual results
```
/usr/local/lib/python3.7/dist-packages/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py in _set_fsspec_for_multiprocess()
     16     See https://github.com/fsspec/gcsfs/issues/379
     17     """
---> 18     fsspec.asyn.iothread[0] = None
     19     fsspec.asyn.loop[0] = None
     20 

AttributeError: module 'fsspec' has no attribute 'asyn'
```

## Environment info
- `datasets` version: 2.3.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 8.0.0
- Pandas version: 1.3.5


Comments: Apparently, `fsspec` does not allow access to attribute-based modules anymore, such as `fsspec.async`.

However, this is a fairly simple fix:
- Change the import to: `from fsspec import asyn`;
- Change line 18 to: `asyn.iothread[0] = None`;
- Change line 19 to `asyn.loop[0] = None`.

Hi! I think it's easier to replace `import fsspec` with `import fsspec.asyn` and leave the rest unchanged. @gugarosa Are you interested in submitting a PR?

Perfect, it is even better!

Just submitted the PR: #4630.

Thank you!
Preserve member order by MockDownloadManager.iter_archive
State: closed | Author: albertvillanova (MEMBER)

Currently, `MockDownloadManager.iter_archive` yields paths to archive members in an order given by `path.rglob("*")`, which migh not be the same order as in the original archive.

See issue in:
- https://github.com/huggingface/datasets/pull/4579#issuecomment-1172135027

This PR fixes the order of the members yielded by `MockDownloadManager.iter_archive` so that it is the same as in the original archive.

Comments: _The documentation is not available anymore as the PR was closed or merged._
librispeech dataset has to download whole subset when specifing the split to use
State: closed | Author: sunhaozhepy (NONE)
Labels: name: bug

## Describe the bug
librispeech dataset has to download whole subset when specifing the split to use

## Steps to reproduce the bug
see below
# Sample code to reproduce the bug
```
!pip install datasets
from datasets import load_dataset
raw_dataset = load_dataset("librispeech_asr", "clean", split="train.100")
```

## Expected results
The split "train.clean.100" is downloaded.

## Actual results
All four splits in "clean" subset is downloaded.

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments: Hi! You can use streaming to fetch only a subset of the data:
```python
raw_dataset = load_dataset("librispeech_asr", "clean", split="train.100", streaming=True)
```
Also, we plan to make it possible to download a particular split in the non-streaming mode, but this task is not easy due to how our dataset scripts are structured.

Hi,

That's a great help. Thank you very much.
Fix xisfile, xgetsize, xisdir, xlistdir in private repo
State: closed | Author: lhoestq (MEMBER)

`xisfile` is working in a private repository when passing a chained URL to a file inside an archive, e.g. `zip://a.txt::https://huggingface/datasets/username/dataset_name/resolve/main/data.zip`. However it's not working when passing a simple file `https://huggingface/datasets/username/dataset_name/resolve/main/data.zip`.

This is because the authentication headers are not passed correctly in this case.

This is causing dataset streaming to fail in private parquet repositories, as noted in https://github.com/huggingface/datasets/issues/4605

I fixed `xisfile` and the other functions that behave the same way: xgetsize, xisdir and xlistdir

TODO:
- [x] tests

fix https://github.com/huggingface/datasets/issues/4605

Comments: _The documentation is not available anymore as the PR was closed or merged._

Added tests for xisfile, xgetsize, xlistdir and xglob for private repos, and also tests for xwalk that was untested
Align more metadata with other repo types (models,spaces)
State: closed | Author: julien-c (MEMBER)

see also associated PR on the `datasets-tagging` Space: https://huggingface.co/spaces/huggingface/datasets-tagging/discussions/2 (to merge after this one is merged)

Comments: _The documentation is not available anymore as the PR was closed or merged._

I just set a default value (None) for the deprecated licenses and languages fields, which should fix most of the CI failures.

Note that the CI should still be red because you edited many dataset cards and they're still missing some content - but this is unrelated to this PR so we can ignore these failures

thanks so much @lhoestq !!

There's also a follow-up PR to this one, in #4613 – I would suggest to merge all of them at the same time and hope not too many things are broken 🙀 🙀 

Alright merging this one now, let's see how broken things get
evaluation result changes after `datasets` version change
State: closed | Author: thnkinbtfly (NONE)
Labels: name: bug

## Describe the bug
evaluation result changes after `datasets` version change

## Steps to reproduce the bug
1. Train a model on WikiAnn
2. reload the ckpt -> test accuracy becomes same as eval accuracy
3. such behavior is gone after downgrading `datasets`

https://colab.research.google.com/drive/1kYz7-aZRGdayaq-gDTt30tyEgsKlpYOw?usp=sharing

## Expected results
evaluation result shouldn't change before/after `datasets` version changes

## Actual results
evaluation result changes before/after `datasets` version changes

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: colab
- Python version: 3.7.13
- PyArrow version: 6.0.1

Q. How could the evaluation result change before/after `datasets` version changes?

Comments: Hi! The GH/no-namespace datasets versioning is synced with the version of the `datasets` lib, which means that the `wikiann` script was modified between the two compared versions. In this scenario, you can ensure reproducibility by pinning the script version, which is done by passing `revision="x.y.z"` (e.g. `revision="2.2.0"`) to `load_dataset.`
Dataset Viewer issue for boris/gis_filtered
State: closed | Author: WaterKnight1998 (NONE)
Labels: name: streaming

### Link

https://huggingface.co/datasets/boris/gis_filtered/viewer/boris--gis_filtered/train

### Description

When I try to access this from the website I get this error:

Status code:   400
Exception:     ClientResponseError
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/boris/gis_filtered/resolve/80b805053ce61d4eb487b6b8d9095d775c2c466e/data/train/0000.parquet')

If I try to load with code I also get the same issue: 
```python
dataset2_train=load_dataset("boris/gis_filtered", use_auth_token=os.environ["HF_TOKEN"],split="train",streaming=True)
dataset2_validation=load_dataset("boris/gis_filtered", use_auth_token=os.environ["HF_TOKEN"], split="validation",streaming=True)
```

### Owner

No

Comments: Yes, this dataset is "gated": you first have to go to https://huggingface.co/datasets/boris/gis_filtered and click "Access repository" (if you accept to share your contact information with the repository authors).

I already did that, it returns error when using streaming

Oh, sorry, I misread. Looking at it. Maybe @huggingface/datasets or @SBrandeis 

I could reproduce the error, even though I provided my token and accepted the gate form. It looks like an error from `datasets`

This is indeed a bug in `datasets`. Parquet datasets in gated/private repositories can't be streamed properly, which caused the viewer to fail. I opened a PR at https://github.com/huggingface/datasets/pull/4608
Remove multiple config section
State: closed | Author: stevhliu (MEMBER)
Labels: name: documentation

This PR removes docs for a future feature and redirects to #4578 instead. See this [discussion](https://huggingface.slack.com/archives/C034N0A7H09/p1656107063801969) for more details :) 

Comments: _The documentation is not available anymore as the PR was closed or merged._
Smooth-BLEU bug fixed
State: closed | Author: Aktsvigun (NONE)
Labels: name: transfer-to-evaluate

Hi,

the current implementation of smooth-BLEU contains a bug: it smoothes unigrams as well. Consequently, when both the reference and translation consist of totally different tokens, it anyway returns a non-zero value (please see the attached image). 

This however contradicts the source paper suggesting the smooth-BLEU _(Chin-Yew Lin, Franz Josef Och. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. COLING 2004.)_ :

> Add one count to the n-gram hit and total ngram count for n > 1. Therefore, for candidate translations with less than n words, they can still get a positive smoothed BLEU score from shorter n-gram matches; however if nothing matches then they will get zero scores. 

This pull request aims at fixing this bug.

I made a pull request in the target repository `tensorflow/nmt`, which implements this script, yet the last commit there is dating 19.02.2019 and I doubt whether this will be fixed promptly. Yet, this bug is critical, for instance for summarization datasets with short summaries (e.g. AESLC), since smoothing needs to be applied there. Therefore, the easiest solution that I found is to fork the repo and download this script directly from the forked fixed repo.

Kind,
Akim Tsvigun

<img width="516" alt="Снимок экрана 2022-06-29 в 17 49 27" src="https://user-images.githubusercontent.com/36672861/176466935-ac579e6d-6a93-4111-ab41-9b33056e7d47.png">
 

Comments: Thanks @Aktsvigun for your fix.

However, metrics in `datasets` are in deprecation mode:
- #4739

You should transfer this PR to the `evaluate` library: https://github.com/huggingface/evaluate

Just for context, here the link to the PR by @Aktsvigun on tensorflow/nmt:
- https://github.com/tensorflow/nmt/pull/488
Streaming issue for financial_phrasebank
State: closed | Author: lewtun (MEMBER)
Labels: name: hosted-on-google-drive

### Link

https://huggingface.co/datasets/financial_phrasebank/viewer/sentences_allagree/train

### Description

As reported by a community member using [AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/5#62bc217436d0e5d316a768f0), there seems to be a problem streaming this dataset:

```
Server error
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No

Comments: cc @huggingface/datasets: it seems like https://www.researchgate.net/ is flaky for datasets hosting (I put the "hosted-on-google-drive" tag since it's the same kind of issue I think)

Let's see if their license allows hosting their data on the Hub.

License is Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0).

We can host their data on the Hub.
Dataset Viewer issue for universal_dependencies
State: closed | Author: Jordy-VL (NONE)
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/universal_dependencies

### Description

invalid json response body at https://datasets-server.huggingface.co/splits?dataset=universal_dependencies reason: Unexpected token I in JSON at position 0

### Owner

_No response_

Comments: Thanks, looking at it!

Finally fixed! We updated the dataset viewer and it fixed the issue.

https://huggingface.co/datasets/universal_dependencies/viewer/aqz_tudet/train

<img width="1561" alt="Capture d’écran 2022-09-07 à 13 29 18" src="https://user-images.githubusercontent.com/1676121/188867795-4f7dd438-d4f2-46cd-8a92-20a37fb2d6bc.png">
Dataset Viewer issue with False positive PII redaction
State: closed | Author: cakiki (CONTRIBUTOR)

### Link

https://huggingface.co/datasets/cakiki/rosetta-code

### Description

Hello, I just noticed an entry being redacted that shouldn't have been:

`RootMeanSquare@Range[10]` is being displayed as `[email protected][10]`

### Owner

_No response_

Comments: The value is in the data, it's not an issue with the "dataset-viewer".

<img width="1161" alt="Capture d’écran 2022-06-29 à 10 25 51" src="https://user-images.githubusercontent.com/1676121/176389325-4d2a9a7f-1583-45b8-aa7a-960ffaa6a36a.png">

 Maybe open a PR: https://huggingface.co/datasets/cakiki/rosetta-code/discussions


This was indeed a scraping issue which I assumed was a display issue; sorry about that!