Minimartzz/github-issues huggingface dataset:
Remove multiple config section
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

This PR removes docs for a future feature and redirects to #4578 instead. See this [discussion](https://huggingface.slack.com/archives/C034N0A7H09/p1656107063801969) for more details :) 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Smooth-BLEU bug fixed
State: closed
Labels: name: transfer-to-evaluate

Hi,

the current implementation of smooth-BLEU contains a bug: it smoothes unigrams as well. Consequently, when both the reference and translation consist of totally different tokens, it anyway returns a non-zero value (please see the attached image). 

This however contradicts the source paper suggesting the smooth-BLEU _(Chin-Yew Lin, Franz Josef Och. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. COLING 2004.)_ :

> Add one count to the n-gram hit and total ngram count for n > 1. Therefore, for candidate translations with less than n words, they can still get a positive smoothed BLEU score from shorter n-gram matches; however if nothing matches then they will get zero scores. 

This pull request aims at fixing this bug.

I made a pull request in the target repository `tensorflow/nmt`, which implements this script, yet the last commit there is dating 19.02.2019 and I doubt whether this will be fixed promptly. Yet, this bug is critical, for instance for summarization datasets with short summaries (e.g. AESLC), since smoothing needs to be applied there. Therefore, the easiest solution that I found is to fork the repo and download this script directly from the forked fixed repo.

Kind,
Akim Tsvigun

<img width="516" alt="Снимок экрана 2022-06-29 в 17 49 27" src="https://user-images.githubusercontent.com/36672861/176466935-ac579e6d-6a93-4111-ab41-9b33056e7d47.png">
 

Comments:
Thanks @Aktsvigun for your fix.

However, metrics in `datasets` are in deprecation mode:
- #4739

You should transfer this PR to the `evaluate` library: https://github.com/huggingface/evaluate

Just for context, here the link to the PR by @Aktsvigun on tensorflow/nmt:
- https://github.com/tensorflow/nmt/pull/488
Streaming issue for financial_phrasebank
State: closed
Labels: name: hosted-on-google-drive

### Link

https://huggingface.co/datasets/financial_phrasebank/viewer/sentences_allagree/train

### Description

As reported by a community member using [AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/5#62bc217436d0e5d316a768f0), there seems to be a problem streaming this dataset:

```
Server error
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No

Comments:
cc @huggingface/datasets: it seems like https://www.researchgate.net/ is flaky for datasets hosting (I put the "hosted-on-google-drive" tag since it's the same kind of issue I think)

Let's see if their license allows hosting their data on the Hub.

License is Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0).

We can host their data on the Hub.
Dataset Viewer issue for universal_dependencies
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/universal_dependencies

### Description

invalid json response body at https://datasets-server.huggingface.co/splits?dataset=universal_dependencies reason: Unexpected token I in JSON at position 0

### Owner

_No response_

Comments:
Thanks, looking at it!

Finally fixed! We updated the dataset viewer and it fixed the issue.

https://huggingface.co/datasets/universal_dependencies/viewer/aqz_tudet/train

<img width="1561" alt="Capture d’écran 2022-09-07 à 13 29 18" src="https://user-images.githubusercontent.com/1676121/188867795-4f7dd438-d4f2-46cd-8a92-20a37fb2d6bc.png">
Dataset Viewer issue with False positive PII redaction
State: closed

### Link

https://huggingface.co/datasets/cakiki/rosetta-code

### Description

Hello, I just noticed an entry being redacted that shouldn't have been:

`RootMeanSquare@Range[10]` is being displayed as `[email protected][10]`

### Owner

_No response_

Comments:
The value is in the data, it's not an issue with the "dataset-viewer".

<img width="1161" alt="Capture d’écran 2022-06-29 à 10 25 51" src="https://user-images.githubusercontent.com/1676121/176389325-4d2a9a7f-1583-45b8-aa7a-960ffaa6a36a.png">

 Maybe open a PR: https://huggingface.co/datasets/cakiki/rosetta-code/discussions


This was indeed a scraping issue which I assumed was a display issue; sorry about that!
load_from_disk suggests incorrect fix when used to load DatasetDict
State: closed
Labels: description: Something isn't working
name: bug

Edit: Please feel free to remove this issue. The problem was not the error message but the fact that the DatasetDict.load_from_disk does not support loading nested splits, i.e. if one of the splits is itself a DatasetDict. If nesting splits is an antipattern, perhaps the load_from_disk function can throw a warning indicating that?

Comments:
Fix error message when using load_from_disk to load DatasetDict
State: closed

Issue #4594 
Issue: When `datasets.load_from_disk` is wrongly used to load a `DatasetDict`, the error message suggests using `datasets.load_from_disk`, which is the same function that generated the error. 
Fix: The appropriate function which should be suggested instead is `datasets.dataset_dict.load_from_disk`.
Changes: Change the suggestion to say "Please use `datasets.dataset_dict.load_from_disk` instead."

Comments:
Generalize meta_path json file creation in load.py [#4540]
State: closed

# What does this PR do?

## Summary

*In function `_copy_script_and_other_resources_in_importable_dir`, using string split when generating `meta_path` throws error in the edge case raised in #4540.*

## Additions
-

## Changes
- Changed meta_path to use `os.path.splitext` instead of using `str.split` to generalize code.

## Deletions
-
## Issues Addressed : 

Fixes #4540

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@albertvillanova,  Can you please review this PR for Issue #4540 

@lhoestq  Thank you for merging the PR . Is there a slack channel for contributing to the datasets library. I would love to work on the library and make meaningful contributions.

Hi ! Sure feel free to join our discord ^^ 
https://discuss.huggingface.co/t/join-the-hugging-face-discord/11263 so that we can discuss together mor eeasily. Otherwise everything happens on github ;)
Host head_qa data on the Hub and fix NonMatchingChecksumError
State: closed

This PR:
- Hosts head_qa data on the Hub instead of Google Drive
- Fixes NonMatchingChecksumError

Fix https://huggingface.co/datasets/head_qa/discussions/1

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @albertvillanova ! Thanks for the fix ;)
Can I safely checkout from this branch to build `datasets` or it is preferable to wait until all CI tests pass?
Thanks 🙏 

@younesbelkada we have just merged this PR.
Validate new_fingerprint passed by user
State: closed

Users can pass the dataset fingerprint they want in `map` and other dataset transforms.

However the fingerprint is used to name cache files so we need to make sure it doesn't contain bad characters as mentioned in https://github.com/huggingface/datasets/issues/1718, and that it's not too long

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add binary classification task IDs
State: closed

As a precursor to aligning the task IDs in `datasets` and AutoTrain, we need a way to distinguish binary vs multiclass vs multilabel classification.

This PR adds binary classification to the task IDs to enable this.

Related AutoTrain issue: https://github.com/huggingface/autonlp-backend/issues/597

cc @abhishekkrthakur @SBrandeis 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4584). All of your documentation changes will be reflected on that endpoint.

> Awesome thanks ! Can you add it to https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts first please ? This is where we define the cross libraries tasks taxonomy ;)

Thanks for the tip! Done in https://github.com/huggingface/hub-docs/pull/217

I don't think we need to update this file anymore. We should remove it IMO, and simply update the dataset [tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging)

I'm closing this PR.
add_column should preserve _indexes
State: open

https://github.com/huggingface/datasets/issues/3769#issuecomment-1167146126

doing `.add_column("x",x_data)` also removed any `_indexes` on the dataset, decided this shouldn't be the case.

This was because `add_column` was creating a new `Dataset(...)` and wasn't possible to pass indexes on init.
with this PR now can pass 'indexes' on init through `IndexableMixin`

- [x] Added test

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4582). All of your documentation changes will be reflected on that endpoint.
Dataset Viewer issue for pn_summary
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/pn_summary/viewer/1.0.0/validation

### Description

Getting an index error on the `validation` and `test` splits:

```
Server error
Status code:   400
Exception:     IndexError
Message:       list index out of range
```

### Owner

No

Comments:
linked to https://github.com/huggingface/datasets/issues/4580#issuecomment-1168373066?

Note that I refreshed twice this dataset, and I still have (another) error on one of the splits

```
Status code:   400
Exception:     ClientResponseError
Message:       403, message='Forbidden', url=URL('https://doc-14-4c-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/pgotjmcuh77q0lk7p44rparfrhv459kp/1656403650000/11771870722949762109/*/16OgJ_OrfzUF_i3ftLjFn9kpcyoi7UJeO?e=download')
```

Like the three splits are processed in parallel by the workers, I imagine that the Google hosting is rate-limiting us.

cc @albertvillanova 



Exactly, Google Drive bans our loading scripts.

When possible, we should host somewhere else.
Dataset Viewer issue for multi_news
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/multi_news

### Description

Not sure what the index error is referring to here:

```
Status code:   400
Exception:     IndexError
Message:       list index out of range
```

### Owner

No

Comments:
Thanks for reporting, @lewtun.

I forced the refreshing of the preview and it worked OK for train and validation splits.

I guess the error has to do with the data files being hosted at Google Drive: this gives errors when requested automatically using scripts.
We should host them to fix the error. Let's see if the license allows that.

I guess we can host the data: https://github.com/Alex-Fabbri/Multi-News/blob/master/LICENSE.txt
Support streaming cfq dataset
State: closed

Support streaming cfq dataset.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lhoestq I've been refactoring a little the code:
- Use less RAM by loading only the required samples: only if its index is in the splits file
- Start yielding "earlier" in streaming mode: for each `split_idx`:
  - either yield from buffer
  - or iterate over samples and either yield or buffer the sample
  
  The speed gain obviously depends on how the indexes are sorted in the split file:
    - Best case: indices are [1, 2, 3]
    - Worst case (no speed gain): indices are [3, 1, 2] or [3, 2, 1]

Let me know what you think.

I have to update the dummy data so that it aligns with the real data (inside the archive, the samples file `dataset.json` is the last member).

There is an issue when testing `test_load_dataset_cfq` with dummy data:
- `MockDownloadManager.iter_archive` yields FIRST `'cfq/dataset.json'`
- [`Streaming`]`DownloadManager.iter_archive` yields LAST `'cfq/dataset.json'` when using real data tar.gz archive

Note that this issue arises only with dummy data: loading the real dataset works smoothly for all configurations: I recreated the `dataset_infos.json` file to check it (it generated the same file).

This PR should be merged first:
- #4611

Impressive, thank you ! :o 

feel free to merge master into this branch, now that the files order is respected. You can merge if the CI is green :)
[Multi Configs] Use directories to differentiate between subsets/configurations
State: open
Labels: description: New feature or request
name: enhancement

Currently to define several subsets/configurations of your dataset, you need to use a dataset script.

However it would be nice to have a no-code way to to this. 

For example we could specify different configurations of a dataset (for example, if a dataset contains different languages) with one directory per configuration.

These structures are not supported right now, but would be nice to have:


```
my_dataset_repository/
├── README.md
├── en/
│   ├── train.csv
│   └── test.csv
└── fr/
    ├── train.csv
    └── test.csv
```

Or with one directory per split:

```
my_dataset_repository/
├── README.md
├── en/
│   ├── train/
│   │   ├── shard_0.csv
│   │   └── shard_1.csv
│   └── test/
│       ├── shard_0.csv
│       └── shard_1.csv
└── fr/
    ├── train/
    │   ├── shard_0.csv
    │   └── shard_1.csv
    └── test/
        ├── shard_0.csv
        └── shard_1.csv
```

cc @stevhliu @albertvillanova 

This can be specified in the README as YAML with

```
configs:
- config_name: en
  data_dir: en
- config_name: fr
  data_dir: fr
```

Comments:
I want to be able to create folders in a model.

How to set new split names, instead of train/test/validation? For example, I have a local dataset, consists of several subsets, named "A", "B", and "C". How can I create a huggingface dataset, with splits A/B/C ?

The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

> The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

It works the same - you just need to use local paths instead of URLs
Include `metadata.jsonl` in resolved data files
State: closed

Include `metadata.jsonl` in resolved data files.

Fix #4548 

@lhoestq ~~https://github.com/huggingface/datasets/commit/d94336d30eef17fc9abc67f67fa1c139661f4e75 adds support for metadata files placed at the root, and https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 accounts for nested metadata files also, but this results in more complex code. Let me know which one of these two approaches you prefer.~~ Maybe https://github.com/huggingface/datasets/commit/d94336d30eef17fc9abc67f67fa1c139661f4e75 is good enough for now (for the sake of simplicity). https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 breaks the imagefolder tests due to duplicates in the resolved metadata files. One way to fix this would be to resolve the metadata pattern only on parent directories, but this adds even more logic to `_get_data_files_patterns`, so not sure if this is what we should do.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I still don't know if the way we implemented data files resolution could support the metadata.jsonl file without bad side effects for the other packaged builders. In particular here if you have a folder of csv/parquet/whatever files and a metadata.jsonl file, it would return 
```
split: patterns_dict[split] + [METADATA_PATTERN]
```
which is a bit unexpected and can lead to errors.

Maybe this logic can be specific to imagefolder somehow ? This could be an additional pattern `["metadata.jsonl", "**/metadata.jsonl"]` just for imagefolder, that is only used when `data_files=` is not specified by the user.

I guess it's ok to have patterns that lead to duplicate metadata.jsonl files for imagefolder, since the imagefolder logic only considers the closest metadata file for each image.

What do you think ?

Yes, that's indeed the problem. My solution in https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 that accounts for that (include metadata files only if image files are present; not ideal): https://github.com/huggingface/datasets/blob/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95/src/datasets/data_files.py#L119-L125.
Perhaps a cleaner approach would be to check for metadata files after the packaged module type is inferred as `imagefolder` and append metadata files to already resolved data files (if there are any). WDYT?

@lhoestq 

> Perhaps a cleaner approach would be to check for metadata files after the packaged module type is inferred as imagefolder and append metadata files to already resolved data files (if there are any). WDYT?

I decided to go with this approach.

 Not sure if you meant the same thing with this comment:

> Maybe this logic can be specific to imagefolder somehow ? This could be an additional pattern ["metadata.jsonl", "**/metadata.jsonl"] just for imagefolder, that is only used when data_files= is not specified by the user.


It adds more code but is easy to follow IMO.


The CI still struggles but you can merge since at least one of the two WIN CI succeeded
Fix evaluation metadata for ncbi_disease
State: closed
Labels: description: Contribution to a dataset script
name: dataset contribution

This PR fixes the task in the evaluation metadata and removes the metrics info as we've decided this is not a great way to propagate this information downstream.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As discussed with @lewtun, we are closing this PR, because it requires first the task names to be aligned between AutoTrain and datasets.
Dataset Viewer issue for mlsum
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/mlsum/viewer/de/train

### Description

There's seems to be a problem with the download / streaming of this dataset:

```
Server error
Status code:   400
Exception:     BadZipFile
Message:       File is not a zip file
```

### Owner

No

Comments:
Thanks for reporting, @lewtun.

After investigation, it seems that the server https://gitlab.lip6.fr does not allow HTTP Range requests.

We are trying to find a workaround...
move under the facebook org?
State: open

### Link

https://huggingface.co/datasets/gsarti/flores_101

### Description

It seems like streaming isn't supported for this dataset:

```
Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://dl.fbaipublicfiles.com/flores101/dataset/flores101_dataset.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

### Owner

No

Comments:
Related to https://github.com/huggingface/datasets/issues/4562#issuecomment-1166911751

I'll assign @albertvillanova 

I'm just wondering why we don't have this dataset under:
- the `facebook` namespace
- or the canonical dataset `flores`: why does this only have 2 languages?

fwiw: the dataset viewer is working. Renaming the issue
Dataset sharding non-contiguous?
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
I'm not sure if this is a bug; more likely normal behavior but i wanted to double check.
Is it normal that `datasets.shard` does not produce chunks that, when concatenated produce the original ordering of the sharded dataset? 

This might be related to this pull request (https://github.com/huggingface/datasets/pull/4466) but I have to admit I did not properly look into the changes made.

## Steps to reproduce the bug
```python
max_shard_size = convert_file_size_to_int('300MB')
dataset_nbytes = dataset.data.nbytes
num_shards = int(dataset_nbytes / max_shard_size) + 1
num_shards = max(num_shards, 1)
print(f"{num_shards=}")
for shard_index in range(num_shards):
    shard = dataset.shard(num_shards=num_shards, index=shard_index)
    shard.to_parquet(f"tokenized/tokenized-{shard_index:03d}.parquet")
os.listdir('tokenized/')
```

## Expected results
I expected the shards to match the order of the data of the original dataset; i.e. `dataset[10]` being the same as `shard_1[10]` for example

## Actual results
Only the first element is the same; i.e. `dataset[0]` is the same as `shard_1[0]`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Comments:
This was silly; I was sure I'd looked for a `contiguous` argument, and was certain there wasn't one the first time I looked :smile:

Sorry about that.

Hi! You can pass `contiguous=True` to `.shard()` get contiguous shards. More info on this and the default behavior can be found in the [docs](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.shard).

EDIT: Answered as you closed the thread 😄 

Hahaha I'm sorry; my excuse is: it's Sunday. (Which makes me all the more grateful for your response :smiley: 

@mariosasko Sorry for reviving this, but I was curious as to why `contiguous=False` was the default. This might be a personal bias, but I feel that a user would expect the opposite to be the default. :thinking: 

This project started as a fork of TFDS, and `contiguous=False` is the default behavior [there](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard).
Dataset Viewer issue for sst2
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/sst2

### Description

Not sure what is causing this, however it seems that `load_dataset("sst2")` also hangs (even though it downloads the files without problem):

```
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No

Comments:
Hi @lewtun, thanks for reporting.

I have checked locally and refreshed the preview and it seems working smooth now:
```python
In [8]: ds
Out[8]: 
DatasetDict({
    train: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 872
    })
    test: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 1821
    })
})
```

Could you confirm? 

Thanks @albertvillanova - it is indeed working now (not sure what caused the error in the first place). Closing this :)