AniketTec/github-issues huggingface dataset:
Title: Support streaming cfq dataset
State: closed

Support streaming cfq dataset.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lhoestq I've been refactoring a little the code:
- Use less RAM by loading only the required samples: only if its index is in the splits file
- Start yielding "earlier" in streaming mode: for each `split_idx`:
  - either yield from buffer
  - or iterate over samples and either yield or buffer the sample
  
  The speed gain obviously depends on how the indexes are sorted in the split file:
    - Best case: indices are [1, 2, 3]
    - Worst case (no speed gain): indices are [3, 1, 2] or [3, 2, 1]

Let me know what you think.

I have to update the dummy data so that it aligns with the real data (inside the archive, the samples file `dataset.json` is the last member).

There is an issue when testing `test_load_dataset_cfq` with dummy data:
- `MockDownloadManager.iter_archive` yields FIRST `'cfq/dataset.json'`
- [`Streaming`]`DownloadManager.iter_archive` yields LAST `'cfq/dataset.json'` when using real data tar.gz archive

Note that this issue arises only with dummy data: loading the real dataset works smoothly for all configurations: I recreated the `dataset_infos.json` file to check it (it generated the same file).

This PR should be merged first:
- #4611

Impressive, thank you ! :o 

feel free to merge master into this branch, now that the files order is respected. You can merge if the CI is green :)
Title: [Multi Configs] Use directories to differentiate between subsets/configurations
State: open
Labels: description: New feature or request
name: enhancement

Currently to define several subsets/configurations of your dataset, you need to use a dataset script.

However it would be nice to have a no-code way to to this. 

For example we could specify different configurations of a dataset (for example, if a dataset contains different languages) with one directory per configuration.

These structures are not supported right now, but would be nice to have:


```
my_dataset_repository/
├── README.md
├── en/
│   ├── train.csv
│   └── test.csv
└── fr/
    ├── train.csv
    └── test.csv
```

Or with one directory per split:

```
my_dataset_repository/
├── README.md
├── en/
│   ├── train/
│   │   ├── shard_0.csv
│   │   └── shard_1.csv
│   └── test/
│       ├── shard_0.csv
│       └── shard_1.csv
└── fr/
    ├── train/
    │   ├── shard_0.csv
    │   └── shard_1.csv
    └── test/
        ├── shard_0.csv
        └── shard_1.csv
```

cc @stevhliu @albertvillanova 

This can be specified in the README as YAML with

```
configs:
- config_name: en
  data_dir: en
- config_name: fr
  data_dir: fr
```

Comments:
I want to be able to create folders in a model.

How to set new split names, instead of train/test/validation? For example, I have a local dataset, consists of several subsets, named "A", "B", and "C". How can I create a huggingface dataset, with splits A/B/C ?

The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

> The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

It works the same - you just need to use local paths instead of URLs
Title: Include `metadata.jsonl` in resolved data files
State: closed

Include `metadata.jsonl` in resolved data files.

Fix #4548 

@lhoestq ~~https://github.com/huggingface/datasets/commit/d94336d30eef17fc9abc67f67fa1c139661f4e75 adds support for metadata files placed at the root, and https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 accounts for nested metadata files also, but this results in more complex code. Let me know which one of these two approaches you prefer.~~ Maybe https://github.com/huggingface/datasets/commit/d94336d30eef17fc9abc67f67fa1c139661f4e75 is good enough for now (for the sake of simplicity). https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 breaks the imagefolder tests due to duplicates in the resolved metadata files. One way to fix this would be to resolve the metadata pattern only on parent directories, but this adds even more logic to `_get_data_files_patterns`, so not sure if this is what we should do.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I still don't know if the way we implemented data files resolution could support the metadata.jsonl file without bad side effects for the other packaged builders. In particular here if you have a folder of csv/parquet/whatever files and a metadata.jsonl file, it would return 
```
split: patterns_dict[split] + [METADATA_PATTERN]
```
which is a bit unexpected and can lead to errors.

Maybe this logic can be specific to imagefolder somehow ? This could be an additional pattern `["metadata.jsonl", "**/metadata.jsonl"]` just for imagefolder, that is only used when `data_files=` is not specified by the user.

I guess it's ok to have patterns that lead to duplicate metadata.jsonl files for imagefolder, since the imagefolder logic only considers the closest metadata file for each image.

What do you think ?

Yes, that's indeed the problem. My solution in https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 that accounts for that (include metadata files only if image files are present; not ideal): https://github.com/huggingface/datasets/blob/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95/src/datasets/data_files.py#L119-L125.
Perhaps a cleaner approach would be to check for metadata files after the packaged module type is inferred as `imagefolder` and append metadata files to already resolved data files (if there are any). WDYT?

@lhoestq 

> Perhaps a cleaner approach would be to check for metadata files after the packaged module type is inferred as imagefolder and append metadata files to already resolved data files (if there are any). WDYT?

I decided to go with this approach.

 Not sure if you meant the same thing with this comment:

> Maybe this logic can be specific to imagefolder somehow ? This could be an additional pattern ["metadata.jsonl", "**/metadata.jsonl"] just for imagefolder, that is only used when data_files= is not specified by the user.


It adds more code but is easy to follow IMO.


The CI still struggles but you can merge since at least one of the two WIN CI succeeded
Title: Fix evaluation metadata for ncbi_disease
State: closed
Labels: description: Contribution to a dataset script
name: dataset contribution

This PR fixes the task in the evaluation metadata and removes the metrics info as we've decided this is not a great way to propagate this information downstream.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As discussed with @lewtun, we are closing this PR, because it requires first the task names to be aligned between AutoTrain and datasets.
Title: Dataset Viewer issue for mlsum
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/mlsum/viewer/de/train

### Description

There's seems to be a problem with the download / streaming of this dataset:

```
Server error
Status code:   400
Exception:     BadZipFile
Message:       File is not a zip file
```

### Owner

No

Comments:
Thanks for reporting, @lewtun.

After investigation, it seems that the server https://gitlab.lip6.fr does not allow HTTP Range requests.

We are trying to find a workaround...
Title: move under the facebook org?
State: open

### Link

https://huggingface.co/datasets/gsarti/flores_101

### Description

It seems like streaming isn't supported for this dataset:

```
Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://dl.fbaipublicfiles.com/flores101/dataset/flores101_dataset.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

### Owner

No

Comments:
Related to https://github.com/huggingface/datasets/issues/4562#issuecomment-1166911751

I'll assign @albertvillanova 

I'm just wondering why we don't have this dataset under:
- the `facebook` namespace
- or the canonical dataset `flores`: why does this only have 2 languages?

fwiw: the dataset viewer is working. Renaming the issue
Title: Dataset sharding non-contiguous?
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
I'm not sure if this is a bug; more likely normal behavior but i wanted to double check.
Is it normal that `datasets.shard` does not produce chunks that, when concatenated produce the original ordering of the sharded dataset? 

This might be related to this pull request (https://github.com/huggingface/datasets/pull/4466) but I have to admit I did not properly look into the changes made.

## Steps to reproduce the bug
```python
max_shard_size = convert_file_size_to_int('300MB')
dataset_nbytes = dataset.data.nbytes
num_shards = int(dataset_nbytes / max_shard_size) + 1
num_shards = max(num_shards, 1)
print(f"{num_shards=}")
for shard_index in range(num_shards):
    shard = dataset.shard(num_shards=num_shards, index=shard_index)
    shard.to_parquet(f"tokenized/tokenized-{shard_index:03d}.parquet")
os.listdir('tokenized/')
```

## Expected results
I expected the shards to match the order of the data of the original dataset; i.e. `dataset[10]` being the same as `shard_1[10]` for example

## Actual results
Only the first element is the same; i.e. `dataset[0]` is the same as `shard_1[0]`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Comments:
This was silly; I was sure I'd looked for a `contiguous` argument, and was certain there wasn't one the first time I looked :smile:

Sorry about that.

Hi! You can pass `contiguous=True` to `.shard()` get contiguous shards. More info on this and the default behavior can be found in the [docs](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.shard).

EDIT: Answered as you closed the thread 😄 

Hahaha I'm sorry; my excuse is: it's Sunday. (Which makes me all the more grateful for your response :smiley: 

@mariosasko Sorry for reviving this, but I was curious as to why `contiguous=False` was the default. This might be a personal bias, but I feel that a user would expect the opposite to be the default. :thinking: 

This project started as a fork of TFDS, and `contiguous=False` is the default behavior [there](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard).
Title: Dataset Viewer issue for sst2
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/sst2

### Description

Not sure what is causing this, however it seems that `load_dataset("sst2")` also hangs (even though it downloads the files without problem):

```
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No

Comments:
Hi @lewtun, thanks for reporting.

I have checked locally and refreshed the preview and it seems working smooth now:
```python
In [8]: ds
Out[8]: 
DatasetDict({
    train: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 872
    })
    test: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 1821
    })
})
```

Could you confirm? 

Thanks @albertvillanova - it is indeed working now (not sure what caused the error in the first place). Closing this :)
Title: Document link #load_dataset_enhancing_performance points to nowhere
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
A clear and concise description of what the bug is.

![image](https://user-images.githubusercontent.com/11674033/175752806-5b066b92-9d28-4771-9112-5c8606f07741.png)


The [load_dataset_enhancing_performance](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#load_dataset_enhancing_performance) link [here](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.load_from_disk.keep_in_memory) points to nowhere, I guess it should point to https://huggingface.co/docs/datasets/v2.3.2/en/cache#improve-performance?


Comments:
Hi! This is indeed the link the docstring should point to. Are you interested in submitting a PR to fix this?

https://github.com/huggingface/datasets/blame/master/docs/source/cache.mdx#L93

There seems already an anchor here. Somehow it doesn't work. I am not very familiar with how this online documentation works.
Title: Add UFSC OCPap dataset
State: closed
Labels: description: Requesting to add a new dataset
name: dataset request

## Adding a Dataset
- **Name:** UFSC OCPap: Papanicolaou Stained Oral Cytology Dataset (v4)
- **Description:** The UFSC OCPap dataset comprises 9,797 labeled images of 1200x1600 pixels acquired from 5 slides of cancer diagnosed and 3 healthy of oral brush samples, from distinct patients.
- **Paper:** https://dx.doi.org/10.2139/ssrn.4119212
- **Data:**  https://data.mendeley.com/datasets/dr7ydy9xbk/1
- **Motivation:** real data of pap stained oral cytology samples

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
I will add this directly on the hub (same as #4486)—in https://huggingface.co/lapix
Title: Dataset Viewer issue for allocine
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/allocine

### Description

Not sure if this is a problem with `bz2` compression, but I thought these datasets could be streamed:

```
Status code:   400
Exception:     AttributeError
Message:       'TarContainedFile' object has no attribute 'readable'
```

### Owner

No

Comments:
I removed my assignment as @huggingface/datasets should be able to answer better than me


Let me have a look...

Thanks for the quick fix @albertvillanova 

Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content *sequentially* (no random access).

> Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content _sequentially_ (no random access).

Ah thanks for the clarification! I'll look out for this next time and implement the fix myself :)
Title: Add evaluation metadata to wmt14
State: closed
Labels: description: Contribution to a dataset script
name: dataset contribution


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4558). All of your documentation changes will be reflected on that endpoint.

As discussed with @lewtun, we are closing this PR, because it requires first the task names to be aligned between AutoTrain and datasets.
Title: Add evaluation metadata to wmt16
State: closed
Labels: description: Contribution to a dataset script
name: dataset contribution

Just to confirm: we should add this metadata via GitHub and not Hub PRs for canonical datasets right?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4557). All of your documentation changes will be reflected on that endpoint.

> Just to confirm: we should add this metadata via GitHub and not Hub PRs for canonical datasets right?

yes :)

As discussed with @lewtun, we are closing this PR, because it requires first the task names to be aligned between AutoTrain and datasets.
Title: Dataset Viewer issue for conll2003
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/conll2003/viewer/conll2003/test

### Description

Seems like a cache problem with this config / split:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/conll2003/__init__.py'
```

### Owner

No

Comments:
Fixed, thanks.
Title: Dataset Viewer issue for xtreme
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/xtreme/viewer/PAN-X.de/test

### Description

There seems to be a problem with the cache of this config / split:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/xtreme/349258adc25bb45e47de193222f95e68a44f7a7ab53c4283b3f007208a11bf7e/xtreme.py'
```

### Owner

No

Comments:
Fixed, thanks.
Title: Fix WMT dataset loading issue and docs update (Re-opened)
State: closed

This PR is a fix for #4354 

Changes are made for `wmt14`, `wmt15`, `wmt16`, `wmt17`, `wmt18`, `wmt19` and `wmt_t2t`. And READMEs are updated for the corresponding datasets.

Let me know, if any additional changes are required.

Thanks

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Stop dropping columns in to_tf_dataset() before we load batches
State: closed

`to_tf_dataset()` dropped unnecessary columns before loading batches from the dataset, but this is causing problems when using a transform, because the dropped columns might be needed to compute the transform. Since there's no real way to check which columns the transform might need, we skip dropping columns and instead drop keys from the batch after we load it.

cc @amyeroberts and https://github.com/huggingface/notebooks/pull/202

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lhoestq Rebasing fixed the test failures, so this should be ready to review now! There's still a failure on Win but it seems unrelated.

Gentle ping @lhoestq ! This is a simple fix (dropping columns after loading a batch from the dataset rather than with `.remove_columns()` to make sure we don't break transforms), and tests are green so we're ready for review!

@lhoestq Test is in!
Title: Tell users to upload on the hub directly
State: closed

As noted in https://github.com/huggingface/datasets/pull/4534, it is still not clear that it is recommended to add datasets on the Hugging Face Hub directly instead of GitHub, so I updated some docs.

Moreover since users won't be able to get reviews from us on the Hub, I added a paragraph to tell users that they can open a discussion and tag `datasets` maintainers for reviews.

Finally I removed the _previous good reasons_ to add a dataset on GitHub to only keep this one:

> In some rare cases it makes more sense to open a PR on GitHub. For example when you are not the author of the dataset and there is no clear organization / namespace that you can put the dataset under.

Does it sound good to you @albertvillanova @julien-c ?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks ! I updated the two remaining files
Title: Perform hidden file check on relative data file path
State: closed

Fix #4549 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm aware of this behavior, which is tricky to solve due to fsspec's hidden file handling (see https://github.com/huggingface/datasets/issues/4115#issuecomment-1108819538). I've tested some regex patterns to address this, and they seem to work (will push them on Monday; btw they don't break any of fsspec's tests, so maybe we can contribute this as an enhancement to them). Also, perhaps we should include the files starting with `__`  in the results again (we hadn't had issues with this pattern before). WDYT?

I see. Feel free to merge this one if it's good for you btw :)

> Also, perhaps we should include the files starting with __ in the results again (we hadn't had issues with this pattern before)

The point was mainly to ignore `__pycache__` directories for example. Also also for consistency with the iter_files/iter_archive which are already ignoring them

Very elegant solution! Feel free to merge if the CI is green after adding the tests.

CI failure is unrelated to this PR
Title: imdb source error
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
imdb dataset not loading

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("imdb")
```


## Expected results


## Actual results
```bash
06/23/2022 14:45:18 - INFO - datasets.builder - Dataset not on Hf google storage. Downloading and preparing it from source
06/23/2022 14:46:34 - INFO - datasets.utils.file_utils - HEAD request to http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz timed out, retrying... [1.0]
.....
ConnectionError: Couldn't reach http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz (ConnectTimeout(MaxRetryError("HTTPConnectionPool(host='ai.stanford.edu', port=80): Max retries exceeded with url: /~amaas/data/sentiment/aclImdb_v1.tar.gz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2d750cf690>, 'Connection to ai.stanford.edu timed out. (connect timeout=100)'))")))
```
## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments:
Thanks for reporting, @Muhtasham.

Indeed IMDB dataset is not accessible from yesterday, because the data is hosted on the data owners servers at Stanford (http://ai.stanford.edu/) and these are down due to a power outage originated by a fire: https://twitter.com/StanfordAILab/status/1539472302399623170?s=20&t=1HU1hrtaXprtn14U61P55w

As a temporary workaroud, you can load the IMDB dataset with this tweak:
```python
ds = load_dataset("imdb", revision="tmp-fix-imdb")
```
Title: FileNotFoundError when passing a data_file inside a directory starting with double underscores
State: closed
Labels: description: Something isn't working
name: bug

Bug experienced in the `accelerate` CI: https://github.com/huggingface/accelerate/runs/7016055148?check_suite_focus=true

This is related to https://github.com/huggingface/datasets/pull/4505 and the changes from https://github.com/huggingface/datasets/pull/4412

Comments:
I have consistently experienced this bug on GitHub actions when bumping to `2.3.2`

We're working on a fix ;)