ahmedfarazsyk/datasets-github-issues huggingface dataset:
Dataset Viewer issue with False positive PII redaction
Author: cakiki
State: closed

### Link

https://huggingface.co/datasets/cakiki/rosetta-code

### Description

Hello, I just noticed an entry being redacted that shouldn't have been:

`RootMeanSquare@Range[10]` is being displayed as `[email protected][10]`

### Owner

_No response_

Comments: The value is in the data, it's not an issue with the "dataset-viewer".

<img width="1161" alt="Capture d’écran 2022-06-29 à 10 25 51" src="https://user-images.githubusercontent.com/1676121/176389325-4d2a9a7f-1583-45b8-aa7a-960ffaa6a36a.png">

 Maybe open a PR: https://huggingface.co/datasets/cakiki/rosetta-code/discussions


This was indeed a scraping issue which I assumed was a display issue; sorry about that!
load_from_disk suggests incorrect fix when used to load DatasetDict
Author: dvsth
State: closed
Labels: name: bug

Edit: Please feel free to remove this issue. The problem was not the error message but the fact that the DatasetDict.load_from_disk does not support loading nested splits, i.e. if one of the splits is itself a DatasetDict. If nesting splits is an antipattern, perhaps the load_from_disk function can throw a warning indicating that?
Dataset Viewer issue for pn_summary
Author: lewtun
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/pn_summary/viewer/1.0.0/validation

### Description

Getting an index error on the `validation` and `test` splits:

```
Server error
Status code:   400
Exception:     IndexError
Message:       list index out of range
```

### Owner

No

Comments: linked to https://github.com/huggingface/datasets/issues/4580#issuecomment-1168373066?

Note that I refreshed twice this dataset, and I still have (another) error on one of the splits

```
Status code:   400
Exception:     ClientResponseError
Message:       403, message='Forbidden', url=URL('https://doc-14-4c-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/pgotjmcuh77q0lk7p44rparfrhv459kp/1656403650000/11771870722949762109/*/16OgJ_OrfzUF_i3ftLjFn9kpcyoi7UJeO?e=download')
```

Like the three splits are processed in parallel by the workers, I imagine that the Google hosting is rate-limiting us.

cc @albertvillanova 



Exactly, Google Drive bans our loading scripts.

When possible, we should host somewhere else.
Dataset Viewer issue for multi_news
Author: lewtun
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/multi_news

### Description

Not sure what the index error is referring to here:

```
Status code:   400
Exception:     IndexError
Message:       list index out of range
```

### Owner

No

Comments: Thanks for reporting, @lewtun.

I forced the refreshing of the preview and it worked OK for train and validation splits.

I guess the error has to do with the data files being hosted at Google Drive: this gives errors when requested automatically using scripts.
We should host them to fix the error. Let's see if the license allows that.

I guess we can host the data: https://github.com/Alex-Fabbri/Multi-News/blob/master/LICENSE.txt
[Multi Configs] Use directories to differentiate between subsets/configurations
Author: lhoestq
State: open
Labels: name: enhancement

Currently to define several subsets/configurations of your dataset, you need to use a dataset script.

However it would be nice to have a no-code way to to this. 

For example we could specify different configurations of a dataset (for example, if a dataset contains different languages) with one directory per configuration.

These structures are not supported right now, but would be nice to have:


```
my_dataset_repository/
├── README.md
├── en/
│   ├── train.csv
│   └── test.csv
└── fr/
    ├── train.csv
    └── test.csv
```

Or with one directory per split:

```
my_dataset_repository/
├── README.md
├── en/
│   ├── train/
│   │   ├── shard_0.csv
│   │   └── shard_1.csv
│   └── test/
│       ├── shard_0.csv
│       └── shard_1.csv
└── fr/
    ├── train/
    │   ├── shard_0.csv
    │   └── shard_1.csv
    └── test/
        ├── shard_0.csv
        └── shard_1.csv
```

cc @stevhliu @albertvillanova 

This can be specified in the README as YAML with

```
configs:
- config_name: en
  data_dir: en
- config_name: fr
  data_dir: fr
```

Comments: I want to be able to create folders in a model.

How to set new split names, instead of train/test/validation? For example, I have a local dataset, consists of several subsets, named "A", "B", and "C". How can I create a huggingface dataset, with splits A/B/C ?

The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

> The document in https://huggingface.co/docs/datasets/dataset_script only tells me how to create datasets with subsets that is hosted on another server. How to do it if my datasets are local?

It works the same - you just need to use local paths instead of URLs
Dataset Viewer issue for mlsum
Author: lewtun
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/mlsum/viewer/de/train

### Description

There's seems to be a problem with the download / streaming of this dataset:

```
Server error
Status code:   400
Exception:     BadZipFile
Message:       File is not a zip file
```

### Owner

No

Comments: Thanks for reporting, @lewtun.

After investigation, it seems that the server https://gitlab.lip6.fr does not allow HTTP Range requests.

We are trying to find a workaround...
move under the facebook org?
Author: lewtun
State: open

### Link

https://huggingface.co/datasets/gsarti/flores_101

### Description

It seems like streaming isn't supported for this dataset:

```
Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://dl.fbaipublicfiles.com/flores101/dataset/flores101_dataset.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

### Owner

No

Comments: Related to https://github.com/huggingface/datasets/issues/4562#issuecomment-1166911751

I'll assign @albertvillanova 

I'm just wondering why we don't have this dataset under:
- the `facebook` namespace
- or the canonical dataset `flores`: why does this only have 2 languages?

fwiw: the dataset viewer is working. Renaming the issue
Dataset sharding non-contiguous?
Author: cakiki
State: closed
Labels: name: bug

## Describe the bug
I'm not sure if this is a bug; more likely normal behavior but i wanted to double check.
Is it normal that `datasets.shard` does not produce chunks that, when concatenated produce the original ordering of the sharded dataset? 

This might be related to this pull request (https://github.com/huggingface/datasets/pull/4466) but I have to admit I did not properly look into the changes made.

## Steps to reproduce the bug
```python
max_shard_size = convert_file_size_to_int('300MB')
dataset_nbytes = dataset.data.nbytes
num_shards = int(dataset_nbytes / max_shard_size) + 1
num_shards = max(num_shards, 1)
print(f"{num_shards=}")
for shard_index in range(num_shards):
    shard = dataset.shard(num_shards=num_shards, index=shard_index)
    shard.to_parquet(f"tokenized/tokenized-{shard_index:03d}.parquet")
os.listdir('tokenized/')
```

## Expected results
I expected the shards to match the order of the data of the original dataset; i.e. `dataset[10]` being the same as `shard_1[10]` for example

## Actual results
Only the first element is the same; i.e. `dataset[0]` is the same as `shard_1[0]`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Comments: This was silly; I was sure I'd looked for a `contiguous` argument, and was certain there wasn't one the first time I looked :smile:

Sorry about that.

Hi! You can pass `contiguous=True` to `.shard()` get contiguous shards. More info on this and the default behavior can be found in the [docs](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.shard).

EDIT: Answered as you closed the thread 😄 

Hahaha I'm sorry; my excuse is: it's Sunday. (Which makes me all the more grateful for your response :smiley: 

@mariosasko Sorry for reviving this, but I was curious as to why `contiguous=False` was the default. This might be a personal bias, but I feel that a user would expect the opposite to be the default. :thinking: 

This project started as a fork of TFDS, and `contiguous=False` is the default behavior [there](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard).
Dataset Viewer issue for sst2
Author: lewtun
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/sst2

### Description

Not sure what is causing this, however it seems that `load_dataset("sst2")` also hangs (even though it downloads the files without problem):

```
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No

Comments: Hi @lewtun, thanks for reporting.

I have checked locally and refreshed the preview and it seems working smooth now:
```python
In [8]: ds
Out[8]: 
DatasetDict({
    train: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 872
    })
    test: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 1821
    })
})
```

Could you confirm? 

Thanks @albertvillanova - it is indeed working now (not sure what caused the error in the first place). Closing this :)
Document link #load_dataset_enhancing_performance points to nowhere
Author: subercui
State: closed
Labels: name: bug

## Describe the bug
A clear and concise description of what the bug is.

![image](https://user-images.githubusercontent.com/11674033/175752806-5b066b92-9d28-4771-9112-5c8606f07741.png)


The [load_dataset_enhancing_performance](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#load_dataset_enhancing_performance) link [here](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.load_from_disk.keep_in_memory) points to nowhere, I guess it should point to https://huggingface.co/docs/datasets/v2.3.2/en/cache#improve-performance?


Comments: Hi! This is indeed the link the docstring should point to. Are you interested in submitting a PR to fix this?

https://github.com/huggingface/datasets/blame/master/docs/source/cache.mdx#L93

There seems already an anchor here. Somehow it doesn't work. I am not very familiar with how this online documentation works.
Add UFSC OCPap dataset
Author: johnnv1
State: closed
Labels: name: dataset request

## Adding a Dataset
- **Name:** UFSC OCPap: Papanicolaou Stained Oral Cytology Dataset (v4)
- **Description:** The UFSC OCPap dataset comprises 9,797 labeled images of 1200x1600 pixels acquired from 5 slides of cancer diagnosed and 3 healthy of oral brush samples, from distinct patients.
- **Paper:** https://dx.doi.org/10.2139/ssrn.4119212
- **Data:**  https://data.mendeley.com/datasets/dr7ydy9xbk/1
- **Motivation:** real data of pap stained oral cytology samples

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: I will add this directly on the hub (same as #4486)—in https://huggingface.co/lapix
Dataset Viewer issue for allocine
Author: lewtun
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/allocine

### Description

Not sure if this is a problem with `bz2` compression, but I thought these datasets could be streamed:

```
Status code:   400
Exception:     AttributeError
Message:       'TarContainedFile' object has no attribute 'readable'
```

### Owner

No

Comments: I removed my assignment as @huggingface/datasets should be able to answer better than me


Let me have a look...

Thanks for the quick fix @albertvillanova 

Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content *sequentially* (no random access).

> Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content _sequentially_ (no random access).

Ah thanks for the clarification! I'll look out for this next time and implement the fix myself :)
Dataset Viewer issue for conll2003
Author: lewtun
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/conll2003/viewer/conll2003/test

### Description

Seems like a cache problem with this config / split:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/conll2003/__init__.py'
```

### Owner

No

Comments: Fixed, thanks.
Dataset Viewer issue for xtreme
Author: lewtun
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/xtreme/viewer/PAN-X.de/test

### Description

There seems to be a problem with the cache of this config / split:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/xtreme/349258adc25bb45e47de193222f95e68a44f7a7ab53c4283b3f007208a11bf7e/xtreme.py'
```

### Owner

No

Comments: Fixed, thanks.
imdb source error
Author: Muhtasham
State: closed
Labels: name: bug

## Describe the bug
imdb dataset not loading

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("imdb")
```


## Expected results


## Actual results
```bash
06/23/2022 14:45:18 - INFO - datasets.builder - Dataset not on Hf google storage. Downloading and preparing it from source
06/23/2022 14:46:34 - INFO - datasets.utils.file_utils - HEAD request to http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz timed out, retrying... [1.0]
.....
ConnectionError: Couldn't reach http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz (ConnectTimeout(MaxRetryError("HTTPConnectionPool(host='ai.stanford.edu', port=80): Max retries exceeded with url: /~amaas/data/sentiment/aclImdb_v1.tar.gz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2d750cf690>, 'Connection to ai.stanford.edu timed out. (connect timeout=100)'))")))
```
## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments: Thanks for reporting, @Muhtasham.

Indeed IMDB dataset is not accessible from yesterday, because the data is hosted on the data owners servers at Stanford (http://ai.stanford.edu/) and these are down due to a power outage originated by a fire: https://twitter.com/StanfordAILab/status/1539472302399623170?s=20&t=1HU1hrtaXprtn14U61P55w

As a temporary workaroud, you can load the IMDB dataset with this tweak:
```python
ds = load_dataset("imdb", revision="tmp-fix-imdb")
```
FileNotFoundError when passing a data_file inside a directory starting with double underscores
Author: lhoestq
State: closed
Labels: name: bug

Bug experienced in the `accelerate` CI: https://github.com/huggingface/accelerate/runs/7016055148?check_suite_focus=true

This is related to https://github.com/huggingface/datasets/pull/4505 and the changes from https://github.com/huggingface/datasets/pull/4412

Comments: I have consistently experienced this bug on GitHub actions when bumping to `2.3.2`

We're working on a fix ;)
Metadata.jsonl for Imagefolder is ignored if it's in a parent directory to the splits directories/do not have "{split}_" prefix
Author: polinaeterna
State: closed

If data contains a single `metadata.jsonl` file for several splits, it won't be included in a dataset's `data_files` and therefore ignored. 
This happens when a directory is structured like as follows:
```
train/
  file_1.jpg
  file_2.jpg
test/
  file_3.jpg
  file_4.jpg
metadata.jsonl
```
or like as follows:
```
train_file_1.jpg
train_file_2.jpg
test_file_3.jpg
test_file_4.jpg
metadata.jsonl
```
The same for HF repos.

because it's ignored by the patterns [here](https://github.com/huggingface/datasets/blob/master/src/datasets/data_files.py#L29)

@lhoestq @mariosasko Do you think it's better to add this functionality in `data_files.py` or just specifically in imagefolder/audiofolder code? In `data_files.py` would me more general but I don't know if there are any other cases when that might be needed.
 

Comments: I agree it would be nice to support this. It doesn't fit really well in the current data_files.py, where files of each splits are separated in different folder though, maybe we have to modify a bit the logic here. 

One idea would be to extend `get_patterns_in_dataset_repository` and `get_patterns_locally` to additionally check for `metadata.json`, but feel free to comment if you have better ideas (I feel like we're reaching the limits of what the current implementation IMO, so we could think of a different way of resolving the data files if necessary)
Avoid splitting by` .py` for the file.
Author: espoirMur
State: closed
Labels: name: good first issue

https://github.com/huggingface/datasets/blob/90b3a98065556fc66380cafd780af9b1814b9426/src/datasets/load.py#L272


Hello, 
Thanks you for this library . 

 I was using it and I had one edge case. my home folder name ends with `.py` it is `/home/espoir.py` so anytime I am running the code to load a local module this code here it is failing because after splitting it is trying to save the code to my home directory.


Step to reproduce.

- If you have a home folder which ends with `.py`

- load a module with a local folder 
`qa_dataset = load_dataset("src/data/build_qa_dataset.py")`
it is failed 
A possible workaround would be to use pathlib at the mentioned line

` meta_path = Path(importable_local_file).parent.joinpath("metadata.json")` this can alivate the issue .

Let me what are your thought on this and I can try to fix it by A PR.


Comments: Hi @espoirMur, thanks for reporting.

You are right: that code line could be improved and made more generically valid.

On the other hand, I would suggest using `os.path.splitext` instead.

Are you willing to open a PR? :)

I will have a look.. 

This weekend .. 

@albertvillanova  , Can you have a look at #4590. 

Thanks 

#self-assign
Dataset Viewer issue for Pile of Law
Author: Breakend
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/pile-of-law/pile-of-law

### Description

Hi, I would like to turn off the dataset viewer for our dataset without enabling access requests. To comply with upstream dataset creator requests/licenses, we would like to make sure that the data is not indexed by search engines and so would like to turn off dataset previews. But we do not want to collect user emails because it would violate single blind review, allowing us to deduce potential reviewers' identities. Is there a way that we can turn off the dataset viewer without collecting identity information?

Thanks so much! 

### Owner

Yes

Comments: Hi @Breakend, yes – we'll propose a solution today

Thanks so much, I appreciate it!

Thanks so much for adding the docs. I was able to successfully hide the viewer using the 
```
viewer: false
```
flag in the README.md of the dataset. I'm closing the issue because this is resolved. Thanks again!

Awesome! Thanks for confirming. cc @severo 

Just for the record:

- the doc
  
<img width="1430" alt="Capture d’écran 2022-06-27 à 09 29 27" src="https://user-images.githubusercontent.com/1676121/175884089-bca6c0d5-6387-473e-98ca-86a910ede4bd.png">

- the dataset main page

<img width="1134" alt="Capture d’écran 2022-06-27 à 09 29 05" src="https://user-images.githubusercontent.com/1676121/175884152-5f285bf0-3471-45de-927a-e141b00ebb33.png">

- the dataset viewer page

<img width="567" alt="Capture d’écran 2022-06-27 à 09 29 16" src="https://user-images.githubusercontent.com/1676121/175884191-ab6a297b-1c11-417e-bbde-0b7623278a79.png">
Timestamp not returned as datetime objects in streaming mode
Author: lhoestq
State: closed
Labels: name: streaming

As reported in (internal) https://github.com/huggingface/datasets-server/issues/397

```python
>>> from datasets import load_dataset
>>> dataset = load_dataset("ett", name="h2", split="test", streaming=True)
>>> d = next(iter(dataset))
>>> d['start']
Timestamp('2016-07-01 00:00:00')
```

while loading in non-streaming mode it returns `datetime.datetime(2016, 7, 1, 0, 0)`
Dataset Viewer issue for CSV datasets
Author: merveenoyan
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/scikit-learn/breast-cancer-wisconsin

### Description

I'm populating CSV datasets [here](https://huggingface.co/scikit-learn) but the viewer is not enabled and it looks for a dataset loading script, the datasets aren't on queue as well. 

You can replicate the problem by simply uploading any CSV dataset.

### Owner

Yes

Comments: this should now be fixed

Confirmed, it's fixed now. Thanks for reporting, and thanks @coyotte508 for fixing it

<img width="1123" alt="Capture d’écran 2022-06-21 à 10 28 05" src="https://user-images.githubusercontent.com/1676121/174753833-1b453a5a-6a90-4717-bca1-1b5fc6b75e4a.png">
Ecoset
Author: DiGyt
State: closed
Labels: name: dataset request

## Adding a Dataset
- **Name:** *Ecoset*
- **Description:** *https://www.kietzmannlab.org/ecoset/*
- **Paper:** *https://doi.org/10.1073/pnas.2011417118*
- **Data:** *https://codeocean.com/capsule/9570390/tree/v1*
- **Motivation:**

**Ecoset** was created as a clean and ecologically valid alternative to **Imagenet**.

It is a large image recognition dataset, similar to Imagenet in size and structure. However, the authors of ecoset claim several improvements over Imagenet, like:
- more ecologically valid classes (e.g. not over-focussed on distinguishing different dog breeds)
- less NSFW content
- 'pre-packed image recognition models' that come with the dataset and can be used for validation of other models.


I am working for one of the authors of the paper with the aim of bringing Ecoset to huggingface datasets. Therefore I can work on this issue personally, but could use some help from devs and experienced users if the dataset is of interest to them. I phrased some of my questions on [discuss.huggingface](https://discuss.huggingface.co/t/handling-large-image-datasets/19373).


Comments: Hi! Very cool dataset! I answered your questions on the forum. Also, feel free to comment `#self-assign` on this issue to self-assign it.

The dataset lives on the Hub [here](https://huggingface.co/datasets/kietzmannlab/ecoset), so I'm closing this issue.

Hey There, thanks for closing 🤗 

Forgot the issue existed, so I didn't close it after implementing the downloader :)