SurAyush/github_issues huggingface dataset:
Issue: Issues with concatenating datasets
State: closed
Author: ChenghaoMou

## Describe the bug
It is impossible to concatenate datasets if a feature is sequence of dict in one dataset and a dict of sequence in another. But based on the document, it should be automatically converted.

> A [datasets.Sequence](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Sequence) with a internal dictionary feature will be automatically converted into a dictionary of lists. This behavior is implemented to have a compatilbity layer with the TensorFlow Datasets library but may be un-wanted in some cases. If you don’t want this behavior, you can use a python list instead of the [datasets.Sequence](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Sequence).

## Steps to reproduce the bug
```python
from datasets import concatenate_datasets, load_dataset

squad = load_dataset("squad_v2")
squad["train"].to_json("output.jsonl", lines=True)

temp = load_dataset("json", data_files={"train": "output.jsonl"})
concatenate_datasets([temp["train"], squad["train"]])
```

## Expected results
No error executing that code

## Actual results
```
ValueError: The features can't be aligned because the key answers of features {'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answers': Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)} has unexpected type - Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None) (expected either {'text': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'answer_start': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)} or Value("null").
```

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.8.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments:
Hi! I agree we should improve the features equality checks to account for this particular case. However, your code fails due to `answer_start` having the dtype `int64` instead of `int32` after loading from JSON (it's not possible to embed type precision info into a JSON file; `save_to_disk` does that for arrow files), which would lead to the concatenation error as PyArrow does not support this sort of type promotion. This can be fixed as follows:
```python
temp = load_dataset("json", data_files={"train": "output.jsonl"}, features=squad["train"].features)
``` 

That makes sense. I totally missed the `int64` and `int32` part. Thanks for pointing it out! Will close this issue for now.
Issue: Unable to create dataset having Python dataset script only
State: closed
Author: aleSuglia

## Describe the bug
Hi there,

I'm trying to add the following dataset to Huggingface datasets: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/blob/

I'm trying to do so using the CLI commands but seems that this command generates the wrong `dataset_info.json` file (you can find it in the repo already):
```
datasets-cli test Heriot-WattUniversity/dialog-babi/dialog_babi.py --save_infos --all-configs
```
while it errors when I remove the python script:
```
datasets-cli test Heriot-WattUniversity/dialog-babi/ --save_infos --all-configs 
```
The error message is the following:
```
FileNotFoundError: Unable to resolve any data file that matches '['**']' at /Users/as2180/workspace/Heriot-WattUniversity/dialog-babi with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Comments:
Hi @aleSuglia, thanks for reporting.

We are having a look at it. 

We transfer this issue to the Community tab of the corresponding Hub dataset: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/discussions
Issue: Add stanford dog dataset
State: closed
Author: khushmeeet

This PR is for adding dataset, related to issue #4504.

We are adding Stanford dog breed dataset. It is a multi class image classification dataset. 
Details can be found here - http://vision.stanford.edu/aditya86/ImageNetDogs/

Tests on dummy data is failing currently, which I am looking into.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @khushmeeet, thanks for your contribution.

But wouldn't it be better to add this dataset to the Hub? 
- https://huggingface.co/docs/datasets/share
- https://huggingface.co/docs/datasets/dataset_script

Hi @albertvillanova 

Dataset is added to Hub - https://huggingface.co/datasets/dgrnd4/stanford_dog_dataset

Great, so I guess we can close this issue, as the dataset is already available on the Hub.

OK I read the discussion on:
- #4504

Currently, priority is adding datasets to the Hub, not here on GitHub.

If you would like to contribute the loading script and all the metadata you generated (README + JSON files), you could:
- Either make a PR to the existing dataset on the Hub
- Create a new dataset on the Hub:
  - Either under your personal namespace
  - or even more professionally, under the namespace `stanfordSVL` (Stanford Vision and Learning Lab: https://svl.stanford.edu/)

You can use the Community tab to ping us if you need help or have any questions.
Issue: Add text decorators
State: closed
Author: stevhliu

This PR adds some decoration to text about different modalities to make it more obvious separate guides exist for audio, vision, and text. The goal is to make it easier for users to discover these guides!

![underline](https://user-images.githubusercontent.com/59462357/178044392-9596693e-9a4a-479a-a282-f1edbd90be1a.png)

TODO:

- [x] Open PR to support new Tailwind classes

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Fix _resolve_single_pattern_locally on Windows with multiple drives
State: closed
Author: albertvillanova

Currently, when `_resolve_single_pattern_locally` is called from a different drive than the one in `pattern`, it raises an exception:
```
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\io\parquet.py:35: in __init__
    **kwargs,
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\builder.py:287: in __init__
    sanitize_patterns(data_files), base_path=base_path, use_auth_token=use_auth_token
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:761: in from_local_or_remote
    if not isinstance(patterns_for_key, DataFilesList)
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:723: in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:321: in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:239: in _resolve_single_pattern_locally
    for filepath in glob_iter
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:242: in <listcomp>
    os.path.relpath(filepath, base_path), os.path.relpath(pattern, base_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\popen-gw0\\data6\\dataset.parquet'
start = '/'

...

E                   ValueError: path is on mount 'C:', start on mount 'D:'
```

This PR makes sure that `base_path` is in the same drive as `pattern`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Good catch ! Sorry I forgot (again) about windows paths when writing this x)
Issue: Add SQuAD2.0 Dataset
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *SQuAD2.0*
- **Description:** *Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.*
- **Paper:** *https://aclanthology.org/P18-2124.pdf*
- **Data:** *https://rajpurkar.github.io/SQuAD-explorer/*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
Hey, It's already present [here](https://huggingface.co/datasets/squad_v2) 

Hi! This dataset is indeed already available on the Hub. Closing.
Issue: Add Amazon-QA Dataset
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *Amazon-QA*
- **Description:** *The dataset is .jsonl format, where each line in the file is a json string that corresponds to a question, existing answers to the question and the extracted review snippets (relevant to the question).*
- **Paper:** *https://github.com/amazonqa/amazonqa/tree/master/paper*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/amazon-qa.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/Amazon-QA).
Issue: Simple Wikipedia
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *Simple Wikipedia*
- **Description:** *Two different versions of the data set now exist. Both were generated by aligning Simple English Wikipedia and English Wikipedia. A complete description of the extraction process can be found in "Simple English Wikipedia: A New Simplification Task", William Coster and David Kauchak (2011).*
- **Paper:** *https://aclanthology.org/P11-2117/*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/SimpleWiki.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/simple-wiki).
Issue: Add Quora Question Triplets Dataset
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *Quora Question Triplets*
- **Description:** *This dataset consists of over 400,000 lines of potential question duplicate pairs. Each line contains IDs for each question in the pair, the full text for each question, and a binary value that indicates whether the line truly contains a duplicate pair.*
- **Paper:** 
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/quora_duplicates_triplets.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/QQP_triplets).
Issue: Add Altlex dataset
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *Altlex*
- **Description:** *Git repository for software associated with the 2016 ACL paper "Identifying Causal Relations Using Parallel Wikipedia Articles.”*
- **Paper:** *https://aclanthology.org/P16-1135.pdf*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/altlex.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*



Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/altlex).
Issue: Add Sentence Compression Dataset
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *Sentence Compression*
- **Description:** *Large corpus of uncompressed and compressed sentences from news articles.*
- **Paper:** *https://www.aclweb.org/anthology/D13-1155/*
- **Data:** *https://github.com/google-research-datasets/sentence-compression/tree/master/data*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/sentence-compression).
Issue: Add Flickr 30k Dataset
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *Flickr 30k*
- **Description:** *To produce the denotation graph, we have created an image caption corpus consisting of 158,915 crowd-sourced captions describing 31,783 images. This is an extension of our previous Flickr 8k Dataset. The new images and captions focus on people involved in everyday activities and events.*
- **Paper:** *https://transacl.org/ojs/index.php/tacl/article/view/229/33*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/flickr30k_captions.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/flickr30k-captions).
Issue: Add SPECTER dataset
State: open
Author: omarespejel

## Adding a Dataset
- **Name:** *SPECTER*
- **Description:** *SPECTER: Document-level Representation Learning using Citation-informed Transformers*
- **Paper:** *https://doi.org/10.18653/v1/2020.acl-main.207*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/specter_train_triples.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/SPECTER)
Issue: Add PAQ dataset
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *PAQ*
- **Description:** *This repository contains code and models to support the research paper PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them*
- **Paper:** *https://arxiv.org/abs/2102.07033*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/PAQ_pairs.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/PAQ_pairs)
Issue: Add WikiAnswers dataset
State: closed
Author: omarespejel

## Adding a Dataset
- **Name:** *WikiAnswers*
- **Description:** *The WikiAnswers corpus contains clusters of questions tagged by WikiAnswers users as paraphrases. Each cluster optionally contains an answer provided by WikiAnswers users.*
- **Paper:** *https://dl.acm.org/doi/10.1145/2623330.2623677*
- **Data:** *https://github.com/afader/oqa#wikianswers-corpus*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Comments:
uploaded dataset [here](https://huggingface.co/datasets/embedding-data/WikiAnswers)
Issue: Add Reddit dataset
State: open
Author: omarespejel

## Adding a Dataset
- **Name:** *Reddit comments (2015-2018)*
- **Description:** *Reddit is an American social news aggregation website, where users can post links, and take part in discussions on these posts. These threaded discussions provide a large corpus, which is converted into a conversational dataset using the tools in this directory.*
- **Paper:** *https://arxiv.org/abs/1904.06472*
- **Data:** *https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit*
- **Motivation:** *Dataset for training and evaluating models of conversational response*



Comments:
Issue: Set HF_SCRIPTS_VERSION to main
State: closed
Author: lhoestq

After renaming "master" to "main", the CI fails with
```
AssertionError: 'https://raw.githubusercontent.com/huggingface/datasets/main/datasets/_dummy/_dummy.py' not found in "Couldn't find a dataset script at /home/circleci/datasets/_dummy/_dummy.py or any data file in the same directory. Couldn't find '_dummy' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/_dummy/_dummy.py"
```

This is because in the CI we were still using `HF_SCRIPTS_VERSION=master`. I changed it to "main"

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Rename master to main
State: closed
Author: lhoestq

This PR renames mentions of "master" by "main" in the code base for several cases:
- set the default dataset script version to "main" if the local installation of `datasets` is a dev installation
- update URLs to this github repository to use "main"
- update the DVC benchmark
- update the github workflows
- update docstrings
- update tests to compare the changes in dataset cards against "main"



Comments:
_The documentation is not available anymore as the PR was closed or merged._

All the mentions I found on google were simple URLs that will be redirected, so it's fine. I also checked the spaces and we should be good:
- dalle-mini used to install the master branch but [it's no longer the case](https://huggingface.co/spaces/flax-community/dalle-mini/commit/b78c972afd5c2d2bed087be6479fe5c9c6cfa741)
- same for [logo generator](https://huggingface.co/spaces/tom-doerr/logo_generator/commit/a9ea330e518870d0ca8f65abb56f71d86750d8e4)
- I opened a PR to fix [vision-datasets-viewer](https://huggingface.co/spaces/nateraw/vision-datasets-viewer/discussions/1)


Ok let's rename the branch, and then we can merge this PR
Issue: Streaming issue for ccdv/pubmed-summarization
State: closed
Author: lewtun

### Link

https://huggingface.co/datasets/ccdv/pubmed-summarization

### Description

This was reported by a [user of AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/7). It seems like streaming doesn't work due to the way the dataset loading script is defined?

```
Status code:   400
Exception:     FileNotFoundError
Message:       https://huggingface.co/datasets/ccdv/pubmed-summarization/resolve/main/train.zip/train.txt
```

### Owner

No

Comments:
Thanks for reporting @lewtun.

I confirm there is an issue with streaming: it does not stream locally. 

Oh, after investigation, the source of the issue is in the Hub dataset loading script.

I'm opening a PR on the Hub dataset.

I've opened a PR on their Hub dataset to support streaming: https://huggingface.co/datasets/ccdv/pubmed-summarization/discussions/2
Issue: Dataset Viewer issue for kmfoda/booksum
State: closed
Author: lewtun

### Link

https://huggingface.co/datasets/kmfoda/booksum

### Description

A [user of AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/9) discovered this dataset cannot be streamed due to:

```
Status code:   400
Exception:     ClientResponseError
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/kmfoda/booksum/resolve/47953f583d6967f086cb16a2f4d2346e9834024d/test.csv')
```

I'm not sure why it says "Unauthorized" since it's just a bunch of CSV files in a repo 

### Owner

No

Comments:
Thanks for reporting, @lewtun.

It works locally in streaming mode:
```
{'bid': 27681,
 'is_aggregate': True,
 'source': 'cliffnotes',
 'chapter_path': 'all_chapterized_books/27681-chapters/chapters_1_to_2.txt',
 'summary_path': 'finished_summaries/cliffnotes/The Last of the Mohicans/section_1_part_0.txt',
 'book_id': 'The Last of the Mohicans.chapters 1-2',
 'summary_id': 'chapters 1-2',
 'content': None,
 'summary': '{"name": "Chapters 1-2", "url": "https://web.archive.org/web/20201101053205/https://www.cliffsnotes.com/literature/l/the-last-of-the-mohicans/summary-and-analysis/chapters-12", "summary": "Before any characters appear, the time and geography are made clear. Though it is the last war that England and France waged for a country that neither would retain, the wilderness between the forces still has to be...
```

I'm forcing the refresh of the preview. 

The preview appears as expected once the refresh forced.

Thank you @albertvillanova 🤗 !
Issue: Add HaGRID -- HAnd Gesture Recognition Image Dataset
State: open
Author: osanseviero

## Adding a Dataset
- **Name:** HaGRID -- HAnd Gesture Recognition Image Dataset
- **Description:** We introduce a large image dataset HaGRID (HAnd Gesture Recognition Image Dataset) for hand gesture recognition (HGR) systems. You can use it for image classification or image detection tasks. Proposed dataset allows to build HGR systems, which can be used in video conferencing services (Zoom, Skype, Discord, Jazz etc.), home automation systems, the automotive sector, etc.
- **Paper:** https://arxiv.org/abs/2206.08219
- **Data:** https://github.com/hukenovs/hagrid

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: