Minimartzz/github-issues huggingface dataset:
Add Jigsaw unintended Bias
State: closed

Hi,

Here's a first attempt at this dataset. Would be great if it could be merged relatively quickly as it is needed for Bigscience-related stuff. 

This requires manual download, and I had some trouble generating dummy_data in this setting, so welcoming feedback there.

Comments:
Note that the tests seem to fail because of a bug in an Exception at the moment, see: https://github.com/huggingface/datasets/pull/2936 for the fix

@lhoestq implemented your changes, I think this might be ready for another look.

Thanks @lhoestq, implemented the changes, let me know if anything else pops up.
to_tf_dataset keeps a reference to the open data somewhere, causing issues on windows
State: closed
Labels: description: Something isn't working
name: bug

To reproduce:
```python
import datasets as ds
import weakref
import gc

d = ds.load_dataset("mnist", split="train")
ref = weakref.ref(d._data.table)
tfd = d.to_tf_dataset("image", batch_size=1, shuffle=False, label_cols="label")
del tfd, d
gc.collect()
assert ref() is None, "Error: there is at least one reference left"
```

This causes issues because the table holds a reference to an open arrow file that should be closed. So on windows it's not possible to delete or move the arrow file afterwards.

Moreover the CI test of the `to_tf_dataset` method isn't able to clean up the temporary arrow files because of this.

cc @Rocketknight1 

Comments:
I did some investigation and, as it seems, the bug stems from [this line](https://github.com/huggingface/datasets/blob/8004d7c3e1d74b29c3e5b0d1660331cd26758363/src/datasets/arrow_dataset.py#L325). The lifecycle of the dataset from the linked line is bound to one of the returned `tf.data.Dataset`. So my (hacky) solution involves wrapping the linked dataset with `weakref.proxy` and adding a custom `__del__` to `tf.python.data.ops.dataset_ops.TensorSliceDataset` (this is the type of a dataset that is returned by `tf.data.Dataset.from_tensor_slices`; this works for TF 2.x, but I'm not sure `tf.python.data.ops.dataset_ops` is a valid path for TF 1.x) that deletes the linked dataset, which is assigned to the dataset object as a property. Will open a draft PR soon!

Thanks a lot for investigating !
Replace script_version with revision
State: closed

As discussed in https://github.com/huggingface/datasets/pull/2718#discussion_r707013278, the parameter name `script_version` is no longer applicable to datasets without loading script (i.e., datasets only with raw data files).

This PR replaces the parameter name `script_version` with `revision`.

This way, we are also aligned with:
- Transformers: `AutoTokenizer.from_pretrained(..., revision=...)`
- Hub: `HfApi.dataset_info(..., revision=...)`, `HfApi.upload_file(..., revision=...)`

Comments:
I'm also fine with the removal in 1.15
Conda build fails
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
Current `datasets` version in conda is 1.9 instead of 1.12.

The build of the conda package fails.


Comments:
Why 1.9 ?

https://anaconda.org/HuggingFace/datasets currently says 1.11

Alright I added 1.12.0 and 1.12.1 and fixed the conda build #2952
Mutable columns argument breaks set_format
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
If you pass a mutable list to the `columns` argument of `set_format` and then change the list afterwards, the returned columns also change.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("glue", "cola")

column_list = ["idx", "label"]
dataset.set_format("python", columns=column_list)
column_list[1] = "foo"  # Change the list after we call `set_format`
dataset['train'][:4].keys()
```

## Expected results
```python
dict_keys(['idx', 'label'])
```

## Actual results
```python
dict_keys(['idx'])
```

Comments:
Pushed a fix to my branch #2731
Error when downloading datasets to non-traditional cache directories
State: open
Labels: description: Something isn't working
name: bug

## Describe the bug
When the cache directory is linked (soft link) to a directory on a NetApp device, the download fails. 

## Steps to reproduce the bug
```bash
ln -s /path/to/netapp/.cache ~/.cache
```

```python
load_dataset("imdb")
```

## Expected results
Successfully loading IMDB dataset

## Actual results
```
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=33432835, 
num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0,
 dataset_name='imdb')}, {'expected': SplitInfo(name='test', num_bytes=32650697, num_examples=25000, dataset_name='imdb'),
 'recorded': SplitInfo(name='test', num_bytes=659932, num_examples=503, dataset_name='imdb')}, {'expected':
 SplitInfo(name='unsupervised', num_bytes=67106814, num_examples=50000, dataset_name='imdb'), 'recorded':
 SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.1.2
- Platform: Ubuntu 
- Python version: 3.8

## Extra notes
Stranger yet, trying to debug the phenomenon, I found the range of results to vary a lot without clear direction:
 - With `cache_dir="/path/to/netapp/.cache"` the same thing happens.
 - However, when linking `~/netapp/` to `/path/to/netapp` *and* setting `cache_dir="~/netapp/.cache/huggingface/datasets"` - it does work
 - On the other hand, when linking `~/.cache` to `~/netapp/.cache` without using `cache_dir`, it does work anymore.

While I could test it only for a NetApp device, it might have to do with any other mounted FS.

Thanks :)



Comments:
Same here !
Add tutorial for no-code dataset upload
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

This PR is for a tutorial for uploading a dataset to the Hub. It relies on the Hub UI elements to upload a dataset, introduces the online tagging tool for creating tags, and the Dataset card template to get a head start on filling it out. The addition of this tutorial should make it easier for beginners to upload a dataset without accessing the terminal or knowing Git.

Comments:
Cool, love it ! :)

Feel free to add a paragraph saying how to load the dataset:
```python
from datasets import load_dataset

dataset = load_dataset("stevhliu/demo")

# or to separate each csv file into several splits
data_files = {"train": "train.csv", "test": "test.csv"}
dataset = load_dataset("stevhliu/demo", data_files=data_files)
print(dataset["train"][0])
```

Perfect, feel free to mark this PR ready for review :)

cc @albertvillanova do you have any comment ? You can check the tutorial here:
https://47389-250213286-gh.circle-artifacts.com/0/docs/_build/html/no_code_upload.html

Maybe we can just add a list of supported file types:
- csv
- json
- json lines
- text
- parquet

I just added a mention of the login for private datasets. Don't hesitate to edit or comment.

Otherwise I think it's all good, feel free to merge it @stevhliu  if you don't have other changes to make :)
Fix conversion of multidim arrays in list to arrow
State: closed

Arrow only supports 1-dim arrays. Previously we were converting all the numpy arrays to python list before instantiating arrow arrays to workaround this limitation.
However in #2361 we started to keep numpy arrays in order to keep their dtypes.
It works when we pass any multi-dim numpy array (the conversion to arrow has been added on our side), but not for lists of multi-dim numpy arrays.

In this PR I added two strategies:
- one that takes a list of multi-dim numpy arrays on returns an arrow array in an optimized way (more common case)
- one that takes a list of possibly very nested data (lists, dicts, tuples) containing multi-dim arrays. This one is less optimized since it converts all the multi-dim numpy arrays into lists of 1-d arrays for compatibility with arrow. This strategy is simpler that just trying to create the arrow array from a possibly very nested data structure, but in the future we can improve it if needed.

Fix https://github.com/huggingface/datasets/issues/2921

Comments:
Using a list of multi-dim numpy arrays raises an error "can only convert 1-dimensional array values"
State: closed

This error has been introduced in https://github.com/huggingface/datasets/pull/2361

To reproduce:
```python
import numpy as np
from datasets import Dataset

d = Dataset.from_dict({"a": [np.zeros((2, 2))]})
```
raises
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 5, in <module>
    d = Dataset.from_dict({"a": [np.zeros((2, 2))]}).with_format("torch")
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_dataset.py", line 458, in from_dict
    pa_table = InMemoryTable.from_pydict(mapping=mapping)
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 365, in from_pydict
    return cls(pa.Table.from_pydict(*args, **kwargs))
  File "pyarrow/table.pxi", line 1639, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 332, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 223, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_writer.py", line 107, in __arrow_array__
    out = pa.array(self.data, type=type)
  File "pyarrow/array.pxi", line 306, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values

Comments:
Unwanted progress bars when accessing examples
State: closed

When accessing examples from a dataset formatted for pytorch, some progress bars appear when accessing examples:
```python
In [1]: import datasets as ds                                        

In [2]: d = ds.Dataset.from_dict({"a": [0, 1, 2]}).with_format("torch")                                                           

In [3]: d[0]                                                         
100%|████████████████████████████████| 1/1 [00:00<00:00, 3172.70it/s]
Out[3]: {'a': tensor(0)}
```

This is because the pytorch formatter calls `map_nested` that uses progress bars

cc @sgugger 

Comments:
doing a patch release now :)
windows download abnormal
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
The script clearly exists (accessible from the browser), but the script download fails on windows. Then I tried it again and it can be downloaded normally on linux. why??
## Steps to reproduce the bug
```python3.7 + windows
![image](https://user-images.githubusercontent.com/52347799/133436174-4303f847-55d5-434f-a749-08da3bb9b654.png)


# Sample code to reproduce the bug
```

## Expected results
It can be downloaded normally.

## Actual results
it cann't

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.11.0
- Platform:windows
- Python version:3.7
- PyArrow version:


Comments:
Hi ! Is there some kind of proxy that is configured in your browser that gives you access to internet ? If it's the case it could explain why it doesn't work in the code, since the proxy wouldn't be used

It is indeed an agency problem, thank you very, very much

Let me know if you have other questions :)

Closing this issue now
Add OpenAI's pass@k code evaluation metric
State: closed

This PR introduces the `code_eval` metric which implements [OpenAI's code evaluation harness](https://github.com/openai/human-eval) introduced in the [Codex paper](https://arxiv.org/abs/2107.03374). It is heavily based on the original implementation and just adapts the interface to follow the `predictions`/`references` convention.

The addition of this metric should enable the evaluation against the code evaluation datasets added in #2897 and #2893.

A few open questions:

- The implementation makes heavy use of multiprocessing which this PR does not touch. Is this conflicting with multiprocessing natively integrated in `datasets`?
- This metric executes generated Python code and as such it poses dangers of executing malicious code. OpenAI addresses this issue by 1) commenting the `exec` call in the code so the user has to actively uncomment it and read the warning and 2) suggests using a sandbox environment (gVisor container). Should we add a similar safeguard? E.g. a prompt that needs to be answered when initialising the metric? Or at least a warning message?
- Naming: the implementation sticks to the `predictions`/`references` naming, however, the references are not reference solutions but unittest to test the solution. While reference solutions are also available they are not used. Should the naming be adapted?

Comments:
> The implementation makes heavy use of multiprocessing which this PR does not touch. Is this conflicting with multiprocessing natively integrated in datasets?

It should work normally, but feel free to test it.
There is some documentation about using metrics in a distributed setup that uses multiprocessing [here](https://huggingface.co/docs/datasets/loading.html?highlight=rank#distributed-setup)
You can test to spawn several processes where each process would load the metric. Then in each process you add some references/predictions to the metric. Finally you call compute() in each process and on process 0 it should return the result on all the references/predictions

Let me know if you have questions or if I can help

Is there a good way to debug the Windows tests? I suspect it is an issue with `multiprocessing`, but I can't see the error messages.

Indeed it has an issue on windows.
In your example it's supposed to output
```python
{'pass@1': 0.5, 'pass@2': 1.0}
```
but it gets
```python
{'pass@1': 0.0, 'pass@2': 0.0}
```

I'm not on my windows machine today so I can't take a look at it. I can dive into it early next week if you want

> I'm not on my windows machine today so I can't take a look at it. I can dive into it early next week if you want

That would be great - unfortunately I have no access to a windows machine at the moment. I am quite sure it is an issue with in exectue.py because of multiprocessing.
timit_asr dataset only includes one text phrase
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
The dataset 'timit_asr' only includes one text phrase. It only includes the transcription "Would such an act of refusal be useful?" multiple times rather than different phrases.

## Steps to reproduce the bug
Note: I am following the tutorial https://huggingface.co/blog/fine-tune-wav2vec2-english

1. Install the dataset and other packages
```python
!pip install datasets>=1.5.0
!pip install transformers==4.4.0
!pip install soundfile
!pip install jiwer
```
2. Load the dataset
```python
from datasets import load_dataset, load_metric

timit = load_dataset("timit_asr")
```
3. Remove columns that we don't want
```python
timit = timit.remove_columns(["phonetic_detail", "word_detail", "dialect_region", "id", "sentence_type", "speaker_id"])
```
4. Write a short function to display some random samples of the dataset.
```python
from datasets import ClassLabel
import random
import pandas as pd
from IPython.display import display, HTML

def show_random_elements(dataset, num_examples=10):
    assert num_examples <= len(dataset), "Can't pick more elements than there are in the dataset."
    picks = []
    for _ in range(num_examples):
        pick = random.randint(0, len(dataset)-1)
        while pick in picks:
            pick = random.randint(0, len(dataset)-1)
        picks.append(pick)
    
    df = pd.DataFrame(dataset[picks])
    display(HTML(df.to_html()))

show_random_elements(timit["train"].remove_columns(["file"]))
```

## Expected results
10 random different transcription phrases.

## Actual results
10 of the same transcription phrase "Would such an act of refusal be useful?"

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.4.1
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: not listed


Comments:
Hi @margotwagner, 
This bug was fixed in #1995. Upgrading the datasets should work (min v1.8.0 ideally)

Hi @margotwagner,

Yes, as @bhavitvyamalik has commented, this bug was fixed in `datasets` version 1.5.0. You need to update it, as your current version is 1.4.1:
> Environment info
> - `datasets` version: 1.4.1
feat: 🎸 add a function to get a dataset config's split names
State: closed

Also: pass additional arguments (use_auth_token) to get private configs + info of private datasets on the hub

Questions:

- [x] I'm not sure how the versions work: I changed 1.12.1.dev0 to 1.12.1.dev1, was it correct?
    -> no: reverted
- [x] Should I add a section in https://github.com/huggingface/datasets/blob/master/docs/source/load_hub.rst? (there is no section for get_dataset_infos)
    -> yes: added

Comments:
> Should I add a section in https://github.com/huggingface/datasets/blob/master/docs/source/load_hub.rst? (there is no section for get_dataset_infos)

Yes totally :) This tutorial should indeed mention this, given how fundamental it is
FORCE_REDOWNLOAD does not work
State: open
Labels: description: Something isn't working
name: bug

## Describe the bug
With GenerateMode.FORCE_REDOWNLOAD, the documentation says 
    +------------------------------------+-----------+---------+
    |                                    | Downloads | Dataset |
    +====================================+===========+=========+
    | `REUSE_DATASET_IF_EXISTS` (default)| Reuse     | Reuse   |
    +------------------------------------+-----------+---------+
    | `REUSE_CACHE_IF_EXISTS`            | Reuse     | Fresh   |
    +------------------------------------+-----------+---------+
    | `FORCE_REDOWNLOAD`                 | Fresh     | Fresh   |
    +------------------------------------+-----------+---------+

However, the old dataset is loaded even when FORCE_REDOWNLOAD is chosen.

## Steps to reproduce the bug
```python

import pandas as pd
from datasets import load_dataset, GenerateMode
pd.DataFrame(range(5), columns=['numbers']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)
pd.DataFrame(range(10), columns=['numerals']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)

```

## Expected results
Dataset({
    features: ['numbers'],
    num_rows: 5
})
Dataset({
    features: ['numerals'],
    num_rows: 10
})

## Actual results
Dataset({
    features: ['numbers'],
    num_rows: 5
})
Dataset({
    features: ['numbers'],
    num_rows: 5
})


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-4.14.181-108.257.amzn1.x86_64-x86_64-with-glibc2.10
- Python version: 3.7.10
- PyArrow version: 3.0.0


Comments:
Hi ! Thanks for reporting. The error seems to happen only if you use compressed files.

The second dataset is prepared in another dataset cache directory than the first - which is normal, since the source file is different. However, it doesn't uncompress the new data file because it finds the old uncompressed data in the extraction cache directory.

If we fix the extraction cache mechanism to uncompress a local file if it changed then it should fix the issue.
Currently the extraction cache mechanism only takes into account the path of the compressed file, which is an issue.

Facing the same issue, is there any way to overtake this issue until it will be fixed? 

You can clear your extraction cache in the meantime (by default at `~/.cache/huggingface/datasets/downloads/extracted`)
Add WIT Dataset
State: closed
Labels: description: Requesting to add a new dataset
name: dataset request

## Adding a Dataset
- **Name:** *WIT*
- **Description:** *Wikipedia-based Image Text Dataset*
- **Paper:** *[WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
](https://arxiv.org/abs/2103.01913)*
- **Data:** *https://github.com/google-research-datasets/wit*
- **Motivation:**  (excerpt from their Github README.md)

> - The largest multimodal dataset (publicly available at the time of this writing) by the number of image-text examples.
> - A massively multilingual dataset (first of its kind) with coverage for over 100+ languages.
> - A collection of diverse set of concepts and real world entities.
> - Brings forth challenging real-world test sets.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
@hassiahk is working on it  #2810 

WikiMedia is now hosting the pixel values directly which should make it a lot easier!
The files can be found here:
https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/
https://analytics.wikimedia.org/published/datasets/one-off/caption_competition/training/image_pixels/

> @hassiahk is working on it #2810

Thank you @bhavitvyamalik! Added this issue so we could track progress 😄 . Just linked the PR as well for visibility. 

Hey folks, we are now hosting the merged pixel values + embeddings + metadata ourselves. I gave it a try - [nateraw/wit](https://huggingface.co/datasets/nateraw/wit)

**⚠️ - Make sure you add `streaming=True` unless you're prepared to download 400GB of data!**

```python
from datasets import load_dataset

ds = load_dataset('nateraw/wit', streaming=True)
example = next(iter(ds))
```

```python
>>> example = next(iter(ds['train']))
>>> example.keys()
dict_keys(['b64_bytes', 'original_width', 'image_url', 'wit_features', 'original_height', 'metadata_url', 'mime_type', 'caption_attribution_description', 'embedding'])
>>> example['wit_features'].keys()
dict_keys(['hierarchical_section_title', 'language', 'attribution_passes_lang_id', 'context_section_description', 'is_main_image', 'page_title', 'caption_title_and_reference_description', 'caption_alt_text_description', 'caption_reference_description', 'page_url', 'context_page_description', 'section_title', 'page_changed_recently'])
```

Hi! `datasets` now hosts two versions of the WIT dataset:
* [`google/wit`](https://huggingface.co/datasets/google/wit): Google's version with the image URLs
* [`wikimedia/wit_base`](https://huggingface.co/datasets/wikimedia/wit_base): Wikimedia's version with the images + ResNet embeddings, but with less data than Google's version

Does this dataset work with `data_files` parameter. I tried and it doesn't. it attempts to download ~600 GB data even when i specify single file
Dataset
State: closed
Labels: description: Requesting to add a new dataset
name: dataset request

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments: