helmo/github-issues huggingface dataset:
Protect master branch
State: closed | Author: albertvillanova (MEMBER)
Labels: name: enhancement

After accidental merge commit (91c55355b634d0dc73350a7ddee1a6776dbbdd69) into `datasets` master branch, all commits present in the feature branch were permanently added to `datasets` master branch history, as e.g.:
- 00cc036fea7c7745cfe722360036ed306796a3f2
- 13ae8c98602bbad8197de3b9b425f4c78f582af1
- ...

I propose to protect our master branch, so that we avoid we can accidentally make this kind of mistakes in the future:
- [x] For Pull Requests using GitHub, allow only squash merging, so that only a single commit per Pull Request is merged into the master branch
  - Currently, simple merge commits are already disabled
  - I propose to disable rebase merging as well
- ~~Protect the master branch from direct pushes (to avoid accidentally pushing of merge commits)~~
  - ~~This protection would reject direct pushes to master branch~~
  - ~~If so, for each release (when we need to commit directly to the master branch), we should previously disable the protection and re-enable it again after the release~~
- [x] Protect the master branch only from direct pushing of **merge commits**
  - GitHub offers the possibility to protect the master branch only from merge commits (which are the ones that introduce all the commits from the feature branch into the master branch).
  - No need to disable/re-enable this protection on each release 

This purpose of this Issue is to open a discussion about this problem and to agree in a solution.

Comments: Cool, I think we can do both :)

@lhoestq now the 2 are implemented.

Please note that for the the second protection, finally I have chosen to protect the master branch only from **merge commits** (see update comment above), so no need to disable/re-enable the protection on each release (direct commits, different from merge commits, can be pushed to the remote master branch; and eventually reverted without messing up the repo history).
Add  `remove_columns` to `IterableDataset ` 
State: closed | Author: changjonathanc (CONTRIBUTOR)
Labels: name: enhancement

name: good first issue

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

```python
from datasets import load_dataset
dataset = load_dataset("c4", 'realnewslike', streaming =True, split='train')
dataset = dataset.remove_columns('url')
```
```
AttributeError: 'IterableDataset' object has no attribute 'remove_columns'
```

**Describe the solution you'd like**

It would be nice to have `.remove_columns()` to match the `Datasets` api. 


**Describe alternatives you've considered**

This can be done with a single call to `.map()`, 

I can try to help add this. 🤗

Comments: Hi ! Good idea :)
If you are interested in contributing, feel free to give it a try and open a Pull Request. Also let me know if I can help you with this or if you have questions
Add SEDE dataset
State: closed | Author: Hazoom (NONE)

This PR adds the SEDE dataset for the task of realistic Text-to-SQL, following the instructions of how to add a database and a dataset card.

Please see our paper for more details: https://arxiv.org/abs/2106.05006

Comments: Thanks @albertvillanova for your great suggestions! I just pushed a new commit with the necessary fixes. For some reason, the test `test_metric_common` failed for `meteor` metric, which doesn't have any connection to this PR, so I'm trying to rebase and see if it helps.

Hi @Hazoom,

You were right: the non-passing test had nothing to do with this PR.

Unfortunately, you did a git rebase (instead of a git merge), which is not recommended once you have already opened a pull request because you mess up your pull request history. You can see that your pull request now contains:
- your commits repeated two times
- and commits which are not yours from the master branch

If you would like to clean your pull request, please make:
```
git reset --hard 587b93a
git fetch upstream master
git merge upstream/master
git push --force origin sede
```

> Hi @Hazoom,
> 
> You were right: the non-passing test had nothing to do with this PR.
> 
> Unfortunately, you did a git rebase (instead of a git merge), which is not recommended once you have already opened a pull request because you mess up your pull request history. You can see that your pull request now contains:
> 
> * your commits repeated two times
> * and commits which are not yours from the master branch
> 
> If you would like to clean your pull request, please make:
> 
> ```
> git reset --hard 587b93a
> git fetch upstream master
> git merge upstream/master
> git push --force origin sede
> ```

Thanks @albertvillanova 

> Nice! Just one final request before approving your pull request:
> 
> As you have updated the "QuerySetId" field data type, the size of the dataset is smaller now. You should regenerate the metadata. Please run:
> 
> ```
> rm datasets/sede/dataset_infos.json
> datasets-cli test datasets/sede --save_infos --all_configs
> ```

@albertvillanova Good catch, just fixed it.
OSCAR unshuffled_original_ko: NonMatchingSplitsSizesError
State: open | Author: ayaka14732 (NONE)
Labels: name: bug

name: dataset bug

## Describe the bug

Cannot download OSCAR `unshuffled_original_ko` due to `NonMatchingSplitsSizesError`.

## Steps to reproduce the bug

```python
>>> dataset = datasets.load_dataset('oscar', 'unshuffled_original_ko')
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=25292102197, num_examples=7345075, dataset_name='oscar'), 'recorded': SplitInfo(name='train', num_bytes=25284578514, num_examples=7344907, dataset_name='oscar')}]
```

## Expected results

Loading is successful.

## Actual results

Loading throws above error.

## Environment info

- `datasets` version: 1.12.1
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0


Comments: I tried `unshuffled_original_da` and it is also not working
Take namespace into account in caching
State: closed | Author: lhoestq (MEMBER)

Loading a dataset "username/dataset_name" hosted by a user on the hub used to cache the dataset only taking into account the dataset name, and ignorign the username. Because of this, if a user later loads "dataset_name" without specifying the username, it would reload the dataset from the cache instead of failing.

I changed the dataset cache and module cache mechanism to include the username in the name of the cache directory that is used:


`~/.cache/huggingface/datasets/username/dataset_name`  for the data
`~/.cache/huggingface/modules/datasets_modules/datasets/username/dataset_name` for the python files


EDIT: actually using three underscores:
`~/.cache/huggingface/datasets/username___dataset_name` for the data
`~/.cache/huggingface/modules/datasets_modules/datasets/username___dataset_name` for the python files

This PR should fix the issue https://github.com/huggingface/datasets/issues/2842

cc @stas00 

Comments: We might have collisions if a username and a dataset_name are the same. Maybe instead serialize the dataset name by replacing `/` with some string, eg `__SLASH__`, that will hopefully never appear in a dataset or user name (it's what I did in https://github.com/huggingface/datasets-preview-backend/blob/master/benchmark/scripts/serialize.py. That way, all the datasets are one-level deep directories

IIRC we enforce that no repo id or username can contain `___` (exactly 3 underscores) specifically for this reason, so you can use that string (that we use in other projects)

cc @Pierrci 

> IIRC we enforce that no repo id or username can contain ___ (exactly 3 underscores) specifically for this reason, so you can use that string (that we use in other projects)

out of curiosity: where is it enforced?

> where is it enforced?

Nowhere yet but we should :) feel free to track in internal tracker and/or implement, as this will be useful in the future

Thanks for the trick, I'm doing the change :)
We can use
`~/.cache/huggingface/datasets/username___dataset_name` for the data
`~/.cache/huggingface/modules/datasets_modules/datasets/username___dataset_name` for the python files

Merging, though it will have to be integrated again the refactor at #2986

@lhoestq we changed a bit the naming policy on the Hub, and the substring '--' is now forbidden, which makes it available for serializing the repo names (`namespace/repo` -> `namespace--repo`). See https://github.com/huggingface/moon-landing/pull/1657 and https://github.com/huggingface/huggingface_hub/pull/545
Add Jigsaw unintended Bias
State: closed | Author: Iwontbecreative (CONTRIBUTOR)

Hi,

Here's a first attempt at this dataset. Would be great if it could be merged relatively quickly as it is needed for Bigscience-related stuff. 

This requires manual download, and I had some trouble generating dummy_data in this setting, so welcoming feedback there.

Comments: Note that the tests seem to fail because of a bug in an Exception at the moment, see: https://github.com/huggingface/datasets/pull/2936 for the fix

@lhoestq implemented your changes, I think this might be ready for another look.

Thanks @lhoestq, implemented the changes, let me know if anything else pops up.
to_tf_dataset keeps a reference to the open data somewhere, causing issues on windows
State: closed | Author: lhoestq (MEMBER)
Labels: name: bug

To reproduce:
```python
import datasets as ds
import weakref
import gc

d = ds.load_dataset("mnist", split="train")
ref = weakref.ref(d._data.table)
tfd = d.to_tf_dataset("image", batch_size=1, shuffle=False, label_cols="label")
del tfd, d
gc.collect()
assert ref() is None, "Error: there is at least one reference left"
```

This causes issues because the table holds a reference to an open arrow file that should be closed. So on windows it's not possible to delete or move the arrow file afterwards.

Moreover the CI test of the `to_tf_dataset` method isn't able to clean up the temporary arrow files because of this.

cc @Rocketknight1 

Comments: I did some investigation and, as it seems, the bug stems from [this line](https://github.com/huggingface/datasets/blob/8004d7c3e1d74b29c3e5b0d1660331cd26758363/src/datasets/arrow_dataset.py#L325). The lifecycle of the dataset from the linked line is bound to one of the returned `tf.data.Dataset`. So my (hacky) solution involves wrapping the linked dataset with `weakref.proxy` and adding a custom `__del__` to `tf.python.data.ops.dataset_ops.TensorSliceDataset` (this is the type of a dataset that is returned by `tf.data.Dataset.from_tensor_slices`; this works for TF 2.x, but I'm not sure `tf.python.data.ops.dataset_ops` is a valid path for TF 1.x) that deletes the linked dataset, which is assigned to the dataset object as a property. Will open a draft PR soon!

Thanks a lot for investigating !
Replace script_version with revision
State: closed | Author: albertvillanova (MEMBER)

As discussed in https://github.com/huggingface/datasets/pull/2718#discussion_r707013278, the parameter name `script_version` is no longer applicable to datasets without loading script (i.e., datasets only with raw data files).

This PR replaces the parameter name `script_version` with `revision`.

This way, we are also aligned with:
- Transformers: `AutoTokenizer.from_pretrained(..., revision=...)`
- Hub: `HfApi.dataset_info(..., revision=...)`, `HfApi.upload_file(..., revision=...)`

Comments: I'm also fine with the removal in 1.15
Conda build fails
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

## Describe the bug
Current `datasets` version in conda is 1.9 instead of 1.12.

The build of the conda package fails.


Comments: Why 1.9 ?

https://anaconda.org/HuggingFace/datasets currently says 1.11

Alright I added 1.12.0 and 1.12.1 and fixed the conda build #2952
Mutable columns argument breaks set_format
State: closed | Author: Rocketknight1 (MEMBER)
Labels: name: bug

## Describe the bug
If you pass a mutable list to the `columns` argument of `set_format` and then change the list afterwards, the returned columns also change.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("glue", "cola")

column_list = ["idx", "label"]
dataset.set_format("python", columns=column_list)
column_list[1] = "foo"  # Change the list after we call `set_format`
dataset['train'][:4].keys()
```

## Expected results
```python
dict_keys(['idx', 'label'])
```

## Actual results
```python
dict_keys(['idx'])
```

Comments: Pushed a fix to my branch #2731
Error when downloading datasets to non-traditional cache directories
State: open | Author: dar-tau (NONE)
Labels: name: bug

## Describe the bug
When the cache directory is linked (soft link) to a directory on a NetApp device, the download fails. 

## Steps to reproduce the bug
```bash
ln -s /path/to/netapp/.cache ~/.cache
```

```python
load_dataset("imdb")
```

## Expected results
Successfully loading IMDB dataset

## Actual results
```
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=33432835, 
num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0,
 dataset_name='imdb')}, {'expected': SplitInfo(name='test', num_bytes=32650697, num_examples=25000, dataset_name='imdb'),
 'recorded': SplitInfo(name='test', num_bytes=659932, num_examples=503, dataset_name='imdb')}, {'expected':
 SplitInfo(name='unsupervised', num_bytes=67106814, num_examples=50000, dataset_name='imdb'), 'recorded':
 SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.1.2
- Platform: Ubuntu 
- Python version: 3.8

## Extra notes
Stranger yet, trying to debug the phenomenon, I found the range of results to vary a lot without clear direction:
 - With `cache_dir="/path/to/netapp/.cache"` the same thing happens.
 - However, when linking `~/netapp/` to `/path/to/netapp` *and* setting `cache_dir="~/netapp/.cache/huggingface/datasets"` - it does work
 - On the other hand, when linking `~/.cache` to `~/netapp/.cache` without using `cache_dir`, it does work anymore.

While I could test it only for a NetApp device, it might have to do with any other mounted FS.

Thanks :)



Comments: Same here !
Add tutorial for no-code dataset upload
State: closed | Author: stevhliu (MEMBER)
Labels: name: documentation

This PR is for a tutorial for uploading a dataset to the Hub. It relies on the Hub UI elements to upload a dataset, introduces the online tagging tool for creating tags, and the Dataset card template to get a head start on filling it out. The addition of this tutorial should make it easier for beginners to upload a dataset without accessing the terminal or knowing Git.

Comments: Cool, love it ! :)

Feel free to add a paragraph saying how to load the dataset:
```python
from datasets import load_dataset

dataset = load_dataset("stevhliu/demo")

# or to separate each csv file into several splits
data_files = {"train": "train.csv", "test": "test.csv"}
dataset = load_dataset("stevhliu/demo", data_files=data_files)
print(dataset["train"][0])
```

Perfect, feel free to mark this PR ready for review :)

cc @albertvillanova do you have any comment ? You can check the tutorial here:
https://47389-250213286-gh.circle-artifacts.com/0/docs/_build/html/no_code_upload.html

Maybe we can just add a list of supported file types:
- csv
- json
- json lines
- text
- parquet

I just added a mention of the login for private datasets. Don't hesitate to edit or comment.

Otherwise I think it's all good, feel free to merge it @stevhliu  if you don't have other changes to make :)
Fix conversion of multidim arrays in list to arrow
State: closed | Author: lhoestq (MEMBER)

Arrow only supports 1-dim arrays. Previously we were converting all the numpy arrays to python list before instantiating arrow arrays to workaround this limitation.
However in #2361 we started to keep numpy arrays in order to keep their dtypes.
It works when we pass any multi-dim numpy array (the conversion to arrow has been added on our side), but not for lists of multi-dim numpy arrays.

In this PR I added two strategies:
- one that takes a list of multi-dim numpy arrays on returns an arrow array in an optimized way (more common case)
- one that takes a list of possibly very nested data (lists, dicts, tuples) containing multi-dim arrays. This one is less optimized since it converts all the multi-dim numpy arrays into lists of 1-d arrays for compatibility with arrow. This strategy is simpler that just trying to create the arrow array from a possibly very nested data structure, but in the future we can improve it if needed.

Fix https://github.com/huggingface/datasets/issues/2921
Using a list of multi-dim numpy arrays raises an error "can only convert 1-dimensional array values"
State: closed | Author: lhoestq (MEMBER)

This error has been introduced in https://github.com/huggingface/datasets/pull/2361

To reproduce:
```python
import numpy as np
from datasets import Dataset

d = Dataset.from_dict({"a": [np.zeros((2, 2))]})
```
raises
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 5, in <module>
    d = Dataset.from_dict({"a": [np.zeros((2, 2))]}).with_format("torch")
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_dataset.py", line 458, in from_dict
    pa_table = InMemoryTable.from_pydict(mapping=mapping)
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 365, in from_pydict
    return cls(pa.Table.from_pydict(*args, **kwargs))
  File "pyarrow/table.pxi", line 1639, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 332, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 223, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_writer.py", line 107, in __arrow_array__
    out = pa.array(self.data, type=type)
  File "pyarrow/array.pxi", line 306, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
Unwanted progress bars when accessing examples
State: closed | Author: lhoestq (MEMBER)

When accessing examples from a dataset formatted for pytorch, some progress bars appear when accessing examples:
```python
In [1]: import datasets as ds                                        

In [2]: d = ds.Dataset.from_dict({"a": [0, 1, 2]}).with_format("torch")                                                           

In [3]: d[0]                                                         
100%|████████████████████████████████| 1/1 [00:00<00:00, 3172.70it/s]
Out[3]: {'a': tensor(0)}
```

This is because the pytorch formatter calls `map_nested` that uses progress bars

cc @sgugger 

Comments: doing a patch release now :)
windows download abnormal
State: closed | Author: wei1826676931 (NONE)
Labels: name: bug

## Describe the bug
The script clearly exists (accessible from the browser), but the script download fails on windows. Then I tried it again and it can be downloaded normally on linux. why??
## Steps to reproduce the bug
```python3.7 + windows
![image](https://user-images.githubusercontent.com/52347799/133436174-4303f847-55d5-434f-a749-08da3bb9b654.png)


# Sample code to reproduce the bug
```

## Expected results
It can be downloaded normally.

## Actual results
it cann't

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.11.0
- Platform:windows
- Python version:3.7
- PyArrow version:


Comments: Hi ! Is there some kind of proxy that is configured in your browser that gives you access to internet ? If it's the case it could explain why it doesn't work in the code, since the proxy wouldn't be used

It is indeed an agency problem, thank you very, very much

Let me know if you have other questions :)

Closing this issue now
Add OpenAI's pass@k code evaluation metric
State: closed | Author: lvwerra (MEMBER)

This PR introduces the `code_eval` metric which implements [OpenAI's code evaluation harness](https://github.com/openai/human-eval) introduced in the [Codex paper](https://arxiv.org/abs/2107.03374). It is heavily based on the original implementation and just adapts the interface to follow the `predictions`/`references` convention.

The addition of this metric should enable the evaluation against the code evaluation datasets added in #2897 and #2893.

A few open questions:

- The implementation makes heavy use of multiprocessing which this PR does not touch. Is this conflicting with multiprocessing natively integrated in `datasets`?
- This metric executes generated Python code and as such it poses dangers of executing malicious code. OpenAI addresses this issue by 1) commenting the `exec` call in the code so the user has to actively uncomment it and read the warning and 2) suggests using a sandbox environment (gVisor container). Should we add a similar safeguard? E.g. a prompt that needs to be answered when initialising the metric? Or at least a warning message?
- Naming: the implementation sticks to the `predictions`/`references` naming, however, the references are not reference solutions but unittest to test the solution. While reference solutions are also available they are not used. Should the naming be adapted?

Comments: > The implementation makes heavy use of multiprocessing which this PR does not touch. Is this conflicting with multiprocessing natively integrated in datasets?

It should work normally, but feel free to test it.
There is some documentation about using metrics in a distributed setup that uses multiprocessing [here](https://huggingface.co/docs/datasets/loading.html?highlight=rank#distributed-setup)
You can test to spawn several processes where each process would load the metric. Then in each process you add some references/predictions to the metric. Finally you call compute() in each process and on process 0 it should return the result on all the references/predictions

Let me know if you have questions or if I can help

Is there a good way to debug the Windows tests? I suspect it is an issue with `multiprocessing`, but I can't see the error messages.

Indeed it has an issue on windows.
In your example it's supposed to output
```python
{'pass@1': 0.5, 'pass@2': 1.0}
```
but it gets
```python
{'pass@1': 0.0, 'pass@2': 0.0}
```

I'm not on my windows machine today so I can't take a look at it. I can dive into it early next week if you want

> I'm not on my windows machine today so I can't take a look at it. I can dive into it early next week if you want

That would be great - unfortunately I have no access to a windows machine at the moment. I am quite sure it is an issue with in exectue.py because of multiprocessing.