josteece/github-issues huggingface dataset:
Cache problem in the `load_dataset` method for local compressed file(s)
Opened by SaulLu (open)

## Describe the bug

Cache problem in the `load_dataset` method: when modifying a compressed file in a local folder `load_dataset` doesn't detect the change and load the previous version.

## Steps to reproduce the bug

To test it directly, I have prepared a [Google Colaboratory notebook](https://colab.research.google.com/drive/11Em_Amoc-aPGhSBIkSHU2AvEh24nVayy?usp=sharing) that shows this behavior. 

For this example, I have created a toy dataset at: https://huggingface.co/datasets/SaulLu/toy_struc_dataset

This dataset is composed of two versions: 
- v1 on commit `a6beb46` which has a single example `{'id': 1, 'value': {'tag': 'a', 'value': 1}}` in file `train.jsonl.gz`
- v2 on commit `e7935f4` (`main` head) which has a single example `{'attr': 1, 'id': 1, 'value': 'a'}` in file `train.jsonl.gz`

With a terminal, we can start to get the v1 version of the dataset
```bash
git lfs install
git clone https://huggingface.co/datasets/SaulLu/toy_struc_dataset
cd toy_struc_dataset
git checkout a6beb46
```
Then we can load it with python and look at the content:
```python
from datasets import load_dataset

path = "/content/toy_struc_dataset"
dataset = load_dataset(path, data_files={"train": "*.jsonl.gz"})
print(dataset["train"][0])
```
Output
```
{'id': 1, 'value': {'tag': 'a', 'value': 1}} # This is the example in v1
```

With a terminal, we can now start to get the v1 version of the dataset
```bash
git checkout main
```
Then we can load it with python and look at the content:
```python
from datasets import load_dataset

path = "/content/toy_struc_dataset"
dataset = load_dataset(path, data_files={"train": "*.jsonl.gz"})
print(dataset["train"][0])
```
Output
```
{'id': 1, 'value': {'tag': 'a', 'value': 1}} # This is the example in v1 (not v2)
```
## Expected results
The last output should have been 
```
{"id":1, "value": "a", "attr": 1} # This is the example in v2
```

## Ideas

As discussed offline with Quentin, if the cache hash was ever sensitive to changes in a compressed file we would probably not have the problem anymore.

This situation leads me to suggest 2 other features:
- to also have an `load_from_cache_file` argument in the "load_dataset" method
- to reorganize the cache so that we can delete the caches related to a dataset (cf issue #ToBeFilledSoon)

And thanks again for this great library :hugs: 

## Environment info
- `datasets` version: 1.12.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0


Comments:
The problem is still present. 
One solution would be to add the `download_mode="force_redownload"` argument to load_dataset. 
However, doing so may lead to a `DatasetGenerationError: An error occurred while generating the dataset`. To mitigate, just do:
`rm -r ~/.cache/huggingface/datasets/*`
Update legacy Python image for CI tests in Linux
Opened by albertvillanova (closed)

Instead of legacy, use next-generation convenience images, built from the ground up with CI, efficiency, and determinism in mind. Here are some of the highlights:

- Faster spin-up time - In Docker terminology, these next-gen images will generally have fewer and smaller layers. Using these new images will lead to faster image downloads when a build starts, and a higher likelihood that the image is already cached on the host.

- Improved reliability and stability - The existing legacy convenience images are rebuilt practically every day with potential changes from upstream that we cannot always test fast enough. This leads to frequent breaking changes, which is not the best environment for stable, deterministic builds. Next-gen images will only be rebuilt for security and critical-bugs, leading to more stable and deterministic images.

More info: https://circleci.com/docs/2.0/circleci-images

Comments:
There is an exception when running `pip install .[tests]`:
```
Processing /home/circleci/datasets
Collecting numpy>=1.17 (from datasets==1.12.2.dev0)
  Downloading https://files.pythonhosted.org/packages/45/b2/6c7545bb7a38754d63048c7696804a0d947328125d81bf12beaa692c3ae3/numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl (13.4MB)
    100% |████████████████████████████████| 13.4MB 3.9MB/s  eta 0:00:011

...

Collecting faiss-cpu (from datasets==1.12.2.dev0)
  Downloading https://files.pythonhosted.org/packages/87/91/bf8ea0d42733cbb04f98d3bf27808e4919ceb5ec71102e21119398a97237/faiss-cpu-1.7.1.post2.tar.gz (41kB)
    100% |████████████████████████████████| 51kB 30.9MB/s ta 0:00:01
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/home/circleci/.pyenv/versions/3.6.14/lib/python3.6/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/home/circleci/.pyenv/versions/3.6.14/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/home/circleci/.pyenv/versions/3.6.14/lib/python3.6/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/home/circleci/.pyenv/versions/3.6.14/lib/python3.6/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-1pop4blm/numpy-1.21.2/setup.py", line 34, in <module>
        method can be invoked.
    RuntimeError: Python version >= 3.7 required.
```

Apparently, `numpy-1.21.2` tries to be installed in the temporary directory `/tmp/easy_install-1pop4blm` instead of the downloaded `numpy-1.19.5` (requirement of `datasets`).

This is caused because `pip` downloads the `.tar.gz` (instead of the `.whl`) and tries to build it in a tmp dir.
Run tests in parallel
Opened by albertvillanova (closed)

Run CI tests in parallel to speed up the test suite.

Speed up results:
- Linux: from `7m 30s` to `5m 32s`
- Windows: from `13m 52s` to `11m 10s`


Comments:
There is a speed up in Windows machines:
- From `13m 52s` to `11m 10s`

In Linux machines, some workers crash with error message:
```
OSError: [Errno 12] Cannot allocate memory
```

There is also a speed up in Linux machines:
- From `7m 30s` to `5m 32s`
Trying to get in touch regarding a security issue
Opened by JamieSlome (closed)

Hey there!

I'd like to report a security issue but cannot find contact instructions on your repository.

If not a hassle, might you kindly add a `SECURITY.md` file with an email, or another contact method? GitHub [recommends](https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository) this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

Comments:
Hi @JamieSlome,

Thanks for reaching out. Yes, you are right: I'm opening a PR to add the `SECURITY.md` file and a contact method.

In the meantime, please feel free to report the security issue to: feedback@huggingface.co
Dummy labels no longer on by default in `to_tf_dataset`
Opened by Rocketknight1 (closed)

After more experimentation, I think I have a way to do things that doesn't depend on adding `dummy_labels` - they were quite a hacky solution anyway!

Comments:
@lhoestq Let me make sure we never need it, and if not then I'll remove it entirely in a follow-up PR.

Thanks ;) it will be less confusing and easier to maintain to not keep unused hacky features
Introduce web and wiki config in triviaqa dataset
Opened by shirte (closed)

The TriviaQA paper suggests that the two subsets (Wikipedia and Web)
should be treated differently. There are also different leaderboards
for the two sets on CodaLab. For that reason, introduce additional
builder configs in the trivia_qa dataset.

Comments:
I just made the dummy data smaller :)
Once github refreshes the change I think we can merge !

Thank you so much for reviewing and accepting my pull request!! :)

I created these rather large dummy data sets to cover all different cases for the row structure. E.g. in the web configuration, it's possible that a row has evidence from wikipedia ("EntityPages") and the web ("SearchResults"). But it also might happen that either EntityPages or SearchResults is empty. Probably, I will add this thought to the dataset description in the future.

Ok I see ! Yes feel free to mention it in the dataset card, this can be useful.

For the dummy data though we can keep the small ones, as the tests are mainly about testing the parsing from the dataset script rather than the actual content of the dataset.
Don't use old, incompatible cache for the new `filter`
Opened by lhoestq (closed)

#2836 changed `Dataset.filter` and the resulting data that are stored in the cache are different and incompatible with the ones of the previous `filter` implementation.

However the caching mechanism wasn't able to differentiate between the old and the new implementation of filter (only the method name was taken into account). 

This is an issue because anyone that update `datasets` and re-runs some code that uses `filter` would see an error, because the cache would try to load an incompatible `filter` result.

To fix this I added the notion of versioning for dataset transform in the caching mechanism, and bumped the version of the `filter` implementation to 2.0.0

This way the new `filter` outputs are now considered different from the old ones from the caching point of view.

This should fix #2943

cc @anton-l

Comments:
Protect master branch
Opened by albertvillanova (closed)

After accidental merge commit (91c55355b634d0dc73350a7ddee1a6776dbbdd69) into `datasets` master branch, all commits present in the feature branch were permanently added to `datasets` master branch history, as e.g.:
- 00cc036fea7c7745cfe722360036ed306796a3f2
- 13ae8c98602bbad8197de3b9b425f4c78f582af1
- ...

I propose to protect our master branch, so that we avoid we can accidentally make this kind of mistakes in the future:
- [x] For Pull Requests using GitHub, allow only squash merging, so that only a single commit per Pull Request is merged into the master branch
  - Currently, simple merge commits are already disabled
  - I propose to disable rebase merging as well
- ~~Protect the master branch from direct pushes (to avoid accidentally pushing of merge commits)~~
  - ~~This protection would reject direct pushes to master branch~~
  - ~~If so, for each release (when we need to commit directly to the master branch), we should previously disable the protection and re-enable it again after the release~~
- [x] Protect the master branch only from direct pushing of **merge commits**
  - GitHub offers the possibility to protect the master branch only from merge commits (which are the ones that introduce all the commits from the feature branch into the master branch).
  - No need to disable/re-enable this protection on each release 

This purpose of this Issue is to open a discussion about this problem and to agree in a solution.

Comments:
Cool, I think we can do both :)

@lhoestq now the 2 are implemented.

Please note that for the the second protection, finally I have chosen to protect the master branch only from **merge commits** (see update comment above), so no need to disable/re-enable the protection on each release (direct commits, different from merge commits, can be pushed to the remote master branch; and eventually reverted without messing up the repo history).
Add  `remove_columns` to `IterableDataset ` 
Opened by changjonathanc (closed)

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

```python
from datasets import load_dataset
dataset = load_dataset("c4", 'realnewslike', streaming =True, split='train')
dataset = dataset.remove_columns('url')
```
```
AttributeError: 'IterableDataset' object has no attribute 'remove_columns'
```

**Describe the solution you'd like**

It would be nice to have `.remove_columns()` to match the `Datasets` api. 


**Describe alternatives you've considered**

This can be done with a single call to `.map()`, 

I can try to help add this. 🤗

Comments:
Hi ! Good idea :)
If you are interested in contributing, feel free to give it a try and open a Pull Request. Also let me know if I can help you with this or if you have questions
Add SEDE dataset
Opened by Hazoom (closed)

This PR adds the SEDE dataset for the task of realistic Text-to-SQL, following the instructions of how to add a database and a dataset card.

Please see our paper for more details: https://arxiv.org/abs/2106.05006

Comments:
Thanks @albertvillanova for your great suggestions! I just pushed a new commit with the necessary fixes. For some reason, the test `test_metric_common` failed for `meteor` metric, which doesn't have any connection to this PR, so I'm trying to rebase and see if it helps.

Hi @Hazoom,

You were right: the non-passing test had nothing to do with this PR.

Unfortunately, you did a git rebase (instead of a git merge), which is not recommended once you have already opened a pull request because you mess up your pull request history. You can see that your pull request now contains:
- your commits repeated two times
- and commits which are not yours from the master branch

If you would like to clean your pull request, please make:
```
git reset --hard 587b93a
git fetch upstream master
git merge upstream/master
git push --force origin sede
```

> Hi @Hazoom,
> 
> You were right: the non-passing test had nothing to do with this PR.
> 
> Unfortunately, you did a git rebase (instead of a git merge), which is not recommended once you have already opened a pull request because you mess up your pull request history. You can see that your pull request now contains:
> 
> * your commits repeated two times
> * and commits which are not yours from the master branch
> 
> If you would like to clean your pull request, please make:
> 
> ```
> git reset --hard 587b93a
> git fetch upstream master
> git merge upstream/master
> git push --force origin sede
> ```

Thanks @albertvillanova 

> Nice! Just one final request before approving your pull request:
> 
> As you have updated the "QuerySetId" field data type, the size of the dataset is smaller now. You should regenerate the metadata. Please run:
> 
> ```
> rm datasets/sede/dataset_infos.json
> datasets-cli test datasets/sede --save_infos --all_configs
> ```

@albertvillanova Good catch, just fixed it.
OSCAR unshuffled_original_ko: NonMatchingSplitsSizesError
Opened by ayaka14732 (open)

## Describe the bug

Cannot download OSCAR `unshuffled_original_ko` due to `NonMatchingSplitsSizesError`.

## Steps to reproduce the bug

```python
>>> dataset = datasets.load_dataset('oscar', 'unshuffled_original_ko')
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=25292102197, num_examples=7345075, dataset_name='oscar'), 'recorded': SplitInfo(name='train', num_bytes=25284578514, num_examples=7344907, dataset_name='oscar')}]
```

## Expected results

Loading is successful.

## Actual results

Loading throws above error.

## Environment info

- `datasets` version: 1.12.1
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0


Comments:
I tried `unshuffled_original_da` and it is also not working
Take namespace into account in caching
Opened by lhoestq (closed)

Loading a dataset "username/dataset_name" hosted by a user on the hub used to cache the dataset only taking into account the dataset name, and ignorign the username. Because of this, if a user later loads "dataset_name" without specifying the username, it would reload the dataset from the cache instead of failing.

I changed the dataset cache and module cache mechanism to include the username in the name of the cache directory that is used:


`~/.cache/huggingface/datasets/username/dataset_name`  for the data
`~/.cache/huggingface/modules/datasets_modules/datasets/username/dataset_name` for the python files


EDIT: actually using three underscores:
`~/.cache/huggingface/datasets/username___dataset_name` for the data
`~/.cache/huggingface/modules/datasets_modules/datasets/username___dataset_name` for the python files

This PR should fix the issue https://github.com/huggingface/datasets/issues/2842

cc @stas00 

Comments:
We might have collisions if a username and a dataset_name are the same. Maybe instead serialize the dataset name by replacing `/` with some string, eg `__SLASH__`, that will hopefully never appear in a dataset or user name (it's what I did in https://github.com/huggingface/datasets-preview-backend/blob/master/benchmark/scripts/serialize.py. That way, all the datasets are one-level deep directories

IIRC we enforce that no repo id or username can contain `___` (exactly 3 underscores) specifically for this reason, so you can use that string (that we use in other projects)

cc @Pierrci 

> IIRC we enforce that no repo id or username can contain ___ (exactly 3 underscores) specifically for this reason, so you can use that string (that we use in other projects)

out of curiosity: where is it enforced?

> where is it enforced?

Nowhere yet but we should :) feel free to track in internal tracker and/or implement, as this will be useful in the future

Thanks for the trick, I'm doing the change :)
We can use
`~/.cache/huggingface/datasets/username___dataset_name` for the data
`~/.cache/huggingface/modules/datasets_modules/datasets/username___dataset_name` for the python files

Merging, though it will have to be integrated again the refactor at #2986

@lhoestq we changed a bit the naming policy on the Hub, and the substring '--' is now forbidden, which makes it available for serializing the repo names (`namespace/repo` -> `namespace--repo`). See https://github.com/huggingface/moon-landing/pull/1657 and https://github.com/huggingface/huggingface_hub/pull/545
Add Jigsaw unintended Bias
Opened by Iwontbecreative (closed)

Hi,

Here's a first attempt at this dataset. Would be great if it could be merged relatively quickly as it is needed for Bigscience-related stuff. 

This requires manual download, and I had some trouble generating dummy_data in this setting, so welcoming feedback there.

Comments:
Note that the tests seem to fail because of a bug in an Exception at the moment, see: https://github.com/huggingface/datasets/pull/2936 for the fix

@lhoestq implemented your changes, I think this might be ready for another look.

Thanks @lhoestq, implemented the changes, let me know if anything else pops up.
to_tf_dataset keeps a reference to the open data somewhere, causing issues on windows
Opened by lhoestq (closed)

To reproduce:
```python
import datasets as ds
import weakref
import gc

d = ds.load_dataset("mnist", split="train")
ref = weakref.ref(d._data.table)
tfd = d.to_tf_dataset("image", batch_size=1, shuffle=False, label_cols="label")
del tfd, d
gc.collect()
assert ref() is None, "Error: there is at least one reference left"
```

This causes issues because the table holds a reference to an open arrow file that should be closed. So on windows it's not possible to delete or move the arrow file afterwards.

Moreover the CI test of the `to_tf_dataset` method isn't able to clean up the temporary arrow files because of this.

cc @Rocketknight1 

Comments:
I did some investigation and, as it seems, the bug stems from [this line](https://github.com/huggingface/datasets/blob/8004d7c3e1d74b29c3e5b0d1660331cd26758363/src/datasets/arrow_dataset.py#L325). The lifecycle of the dataset from the linked line is bound to one of the returned `tf.data.Dataset`. So my (hacky) solution involves wrapping the linked dataset with `weakref.proxy` and adding a custom `__del__` to `tf.python.data.ops.dataset_ops.TensorSliceDataset` (this is the type of a dataset that is returned by `tf.data.Dataset.from_tensor_slices`; this works for TF 2.x, but I'm not sure `tf.python.data.ops.dataset_ops` is a valid path for TF 1.x) that deletes the linked dataset, which is assigned to the dataset object as a property. Will open a draft PR soon!

Thanks a lot for investigating !
Replace script_version with revision
Opened by albertvillanova (closed)

As discussed in https://github.com/huggingface/datasets/pull/2718#discussion_r707013278, the parameter name `script_version` is no longer applicable to datasets without loading script (i.e., datasets only with raw data files).

This PR replaces the parameter name `script_version` with `revision`.

This way, we are also aligned with:
- Transformers: `AutoTokenizer.from_pretrained(..., revision=...)`
- Hub: `HfApi.dataset_info(..., revision=...)`, `HfApi.upload_file(..., revision=...)`

Comments:
I'm also fine with the removal in 1.15
Conda build fails
Opened by albertvillanova (closed)

## Describe the bug
Current `datasets` version in conda is 1.9 instead of 1.12.

The build of the conda package fails.


Comments:
Why 1.9 ?

https://anaconda.org/HuggingFace/datasets currently says 1.11

Alright I added 1.12.0 and 1.12.1 and fixed the conda build #2952
Mutable columns argument breaks set_format
Opened by Rocketknight1 (closed)

## Describe the bug
If you pass a mutable list to the `columns` argument of `set_format` and then change the list afterwards, the returned columns also change.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("glue", "cola")

column_list = ["idx", "label"]
dataset.set_format("python", columns=column_list)
column_list[1] = "foo"  # Change the list after we call `set_format`
dataset['train'][:4].keys()
```

## Expected results
```python
dict_keys(['idx', 'label'])
```

## Actual results
```python
dict_keys(['idx'])
```

Comments:
Pushed a fix to my branch #2731
Error when downloading datasets to non-traditional cache directories
Opened by dar-tau (open)

## Describe the bug
When the cache directory is linked (soft link) to a directory on a NetApp device, the download fails. 

## Steps to reproduce the bug
```bash
ln -s /path/to/netapp/.cache ~/.cache
```

```python
load_dataset("imdb")
```

## Expected results
Successfully loading IMDB dataset

## Actual results
```
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=33432835, 
num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0,
 dataset_name='imdb')}, {'expected': SplitInfo(name='test', num_bytes=32650697, num_examples=25000, dataset_name='imdb'),
 'recorded': SplitInfo(name='test', num_bytes=659932, num_examples=503, dataset_name='imdb')}, {'expected':
 SplitInfo(name='unsupervised', num_bytes=67106814, num_examples=50000, dataset_name='imdb'), 'recorded':
 SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.1.2
- Platform: Ubuntu 
- Python version: 3.8

## Extra notes
Stranger yet, trying to debug the phenomenon, I found the range of results to vary a lot without clear direction:
 - With `cache_dir="/path/to/netapp/.cache"` the same thing happens.
 - However, when linking `~/netapp/` to `/path/to/netapp` *and* setting `cache_dir="~/netapp/.cache/huggingface/datasets"` - it does work
 - On the other hand, when linking `~/.cache` to `~/netapp/.cache` without using `cache_dir`, it does work anymore.

While I could test it only for a NetApp device, it might have to do with any other mounted FS.

Thanks :)



Comments:
Same here !