noamaanMulla-03/datasets-issues huggingface dataset:
Protect master branch

After accidental merge commit (91c55355b634d0dc73350a7ddee1a6776dbbdd69) into `datasets` master branch, all commits present in the feature branch were permanently added to `datasets` master branch history, as e.g.:
- 00cc036fea7c7745cfe722360036ed306796a3f2
- 13ae8c98602bbad8197de3b9b425f4c78f582af1
- ...

I propose to protect our master branch, so that we avoid we can accidentally make this kind of mistakes in the future:
- [x] For Pull Requests using GitHub, allow only squash merging, so that only a single commit per Pull Request is merged into the master branch
  - Currently, simple merge commits are already disabled
  - I propose to disable rebase merging as well
- ~~Protect the master branch from direct pushes (to avoid accidentally pushing of merge commits)~~
  - ~~This protection would reject direct pushes to master branch~~
  - ~~If so, for each release (when we need to commit directly to the master branch), we should previously disable the protection and re-enable it again after the release~~
- [x] Protect the master branch only from direct pushing of **merge commits**
  - GitHub offers the possibility to protect the master branch only from merge commits (which are the ones that introduce all the commits from the feature branch into the master branch).
  - No need to disable/re-enable this protection on each release 

This purpose of this Issue is to open a discussion about this problem and to agree in a solution.

Labels: name: enhancement

Cool, I think we can do both :)

@lhoestq now the 2 are implemented.

Please note that for the the second protection, finally I have chosen to protect the master branch only from **merge commits** (see update comment above), so no need to disable/re-enable the protection on each release (direct commits, different from merge commits, can be pushed to the remote master branch; and eventually reverted without messing up the repo history).
Don't use old, incompatible cache for the new `filter`

#2836 changed `Dataset.filter` and the resulting data that are stored in the cache are different and incompatible with the ones of the previous `filter` implementation.

However the caching mechanism wasn't able to differentiate between the old and the new implementation of filter (only the method name was taken into account). 

This is an issue because anyone that update `datasets` and re-runs some code that uses `filter` would see an error, because the cache would try to load an incompatible `filter` result.

To fix this I added the notion of versioning for dataset transform in the caching mechanism, and bumped the version of the `filter` implementation to 2.0.0

This way the new `filter` outputs are now considered different from the old ones from the caching point of view.

This should fix #2943

cc @anton-l
Dummy labels no longer on by default in `to_tf_dataset`

After more experimentation, I think I have a way to do things that doesn't depend on adding `dummy_labels` - they were quite a hacky solution anyway!


@lhoestq Let me make sure we never need it, and if not then I'll remove it entirely in a follow-up PR.

Thanks ;) it will be less confusing and easier to maintain to not keep unused hacky features
OpenWebText: NonMatchingSplitsSizesError

## Describe the bug

When downloading `openwebtext`, I'm getting:
```
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=39769494896, num_examples=8013769, dataset_name='openwebtext'), 'recorded': SplitInfo(name='train', num_bytes=39611023912, num_examples=7982430, dataset_name='openwebtext')}]
```

I suspect that the file we download from has changed since the size doesn't look like to match with documentation

`Downloading:   0%|          | 0.00/12.9G [00:00<?, ?B/s]` This suggest the total size is 12.9GB, whereas the one documented mentions `Size of downloaded dataset files: 12283.35 MB`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset("openwebtext", download_mode="force_redownload")
```

## Expected results

Loading is successful

## Actual results

Loading throws above error.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.2
- Platform: linux (Redhat version 8.1)
- Python version: 3.8
- PyArrow version: 4.0.1


Labels: name: bug

Thanks for reporting, I'm updating the verifications metadata

I just regenerated the verifications metadata and noticed that nothing changed: the data file is fine (the checksum didn't change), and the number of examples is still 8013769. Not sure how you managed to get 7982430 examples.

Can you try to delete your cache ( by default at `~/.cache/huggingface/datasets`) and try again please ?
Also, on which platform are you (linux/macos/windows) ?

I'll try without deleting the whole cache (we have large datasets already stored). I was under the impression that `download_mode="force_redownload"` would bypass cache.
Sorry plateform should be linux (Redhat version 8.1)

Hi @thomasw21 , are you still having this issue after clearing your cache ?

Sorry I haven't had time to work on this. I'll close and re-open if I can't figure out why I'm having this issue. Thanks for taking a look !
Conda build fails

## Describe the bug
Current `datasets` version in conda is 1.9 instead of 1.12.

The build of the conda package fails.


Labels: name: bug

Why 1.9 ?

https://anaconda.org/HuggingFace/datasets currently says 1.11

Alright I added 1.12.0 and 1.12.1 and fixed the conda build #2952
Use pyarrow.Table.replace_schema_metadata instead of pyarrow.Table.cast

This PR partially addresses #2252.

``update_metadata_with_features`` uses ``Table.cast`` which slows down ``load_from_disk`` (and possibly other methods that use it) for very large datasets. Since ``update_metadata_with_features`` is only updating the schema metadata, it makes more sense to use ``pyarrow.Table.replace_schema_metadata`` which is much faster. This PR adds a ``replace_schema_metadata`` method to all table classes, and modifies  ``update_metadata_with_features`` to use it instead of ``cast``.
SacreBLEU update

With the latest release of [sacrebleu](https://github.com/mjpost/sacrebleu), `datasets.metrics.sacrebleu` is broken, and getting error.

    AttributeError: module 'sacrebleu' has no attribute 'DEFAULT_TOKENIZER'

this happens since in new version of sacrebleu there is no `DEFAULT_TOKENIZER`, but sacrebleu.py tries to import it anyways. This can be fixed currently with fixing `sacrebleu==1.5.0`

## Steps to reproduce the bug
```python
sacrebleu= datasets.load_metric('sacrebleu')
predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
references = ["It is a guide to action that ensures that the military will forever heed Party commands"]
results = sacrebleu.compute(predictions=predictions, references=references)
print(results)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: Python 3.8.0
- PyArrow version: 5.0.0


Labels: name: bug

Hi @devrimcavusoglu, 
I tried your code with latest version of `datasets`and `sacrebleu==1.5.1` and it's running fine after changing one small thing:
```
sacrebleu = datasets.load_metric('sacrebleu')
predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
references = [["It is a guide to action that ensures that the military will forever heed Party commands"]]  # double brackets here should do the work
results = sacrebleu.compute(predictions=predictions, references=references)
print(results)
output: {'score': 41.180376356915765, 'counts': [11, 8, 6, 4], 'totals': [18, 17, 16, 15], 'precisions': [61.111111111111114, 47.05882352941177, 37.5, 26.666666666666668], 'bp': 1.0, 'sys_len': 18, 'ref_len': 16}
```

@bhavitvyamalik hmm. I forgot double brackets, but still didn't work when used it with double brackets. It may be an isseu with platform (using win-10 currently), or versions. What is your platform and your version info for datasets, python, and sacrebleu ?

You can check that here, I've reproduced your code in [Google colab](https://colab.research.google.com/drive/1X90fHRgMLKczOVgVk7NDEw_ciZFDjaCM?usp=sharing). Looks like there was some issue in `sacrebleu` which was fixed later from what I've found [here](https://github.com/pytorch/fairseq/issues/2049#issuecomment-622367967). Upgrading `sacrebleu` to latest version should work.

It seems that next release of `sacrebleu` (v2.0.0) will break our `datasets` implementation to compute it. See my Google Colab: https://colab.research.google.com/drive/1SKmvvjQi6k_3OHsX5NPkZdiaJIfXyv9X?usp=sharing

I'm reopening this Issue and making a Pull Request to fix it.

> It seems that next release of `sacrebleu` (v2.0.0) will break our `datasets` implementation to compute it. See my Google Colab: https://colab.research.google.com/drive/1SKmvvjQi6k_3OHsX5NPkZdiaJIfXyv9X?usp=sharing
> 
> I'm reopening this Issue and making a Pull Request to fix it.

How did you solve him
Update legacy Python image for CI tests in Linux

Instead of legacy, use next-generation convenience images, built from the ground up with CI, efficiency, and determinism in mind. Here are some of the highlights:

- Faster spin-up time - In Docker terminology, these next-gen images will generally have fewer and smaller layers. Using these new images will lead to faster image downloads when a build starts, and a higher likelihood that the image is already cached on the host.

- Improved reliability and stability - The existing legacy convenience images are rebuilt practically every day with potential changes from upstream that we cannot always test fast enough. This leads to frequent breaking changes, which is not the best environment for stable, deterministic builds. Next-gen images will only be rebuilt for security and critical-bugs, leading to more stable and deterministic images.

More info: https://circleci.com/docs/2.0/circleci-images


There is an exception when running `pip install .[tests]`:
```
Processing /home/circleci/datasets
Collecting numpy>=1.17 (from datasets==1.12.2.dev0)
  Downloading https://files.pythonhosted.org/packages/45/b2/6c7545bb7a38754d63048c7696804a0d947328125d81bf12beaa692c3ae3/numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl (13.4MB)
    100% |████████████████████████████████| 13.4MB 3.9MB/s  eta 0:00:011

...

Collecting faiss-cpu (from datasets==1.12.2.dev0)
  Downloading https://files.pythonhosted.org/packages/87/91/bf8ea0d42733cbb04f98d3bf27808e4919ceb5ec71102e21119398a97237/faiss-cpu-1.7.1.post2.tar.gz (41kB)
    100% |████████████████████████████████| 51kB 30.9MB/s ta 0:00:01
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/home/circleci/.pyenv/versions/3.6.14/lib/python3.6/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/home/circleci/.pyenv/versions/3.6.14/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/home/circleci/.pyenv/versions/3.6.14/lib/python3.6/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/home/circleci/.pyenv/versions/3.6.14/lib/python3.6/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-1pop4blm/numpy-1.21.2/setup.py", line 34, in <module>
        method can be invoked.
    RuntimeError: Python version >= 3.7 required.
```

Apparently, `numpy-1.21.2` tries to be installed in the temporary directory `/tmp/easy_install-1pop4blm` instead of the downloaded `numpy-1.19.5` (requirement of `datasets`).

This is caused because `pip` downloads the `.tar.gz` (instead of the `.whl`) and tries to build it in a tmp dir.
Add SEDE dataset

This PR adds the SEDE dataset for the task of realistic Text-to-SQL, following the instructions of how to add a database and a dataset card.

Please see our paper for more details: https://arxiv.org/abs/2106.05006


Thanks @albertvillanova for your great suggestions! I just pushed a new commit with the necessary fixes. For some reason, the test `test_metric_common` failed for `meteor` metric, which doesn't have any connection to this PR, so I'm trying to rebase and see if it helps.

Hi @Hazoom,

You were right: the non-passing test had nothing to do with this PR.

Unfortunately, you did a git rebase (instead of a git merge), which is not recommended once you have already opened a pull request because you mess up your pull request history. You can see that your pull request now contains:
- your commits repeated two times
- and commits which are not yours from the master branch

If you would like to clean your pull request, please make:
```
git reset --hard 587b93a
git fetch upstream master
git merge upstream/master
git push --force origin sede
```

> Hi @Hazoom,
> 
> You were right: the non-passing test had nothing to do with this PR.
> 
> Unfortunately, you did a git rebase (instead of a git merge), which is not recommended once you have already opened a pull request because you mess up your pull request history. You can see that your pull request now contains:
> 
> * your commits repeated two times
> * and commits which are not yours from the master branch
> 
> If you would like to clean your pull request, please make:
> 
> ```
> git reset --hard 587b93a
> git fetch upstream master
> git merge upstream/master
> git push --force origin sede
> ```

Thanks @albertvillanova 

> Nice! Just one final request before approving your pull request:
> 
> As you have updated the "QuerySetId" field data type, the size of the dataset is smaller now. You should regenerate the metadata. Please run:
> 
> ```
> rm datasets/sede/dataset_infos.json
> datasets-cli test datasets/sede --save_infos --all_configs
> ```

@albertvillanova Good catch, just fixed it.
Add Jigsaw unintended Bias

Hi,

Here's a first attempt at this dataset. Would be great if it could be merged relatively quickly as it is needed for Bigscience-related stuff. 

This requires manual download, and I had some trouble generating dummy_data in this setting, so welcoming feedback there.


Note that the tests seem to fail because of a bug in an Exception at the moment, see: https://github.com/huggingface/datasets/pull/2936 for the fix

@lhoestq implemented your changes, I think this might be ready for another look.

Thanks @lhoestq, implemented the changes, let me know if anything else pops up.
raise TypeError( TypeError: Provided `function` which is applied to all elements of table returns a variable of type <class 'list'>. Make sure provided `function` returns a variable of type `dict` to update the dataset or `None` if you are only interested in side effects.

## Describe the bug
A clear and concise description of what the bug is.
I am trying to use Dataset to load my file in order to use Bert embeddings model baut when I finished loading using dataset and I want to pass to the tokenizer using the function map; I get the following error : raise TypeError(
TypeError: Provided `function` which is applied to all elements of table returns a variable of type <class 'list'>. Make sure provided `function` returns a variable of type `dict` to update the dataset or `None` if you are only interested in side effects.

I was able to load my file using dataset before but since this morning , I keep getting this erreor.
## Steps to reproduce the bug
```python
# Xtrain, ytrain, filename, len_labels = read_file_2(fic)
	# Xtrain, lge_size = get_flaubert_layer(Xtrain, path_to_model_lge)

	data_preprocessed = make_new_traindata(Xtrain)
	
	my_dict = {"verbatim": data_preprocessed[1], "label": ytrain} # lemme avec conjonction
	dataset = Dataset.from_dict(my_dict)
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Labels: name: bug
Magnet’s

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Labels: name: dataset request
Allow for incremental cumulative metric updates in a distributed setup

Currently, using a metric allows for one of the following:
- Per example/batch metrics
- Cumulative metrics over the whole data

What I'd like is to have an efficient way to get cumulative metrics over the examples/batches added so far, in order to display it as part of the progress bar during training/evaluation.

Since most metrics are just an average of per-example metrics (which aren't?), an efficient calculation can be done as follows:
`((score_cumulative * n_cumulative) + (score_new * n_new)) / (n_cumulative+ n_new)`
where `n` and `score`  refer to number of examples and metric score, `cumulative` refers to the cumulative metric and `new` refers to the addition of new examples.

If you don't want to add this capability in the library, a simple solution exists so users can do it themselves:
It is easy to implement for a single process setup, but in a distributed one there is no way to get the correct `n_new`.
The solution for this is to return the number of examples that was used to compute the metrics in `.compute()` by adding the following line here:
https://github.com/huggingface/datasets/blob/5a3221785311d0ce86c2785b765e86bd6997d516/src/datasets/metric.py#L402-L403
```
output["number_of_examples"] = len(predictions)
```
and also remove the log message here so it won't spam:
https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/src/datasets/metric.py#L411

If this change is ok with you, I'll open a pull request.


Labels: name: enhancement
masakhaner dataset load problem

## Describe the bug
Masakhaner dataset is not loading

## Steps to reproduce the bug
```
from datasets import load_dataset
dataset = load_dataset("masakhaner",'amh')
```

## Expected results
Expected the return of a dataset

## Actual results

```

NonMatchingSplitsSizesError               Traceback (most recent call last)
<ipython-input-3-a6abc1161d4c> in <module>()
      1 from datasets import load_dataset
      2 
----> 3 dataset = load_dataset("masakhaner",'amh')



3 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py
 in verify_splits(expected_splits, recorded_splits)
     72     ]
     73     if len(bad_splits) > 0:
---> 74         raise NonMatchingSplitsSizesError(str(bad_splits))
     75     logger.info("All the splits matched successfully.")
     76 

NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=639927, num_examples=1751, dataset_name='masakhaner'), 'recorded': SplitInfo(name='train', num_bytes=639911, num_examples=1750, dataset_name='masakhaner')}, {'expected': SplitInfo(name='validation', num_bytes=92768, num_examples=251, dataset_name='masakhaner'), 'recorded': SplitInfo(name='validation', num_bytes=92753, num_examples=250, dataset_name='masakhaner')}, {'expected': SplitInfo(name='test', num_bytes=184286, num_examples=501, dataset_name='masakhaner'), 'recorded': SplitInfo(name='test', num_bytes=184271, num_examples=500, dataset_name='masakhaner')}]
```

## Environment info
Google Colab


Labels: name: bug

Thanks for reporting, @ontocord. We are fixing the wrong metadata.
Add tutorial for no-code dataset upload

This PR is for a tutorial for uploading a dataset to the Hub. It relies on the Hub UI elements to upload a dataset, introduces the online tagging tool for creating tags, and the Dataset card template to get a head start on filling it out. The addition of this tutorial should make it easier for beginners to upload a dataset without accessing the terminal or knowing Git.

Labels: name: documentation

Cool, love it ! :)

Feel free to add a paragraph saying how to load the dataset:
```python
from datasets import load_dataset

dataset = load_dataset("stevhliu/demo")

# or to separate each csv file into several splits
data_files = {"train": "train.csv", "test": "test.csv"}
dataset = load_dataset("stevhliu/demo", data_files=data_files)
print(dataset["train"][0])
```

Perfect, feel free to mark this PR ready for review :)

cc @albertvillanova do you have any comment ? You can check the tutorial here:
https://47389-250213286-gh.circle-artifacts.com/0/docs/_build/html/no_code_upload.html

Maybe we can just add a list of supported file types:
- csv
- json
- json lines
- text
- parquet

I just added a mention of the login for private datasets. Don't hesitate to edit or comment.

Otherwise I think it's all good, feel free to merge it @stevhliu  if you don't have other changes to make :)
Run tests in parallel

Run CI tests in parallel to speed up the test suite.

Speed up results:
- Linux: from `7m 30s` to `5m 32s`
- Windows: from `13m 52s` to `11m 10s`



There is a speed up in Windows machines:
- From `13m 52s` to `11m 10s`

In Linux machines, some workers crash with error message:
```
OSError: [Errno 12] Cannot allocate memory
```

There is also a speed up in Linux machines:
- From `7m 30s` to `5m 32s`
Run CI tests against non-production server

Currently, the CI test suite performs requests to the HF production server.

As discussed with @elishowk, we should refactor our tests to use the HF staging server instead, like `huggingface_hub` and `transformers`.


Hey @albertvillanova could you provide more context, including extracts from the discussion we had ?

Let's ping @Pierrci @julien-c and @n1t0 for their opinion about that

@julien-c increased the huggingface.co production workers in order to see if it solve [the 502 you had this morning](https://app.circleci.com/pipelines/github/huggingface/datasets/7843/workflows/fc83fa32-18f5-4dc3-9e2f-ba277ae1af74)

For the decision process: be aware that moon-staging does not have persistent repos (they are deleted regularly). as a consequence, **if the moon-staging solution is validated**, you should consider a way to keep the repository that are loaded in tests. These are the ones I found : https://github.com/huggingface/datasets/blob/d488db2f64f312f88f72bbc57a09b7eddb329182/tests/test_load.py and https://github.com/huggingface/datasets/blob/40773111c3e7db8a992fa1c48af32d900a1018d6/tests/test_streaming_download_manager.
Don't keep the dummy data folder or dataset_infos.json when resolving data files

When there's no dataset script, all the data files of a folder or a repository on the Hub are loaded as data files.

There are already a few exceptions:
- files starting with "." are ignored
- the dataset card "README.md" is ignored
- any file named "config.json" is ignored (currently it isn't used anywhere, but it could be used in the future to define splits or configs for example, but not 100% sure)

However any data files in a folder named "dummy" should be ignored as well as they should only be used to test the dataset.
Same for "dataset_infos.json" which should only be used to get the `dataset.info`

Labels: name: enhancement

Hi @lhoestq I am new to huggingface datasets, I would like to work on this issue!


Thanks for the help :) 

As mentioned in the PR, excluding files named "dummy_data.zip" is actually more general than excluding the files inside a "dummy" folder. I just did the change in the PR, I think we can merge it now