noamaanMulla-03/datasets-issues huggingface dataset:
FORCE_REDOWNLOAD does not work

## Describe the bug
With GenerateMode.FORCE_REDOWNLOAD, the documentation says 
    +------------------------------------+-----------+---------+
    |                                    | Downloads | Dataset |
    +====================================+===========+=========+
    | `REUSE_DATASET_IF_EXISTS` (default)| Reuse     | Reuse   |
    +------------------------------------+-----------+---------+
    | `REUSE_CACHE_IF_EXISTS`            | Reuse     | Fresh   |
    +------------------------------------+-----------+---------+
    | `FORCE_REDOWNLOAD`                 | Fresh     | Fresh   |
    +------------------------------------+-----------+---------+

However, the old dataset is loaded even when FORCE_REDOWNLOAD is chosen.

## Steps to reproduce the bug
```python

import pandas as pd
from datasets import load_dataset, GenerateMode
pd.DataFrame(range(5), columns=['numbers']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)
pd.DataFrame(range(10), columns=['numerals']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)

```

## Expected results
Dataset({
    features: ['numbers'],
    num_rows: 5
})
Dataset({
    features: ['numerals'],
    num_rows: 10
})

## Actual results
Dataset({
    features: ['numbers'],
    num_rows: 5
})
Dataset({
    features: ['numbers'],
    num_rows: 5
})


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-4.14.181-108.257.amzn1.x86_64-x86_64-with-glibc2.10
- Python version: 3.7.10
- PyArrow version: 3.0.0


Labels: name: bug

Hi ! Thanks for reporting. The error seems to happen only if you use compressed files.

The second dataset is prepared in another dataset cache directory than the first - which is normal, since the source file is different. However, it doesn't uncompress the new data file because it finds the old uncompressed data in the extraction cache directory.

If we fix the extraction cache mechanism to uncompress a local file if it changed then it should fix the issue.
Currently the extraction cache mechanism only takes into account the path of the compressed file, which is an issue.

Facing the same issue, is there any way to overtake this issue until it will be fixed? 

You can clear your extraction cache in the meantime (by default at `~/.cache/huggingface/datasets/downloads/extracted`)
Protect master branch to force contributions via Pull Requests

In order to have a clearer Git history in the master branch, I propose to protect it so that all contributions must be done through a Pull Request and no direct commits to master are allowed.
- The Pull Request allows to give context, discuss any potential issues and improve the quality of the contribution
- The Pull Request will eventually be squashed and merged into master with a single commit that links to the Pull Request page (with all the context/discussions)

Note that we already implemented a protection in the master branch to avoid *merge* commits and ensure a linear history. This proposal goes one step further by avoiding all kind of direct commits and forcing contributions **only** through Pull Requests.

Please note that we can temporarily deactivate this protection if we need to make a direct commit, e.g. at each new version release.

The only way GitHub allows this kind or protection is by requiring a minimal number (at least one) of approvals of the Pull Request. The inconvenient is that the PR creator cannot approve their own PR: another person must approve it before it can be merged into master. To circumvent this, we could eventually disable this protection in the master branch when an urgent commit is needed (e.g. for a hotfix) and there is no other person available at that time to approve the PR.


Labels: name: enhancement

It would be nice to protect the master from direct commits, but still having a way to merge our own PRs when no review is required (for example when updating a dataset_infos.json file, or minor bug fixes - things that happen quite often actually).
Do you know if there's a way ?

you can if you're an admin of the repo

This is done. Now the master branch is protected:
- [x] Require a pull request before merging: all commits must be made to a non-protected branch and submitted via a pull request
  - Required number of approvals before merging: 1 
- [x] Require linear history: prevent merge commits from being pushed
- [x] These requirements are not enforced for administrators
- [x] Additionally, the master branch is also protected against deletion and force pushes

CC: @lhoestq @julien-c @thomwolf
Using sharding to parallelize indexing

**Is your feature request related to a problem? Please describe.**
Creating an elasticsearch index on large dataset could be quite long and cannot be parallelized on shard (the index creation is colliding)

**Describe the solution you'd like**
When working on dataset shards, if an index already exists, its mapping should be checked and if compatible, the indexing process should continue with the shard data. 

Additionally, at the end of the process, the `_indexes` dict should be send back to the original dataset object (from which the shards have been created) to allow to use the index for later filtering on the whole dataset.

**Describe alternatives you've considered**
Each dataset shard could created independent partial indices. then on the whole dataset level, indices should be all referred in `_indexes` dict and be used in querying through `get_nearest_examples()`. The drawback is that the scores will be computed independently on the partial indices leading to inconsistent values for most scoring based on corpus level statistics (tf/idf, BM25).

**Additional context**
The objectives is to parallelize the index creation to speed-up the process (ie surcharging the ES server which is fine to handle large load) while later enabling search on the whole dataset.

Labels: name: enhancement
Add  `remove_columns` to `IterableDataset ` 

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

```python
from datasets import load_dataset
dataset = load_dataset("c4", 'realnewslike', streaming =True, split='train')
dataset = dataset.remove_columns('url')
```
```
AttributeError: 'IterableDataset' object has no attribute 'remove_columns'
```

**Describe the solution you'd like**

It would be nice to have `.remove_columns()` to match the `Datasets` api. 


**Describe alternatives you've considered**

This can be done with a single call to `.map()`, 

I can try to help add this. 🤗

Labels: name: enhancement

name: good first issue

Hi ! Good idea :)
If you are interested in contributing, feel free to give it a try and open a Pull Request. Also let me know if I can help you with this or if you have questions
Add `remove_columns` to `IterableDataset`

Fixes #2944

WIP
* Not tested yet.
* We might want to allow batched remove for efficiency.

@lhoestq Do you think it should have `batched=` and `batch_size=`?


Thanks ! That looks all good :)

I don't think that batching would help. Indeed we're dealing with python iterators that yield elements one by one, so batched `map` needs to accumulate a batch, apply the function, and then yield examples from the batch.

Though once we have parallel processing in `map`, we can reconsider it

Also feel free to check the CI failure - apparently the import of `Union` is missing

Thanks for the review and explaining that! 
On top of what you said, I think `remove_columns` is very unlikely to be a bottleneck, so it doesn't matter anyways.

Thank you for reviewing! @mariosasko 

I wonder how the checking would work. Is there any checking present in `IterableDataset ` now? What if `.remove_columns()` is applied after some arbitrary `.map()`?

> I wonder how the checking would work. Is there any checking present in IterableDataset now? What if .remove_columns() is applied after some arbitrary .map()?

That's the challenge here indeed ^^ In this case it's not trivial to know the names of the columns. Feel free to open an issue so we can discuss this
Identify which shard data belongs to

**Is your feature request related to a problem? Please describe.**

I'm training on a large dataset made of multiple sub-datasets.
During training I can observe some jumps in loss which may correspond to different shards.

![image](https://user-images.githubusercontent.com/715491/136668758-521263aa-a9b2-4ad2-8d22-060b6bf86a1c.png)

My suspicion is that either:
* some of the sub-datasets are harder for the model than others
* some of the sub-datasets are not formatted properly

I'd like to identify which shards correspond to those jumps.

**Describe the solution you'd like**

It would be nice to have a key associated to each data sample or data batch containing details on where the data comes from (shard idx + item idx within the shard).
This should be supported both in local and streaming mode.

**Describe alternatives you've considered**
A fix would be for me to add myself details (shard id, sample id) as part of each data sample.
The inconvenient is that it requires users to process/reupload every dataset when they need this feature.

Labels: name: enhancement

Independently of this I think it raises the need to allow multiprocessing during streaming so that we get samples from multiple shards in one batch.
Error in per class precision computation 

## Describe the bug
When trying to get the per class precision values by providing `average=None`, following error is thrown `ValueError: can only convert an array of size 1 to a Python scalar`

## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
precision_metric = load_metric("precision")
predictions = [0, 2, 1, 0, 0, 1]
references = [0, 1, 2, 0, 1, 2]
results = precision_metric.compute(predictions=predictions, references=references, average=None)
```

## Expected results
`  {'precision': array([0.66666667, 0.        , 0.        ])}`
as per https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py

## Actual results
```
   output = self._compute(predictions=predictions, references=references, **kwargs)
  File "~/.cache/huggingface/modules/datasets_modules/metrics/precision/94709a71c6fe37171ef49d3466fec24dee9a79846c9f176dff66a649e9811690/precision.py", line 110, in _compute
    sample_weight=sample_weight,
ValueError: can only convert an array of size 1 to a Python scalar
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: linux
- Python version: 3.6.9
- PyArrow version: 5.0.0


Labels: name: bug

Hi @tidhamecha2, thanks for reporting.

Indeed, we fixed this issue just one week ago: #3008

The fix will be included in our next version release.

In the meantime, you can incorporate the fix by installing `datasets` from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```
DatasetDict.filter and Dataset.filter crashes with any "fn_kwargs" argument

## Describe the bug
The ".filter" method of DatasetDict or Dataset objects fails when passing any "fn_kwargs" argument

## Steps to reproduce the bug
```python
import datasets

example_dataset = datasets.Dataset.from_dict({"a": {1, 2, 3, 4}})

def filter_value(example, value):
    return example['a'] == value

filtered = example_dataset.filter(filter_value, fn_kwargs={'value': 3})
```

## Expected results
`filtered` is a dataset containing {"a": {3}}

## Actual results

> Traceback (most recent call last):
>   File "C:\Users\qsemi\Documents\git\nlp_experiments\gpt_celebrity\src\test_faulty_filter.py", line 8, in <module>
>     filtered = example_dataset.filter(filter_value, fn_kwargs={'value': 3})
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 185, in wrapper
>     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\fingerprint.py", line 398, in wrapper
>     out = func(self, *args, **kwargs)
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 2169, in filter
>     indices = self.map(
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 1686, in map
>     return self._map_single(
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 185, in wrapper
>     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\fingerprint.py", line 398, in wrapper
>     out = func(self, *args, **kwargs)
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 2048, in _map_single
>     batch = apply_function_on_filtered_inputs(
>   File "C:\Users\qsemi\miniconda3\envs\main\lib\site-packages\datasets\arrow_dataset.py", line 1939, in apply_function_on_filtered_inputs
>     function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
> TypeError: get_indices_from_mask_function() got an unexpected keyword argument 'value'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.9.7
- PyArrow version: 5.0.0


Labels: name: bug

Hi @DrMatters, thanks for reporting.

This issue was fixed 14 days ago: #2950.

Currently, the fix is only in the master branch and will be made available in our next library release.

In the meantime, you can incorporate the fix by installing datasets from the master branch:
```shell
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```shell
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```

Thanks, sorry for bothering
load_dataset cannot download the data and hangs on forever if cache dir specified

## Describe the bug
After updating datasets, a code that ran just fine for ages began to fail. Specifying _datasets.load_dataset_'s _cache_dir_ optional argument on Windows 10 machine results in data download to hang on forever. Same call without cache_dir works just fine. Surprisingly exact same code just runs perfectly fine on Linux docker instance running in cloud.

Unfortunately I updated Windows also at the same time and I can't remember which version of datasets was running in my conda environment prior to the update otherwise I would have tried both to check this out.  :(

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```
cache_dir = 'c:/data/datasets'
dataset = load_dataset('wikipedia', '20200501.en', split='train',cache_dir=cache_dir) 
```
Note that exact same code without specifying _cache_dir_ argument works perfectly fine.
```
cache_dir = 'c:/data/datasets'
dataset = load_dataset('wikipedia', '20200501.en', split='train') 
```

## Expected results
Downloads the dataset and cache is handled in the _cache_dir_ directory
## Actual results
Data download keeps hanging on forever, **NO TRACEBACK**!

## Environment info
- `datasets` version: 1.12.1
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0


Labels: name: bug

Issue was environment inconsistency, updating packages did the trick

`conda install -c huggingface -c conda-forge datasets`

> Collecting package metadata (current_repodata.json): done
> Solving environment: |
> The environment is inconsistent, please check the package plan carefully
> The following packages are causing the inconsistency:
> 
>   - conda-forge/noarch::datasets==1.12.1=pyhd8ed1ab_1
>   - conda-forge/win-64::multiprocess==0.70.12.2=py38h294d835_0
> done
> 
> Package Plan
> 
>   environment location: C:\xxx\anaconda3\envs\UnBias-94-1
> 
>   added / updated specs:
>     - datasets
> 
> 
> The following NEW packages will be INSTALLED:
> 
>   dill               conda-forge/noarch::dill-0.3.4-pyhd8ed1ab_0
> 
> The following packages will be UPDATED:
> 
>   ca-certificates    pkgs/main::ca-certificates-2021.9.30-~ --> conda-forge::ca-certificates-2021.10.8-h5b45459_0
>   certifi            pkgs/main::certifi-2021.5.30-py38haa9~ --> conda-forge::certifi-2021.10.8-py38haa244fe_0
> 
> The following packages will be SUPERSEDED by a higher-priority channel:
>
Fix task reloading from cache

When reloading a dataset from the cache when doing `map`, the tasks templates were kept instead of being updated regarding the output of the `map` function. This is an issue because we drop the tasks templates that are not compatible anymore after `map`, for example if a column of the template was removed.

This PR fixes this and for convenience introduces a decorator `@transmit_tasks` that takes care of doing this verification, similar to the `@transmit_format` decorator.

This should fix issue https://github.com/huggingface/datasets/issues/3047 cc @sgugger
Loading allenai/c4 in streaming mode does too many HEAD requests

This does 60,000+ HEAD requests to get all the ETags of all the data files:
```python
from datasets import load_dataset
load_dataset("allenai/c4", streaming=True)
```
It makes loading the dataset completely impractical.

The ETags are used to compute the config id (it must depend on the data files being used).
Instead of using the ETags, we could simply use the commit hash of the dataset repository on the hub, as well and the glob pattern used to resolve the files (here it's `*` by default, to load all the files of the repository)

Labels: name: enhancement

name: streaming

https://github.com/huggingface/datasets/blob/6c766f9115d686182d76b1b937cb27e099c45d68/src/datasets/builder.py#L179-L186

Thanks a lot!!!
Load private data files + use glob on ZIP archives for json/csv/etc. module inference

As mentioned in https://github.com/huggingface/datasets/issues/3032 loading data files from private repository isn't working correctly because of the data files resolved.

#2986 did a refactor of the data files resolver. I added authentication to it.

I also improved it to glob inside ZIP archives to look for json/csv/etc. files and infer which dataset builder (json/csv/etc.) to use.

Fix https://github.com/huggingface/datasets/issues/3032

Note that #2986 needs to get merged first


I have an error on windows:
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```
at the `fsspec` call in `xglob`:
```python
fs, *_ = fsspec.get_fs_token_paths(urlpath, storage_options=storage_options)
```

Looks like the windows CI has an SSL issue... 

I can reproduce it on my windows machine. On linux it works fine though

I'm just skipping the windows test for now

The Windows CI failure seems unrelated to this PR
```python
ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3
```
Error when loading private dataset with "data_files" arg

## Describe the bug
A clear and concise description of what the bug is.

Private datasets with no loading script can't be loaded using `data_files` parameter.

## Steps to reproduce the bug
```python
from datasets import load_dataset
data_files = {"train": "**/train/*/*.jsonl", "valid": "**/valid/*/*.jsonl"}
dataset = load_dataset('dalle-mini/encoded', data_files=data_files, use_auth_token=True, streaming=True)
```

Same error happens in non-streaming mode.

## Expected results
Files should be loaded (whether in streaming or not).

## Actual results
Error:

```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    539                 try:
--> 540                     local_path = cached_path(file_path, download_config=download_config)
    541                 except FileNotFoundError:

8 frames
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/dalle-mini/encoded/resolve/main/encoded.py

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/datasets/dalle-mini/encoded?full=true

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    547                     except Exception:
    548                         raise FileNotFoundError(
--> 549                             f"Couldn't find a directory or a {resource_type} named '{path}'. "
    550                             f"It doesn't exist locally at {expected_dir_for_combined_path_abs} or remotely on {hf_api.endpoint}/datasets"
    551                         )

FileNotFoundError: Couldn't find a directory or a dataset named 'dalle-mini/encoded'. It doesn't exist locally at /content/dalle-mini/encoded or remotely on https://huggingface.co/datasets
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0

@lhoestq 

Labels: name: bug

We'll do a release tomorrow or on wednesday to make the fix available :)

Thanks for reproting !
Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder

## Adding a Dataset
- **Name:** text, json, csv
- **Description:** I am developing a customized dataset loading script. The problem is  mainly about my custom dataset is seperate into many files and I only find a dataset loading template in [https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py](https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py) that can handle my circumstance. I'm afraid these templates are too old to use. Could you re-add these three templates to current master branch?



Labels: name: dataset request

Hi @zixiliuUSC, 

As explained in the documentation (https://huggingface.co/docs/datasets/loading.html#json), we support loading any dataset in JSON (as well as CSV, text, Parquet) format:
```python
ds = load_dataset('json', data_files='my_file.json')
```