jamesLeeeeeee/datasets-github-issues huggingface dataset:
_prepare_split will overwrite DatasetBuilder.info.features
Author: Gforky | Pull request: no | State: closed

Hi, here is my issue:
I initialized a Csv datasetbuilder with specific features:
```
def get_dataset_features(data_args):
    features = {}
    if data_args.text_features:
        features.update({text_feature: hf_features.Value("string") for text_feature in data_args.text_features.strip().split(",")})
    if data_args.num_features:
        features.update({text_feature: hf_features.Value("float32") for text_feature in data_args.num_features.strip().split(",")})
    if data_args.label_classes:
        features["label"] = hf_features.ClassLabel(names=data_args.label_classes.strip().split(","))
    else:
        features["label"] = hf_features.Value("float32")
    return hf_features.Features(features)

datasets = load_dataset(extension,
                                data_files=data_files,
                                sep=data_args.delimiter,
                                header=data_args.header,
                                column_names=data_args.column_names.split(",") if data_args.column_names else None,
                                features=get_dataset_features(data_args=data_args))
```
The `features` is printout as below before `builder_instance.as_dataset` is called:
```
{'label': ClassLabel(num_classes=2, names=['unacceptable', 'acceptable'], names_file=None, id=None), 'notated': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'src_code': Value(dtype='string', id=None)}
````

But after the `builder_instance.as_dataset` is called for Csv dataset builder, the `features` is changed to:
```
{'label': Value(dtype='int64', id=None), 'notated': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'src_code': Value(dtype='string', id=None)}
```

After digged into the code, I releazed that in `ArrowBasedBuilder._prepare_split`, the DatasetBuilder's info's features will be overwrited by `ArrowWriter`'s `_features`. 
But `ArrowWriter` is initailized without passing `features`.
So my concern is:
It's this overwrite must be done, or, should it be an option to pass features in `_prepare_split` function?
added file_permission in load_dataset
Author: bhavitvyamalik | Pull request: yes | State: closed

As discussed in #2065 I've added `file_permission` argument in `load_dataset`. 

Added mainly 2 things here:
1) Permission of downloaded datasets when converted to .arrow files  can be changed with argument `file_permission` argument in `load_dataset` (default is 0o644 only)
2) Incase the user uses `map` later on to generate another cache file of dataset, it ensures the permissions of newly generated file are similar to that of` *-train.arrow` file inside cache_dir for that dataset.
`load_dataset` caches two arrow files?
Author: hwijeen | Pull request: no | State: closed

Hi,

I am using datasets to load large json file of 587G.
I checked the cached folder and found that there are two arrow files created:
* `cache-ed205e500a7dc44c.arrow` - 355G
*  `json-train.arrow` - 582G

Why is the first file created?
If I delete it, would I still be able to `load_from_disk`?
KeyError: '_indices_files' in `arrow_dataset.py`
Author: samsontmr | Pull request: no | State: closed

After pulling the latest master, I'm getting a crash when `load_from_disk` tries to load my local dataset.

Trace:
```
Traceback (most recent call last):
  File "load_data.py", line 11, in <module>
    dataset = load_from_disk(SRC)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/load.py", line 784, in load_from_disk
    return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/dataset_dict.py", line 692, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 634, in load_from_disk
    if state["_indices_files"]:
KeyError: '_indices_files'
```

I believe this is the line causing the error since there may not be a `_indices_files` key in the older versions:
https://github.com/huggingface/datasets/blob/b70141e3c5149430951773aaa0155555c5fb3e76/src/datasets/arrow_dataset.py#L634

May I suggest using `state.get()` instead of directly indexing the dictionary?

@lhoestq
Filtering/mapping on one column is very slow
Author: norabelrose | Pull request: no | State: closed

I'm currently using the `wikipedia` dataset— I'm tokenizing the articles with the `tokenizers` library using `map()` and also adding a new `num_tokens` column to the dataset as part of that map operation.

I want to be able to _filter_ the dataset based on this `num_tokens` column, but even when I specify `input_columns=['num_tokens']`, it seems that the entirety of each row is loaded into memory, which makes the operation take much longer than it should. Indeed, `filter` currently just calls `map`, and I found that in `_map_single` on lines 1690-1704 of `arrow_dataset.py`, the method is just grabbing slices of _all the rows_ of the dataset and then passing only the specified columns to the map function. It seems that, when the user passes a value for `input_columns`, the `map` function should create a temporary pyarrow table by selecting just those columns, and then get slices from that table. Or something like that— I'm not very familiar with the pyarrow API.

I know that in the meantime I can sort of get around this by simply only returning the rows that match my filter criterion from the tokenizing function I pass to `map()`, but I actually _also_ want to map on just the `num_tokens` column in order to compute batches with a roughly uniform number of tokens per batch. I would also ideally like to be able to change my minimum and maximum article lengths without having to re-tokenize the entire dataset.

PS: This is definitely not a "dataset request." I'm realizing that I don't actually know how to remove labels from my own issues on other people's repos, if that is even possible.
save_to_disk doesn't work when we use concatenate_datasets function before creating the final dataset_object.
Author: shamanez | Pull request: no | State: closed

As you can see, it saves the entire dataset.

@lhoestq 

You can  check by going through the following example,

```
from datasets import load_from_disk,concatenate_datasets

loaded_data=load_from_disk('/home/gsir059/HNSW-ori/my_knowledge_dataset')
n=20
kb_list=[loaded_data.shard(n, i, contiguous=True) for i in range(n)]
final_dataset=concatenate_datasets([kb_list[1],kb_list[2]])
final_dataset.save_to_disk('/home/gsir059/haha/k.arrow')
```
Duplicate data in Timit dataset
Author: thanh-p | Pull request: no | State: closed

I ran a simple code to list all texts in Timit dataset and the texts were all the same.
Is this dataset corrupted?
**Code:**
timit = load_dataset("timit_asr")
print(*timit['train']['text'], sep='\n')
**Result:**
Would such an act of refusal be useful?
Would such an act of refusal be useful?
Would such an act of refusal be useful?
Would such an act of refusal be useful?
...
...
Would such an act of refusal be useful?
Question (potential issue?) related to datasets caching
Author: ioana-blue | Pull request: no | State: open

I thought I had disabled datasets caching in my code, as follows:
```
from datasets import set_caching_enabled
...
def main():

    # disable caching in datasets
    set_caching_enabled(False)
```
However, in my log files I see messages like the following:

```
04/07/2021 18:34:42 - WARNING - datasets.builder -   Using custom data configuration default-888a87931cbc5877
04/07/2021 18:34:42 - WARNING - datasets.builder -   Reusing dataset csv (xxxx/cache-transformers/datasets/csv/default-888a87931cbc5877/0.0.0/965b6429be0fc05f975b608ce64e1fa941cc8fb4f30629b523d2390f3c0e1a93
```
Can you please let me know what this reusing dataset csv means? I wouldn't expect any reusing with the datasets caching disabled. Thank you!
.map() and distributed training
Author: VictorSanh | Pull request: no | State: closed

Hi,
I have a question regarding distributed training and the `.map` call on a dataset.

I have a local dataset "my_custom_dataset" that I am loading with `datasets = load_from_disk(dataset_path=my_path)`.
`dataset` is then tokenized:
```python
datasets = load_from_disk(dataset_path=my_path)

[...]

def tokenize_function(examples):
    return tokenizer(examples[text_column_name])

logger.info("Mapping dataset to tokenized dataset.")
tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=preprocessing_num_workers,
    remove_columns=column_names,
    load_from_cache_file=True,
)
```
I am using 31 workers (`preprocessing_num_workers=31`) and thus it creates 31 `cache*.arrow` files in `my_path/train` (there is only a train split).
When I relaunch the script, the map is tokenization is skipped in favor of loading the 31 previously cached files, and that's perfect.

Everything so far was done by launching a **single process script**.
I now launch the same training script in **distributed mode** (`pytorch -m torch.distributed.launch --nproc_per_node 2`). However, once it reaches the map call, it re-does the tokenization... instead of loading the 31 cached files. 

I tried adding the `cache_file_name` argument: `cache_file_name={"train": my_path/one_of_the_arrow_file}`, but I can't give the 31 cached files, so it probably isn't the right way to do it.

**My question: what is the best way to load cached files if they were pre-processed and dumped in multiple arrow files?** It seems automatically handled for single processes but fails on distributed training.

- I am following the same structure as the examples of transformers (more specifically [run_clm.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) in my case)
- I am using 1.5.0 version of datasets if that matters.
Fix s3fs tests for py36 and py37+
Author: lhoestq | Pull request: yes | State: closed

Recently several changes happened:
1. latest versions of `fsspec` require python>3.7 for async features
2. `s3fs` added a dependency on `aiobotocore`, which is not compatible with the `moto` s3 mock context manager

This PR fixes both issues, by pinning `fsspec` and `s3fs` for python 3.6, and by using `moto` in server mode to support running the tests on python>=3.7 with the latest version of `fsspec` and `s3fs`.

cc @philschmid
Set default in-memory value depending on the dataset size
Author: albertvillanova | Pull request: yes | State: closed

Set a default value for `in_memory` depending on the size of the dataset to be loaded.

Close #2179.

TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~
Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)
Author: hwijeen | Pull request: no | State: closed

Hi, thanks for the great library. I have used the brilliant library for a couple of small projects, and now using it for a fairly big project.
When loading a huge json file of 500GB, pyarrow complains as follows:
```
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 531, in incomplete_dir
    yield tmp_dir
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 573, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 650, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 1027, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/app/.cache/huggingface/modules/datasets_modules/datasets/json/9498524fd296a6cca99c66d6c5be507d1c0991f5a814e535b507f4a66096a641/json.py", line 83, in _generate_tables
    parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 247, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```
When using only a small portion of the sample file, say first 100 lines, it works perfectly well..

I see that it is the error from pyarrow, but could you give me a hint or possible solutions?
#369 describes the same error and #372 claims to have fixed the issue, but I have no clue why I am still getting this one. Thanks in advance!
Load small datasets in-memory instead of using memory map
Author: lhoestq | Pull request: no | State: closed

Currently all datasets are loaded using memory mapping by default in `load_dataset`.
However this might not be necessary for small datasets. If a dataset is small enough, then it can be loaded in-memory and:
- its memory footprint would be small so it's ok
- in-memory computations/queries would be faster
- the caching on-disk would be disabled, making computations even faster (no I/O bound because of the disk)
- but running the same computation a second time would recompute everything since there would be no cached results on-disk. But this is probably fine since computations would be fast anyway + users should be able to provide a cache filename if needed.

Therefore, maybe the default behavior of `load_dataset` should be to load small datasets in-memory and big datasets using memory mapping.
Fix cast memory usage by using map on subtables
Author: lhoestq | Pull request: yes | State: closed

The `cast` operation on a pyarrow Table may create new arrays in memory.
This is an issue since users expect memory mapped datasets to not fill up the RAM.

To fix that I used `map` to write a new arrow file on disk when cast is used.
To make things more convenient I introduced the `arrow` formatting of a dataset, to make it return pyarrow tables instead of python dicts. This way one can use pyarrow transforms directly when using `map`.

edit: we'll use the same mechanism for `filter`
add social thumbnial
Author: philschmid | Pull request: yes | State: closed

# What does this PR do?

I added OpenGraph/ Twitter Card support to the docs to create nice social thumbnails.

![Bildschirmfoto 2021-04-07 um 08 36 50](https://user-images.githubusercontent.com/32632186/113821698-bac2ce80-977c-11eb-81aa-d8f16355857e.png)

To be able to add these I needed to install `sphinxext-opengraph`. I came across this [issue](https://github.com/readthedocs/readthedocs.org/issues/1758) on the readthedocs repo saying that since someone has built this plugin they are not integrating and providing documentation to it. That's why I added it for creating the documentation. The repository can be found [here](https://github.com/wpilibsuite/sphinxext-opengraph/tree/main).

P.S. It seemed that `make style` never ran for `docs/` i hope the changes are okay otherwise I'll revert it.
dataset.search_batch() function outputs all -1 indices sometime.
Author: shamanez | Pull request: no | State: closed

I am working with RAG and playing around with different faiss indexes. At the moment I use **index = faiss.index_factory(768, "IVF65536_HNSW32,Flat")**.

During the retrieval phase exactly in [this line of retrieval_rag.py](https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/retrieval_rag.py#L231) an error issue when all retrieved indices are -1.  Please refer to the screenshot of a PID worker. 

![image](https://user-images.githubusercontent.com/16892570/113782387-37a67600-9786-11eb-9c29-acad661a9648.png)


Here, my retrieve batch size is 2 and n_docs is 5. I can solve this by working around np. stack, but I want to ask, why we get an output index of -1. Do you have any idea :) ?

Is this a problem of the index, where the faiss can't find any similar vector?
Is there documentation on the output index being -1?

@lhoestq
Pin docutils for better doc
Author: sgugger | Pull request: yes | State: closed

The latest release of docutils make the navbar in the documentation weird and the Markdown wrongly interpreted:

![image](https://user-images.githubusercontent.com/35901082/113711773-5be55280-96b3-11eb-9b3b-9794f17709aa.png)

We had the same problem in Transformers and solved it by pinning docutils (a dep of sphinx).

You can see the version after the change [here](https://32769-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html).
Add OpenSLR dataset
Author: cahya-wirawan | Pull request: yes | State: closed

OpenSLR (https://openslr.org/) is a site devoted to hosting speech and language resources, such as training corpora for speech recognition, and software related to speech recognition. There are around 80 speech datasets listed in OpenSLR, currently this PR includes only 9 speech datasets SLR41, SLR42, SLR43, SLR44, SLR63, SLR64, SLR65, SLR66 and SLR69 (Javanese, Khmer, Nepali and Sundanese, Malayalam, Marathi, Tamil, Telugu and Catalan). I can add other speech datasets gradually next time.
Pin fsspec lower than 0.9.0
Author: lhoestq | Pull request: yes | State: closed

Today's release of `fsspec` 0.9.0 implied a new release of `s3fs` 0.6.0 but this version breaks the CI (see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/5312/workflows/490f3240-cd1c-4dd1-bb60-b416771c5584/jobs/32734) for example)

I'm pinning `fsspec` until this has been resolved
Preserve split type when realoding dataset
Author: mariosasko | Pull request: yes | State: closed

Fixes #2167 

Using `eval` is not ideal for security reasons (in web apps I assume), but without it the code would be much more complex IMO.

In terms of style, instead of explicitly importing a private member (`_RelativeInstruction`), we can add these imports at the top of the module:
```python
from . import arrow_reader  # gives us access to ReadInstruction and _RelativeInstruction
from . import splits # gives us access to NamedSplit
```

and then define the `eval` globals as follows:
```python
{**arrow_reader.__dict__, **splits.__dict__}
```
Split type not preserved when reloading the dataset
Author: mariosasko | Pull request: no | State: closed

A minimal reproducible example:
```python
>>> from datasets import load_dataset, Dataset
>>> dset = load_dataset("sst", split="train")
>>> dset.save_to_disk("sst")
>>> type(dset.split)
<class 'datasets.splits.NamedSplit'>
>>> dset = Dataset.load_from_disk("sst")
>>> type(dset.split)  # NamedSplit expected
<class 'str'>
```

It seems like this bug was introduced in #2025.
Regarding Test Sets for the GEM datasets
Author: vyraun | Pull request: no | State: closed

@yjernite Hi, are the test sets for the GEM datasets scheduled to be [added soon](https://gem-benchmark.com/shared_task)? 

e.g.

```
from datasets import load_dataset
DATASET_NAME="common_gen"
data = load_dataset("gem", DATASET_NAME)
```

The test set doesn't have the target or references.

```
data['test'][0]
{'concept_set_id': 0, 'concepts': ['drill', 'field', 'run', 'team'], 'gem_id': 'common_gen-test-0', 'gem_parent_id': 'common_gen-test-0', 'references': [], 'target': ''}
```
How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset
Author: y-rokutan | Pull request: no | State: closed

Hi,

I'm trying to pretraine deep-speed model using HF arxiv dataset like:
```
train_ds = nlp.load_dataset('scientific_papers', 'arxiv')
train_ds.set_format(
        type="torch",
        columns=["input_ids", "attention_mask", "global_attention_mask", "labels"],
    )
engine, _, _, _ = deepspeed.initialize(
    args=args,
    model=model,
    model_parameters=[p for p in model.parameters() if p.requires_grad],
    training_data=train_ds)
```
but deepspeed.initialize accepts torch.utils.data.Dataset only. How can I convert HF-style dataset to torch-style dataset?
data_args.preprocessing_num_workers almost freezes 
Author: dorost1234 | Pull request: no | State: closed

Hi @lhoestq 

I am running this code from huggingface transformers https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py 

to speed up tokenization, since I am running on multiple datasets, I am using data_args.preprocessing_num_workers = 4 with opus100 corpus but this moves on till a point and then this freezes almost for sometime during  tokenization steps and then this is back again, overall to me taking more time than normal case, I appreciate your advice on how I can use this option properly to speed up.

thanks
adding ccnet dataset
Author: dorost1234 | Pull request: no | State: closed

## Adding a Dataset
- **Name:**  ccnet

- **Description:** 
Common Crawl

- **Paper:** 
https://arxiv.org/abs/1911.00359

- **Data:** 
https://github.com/facebookresearch/cc_net

- **Motivation:**
this is one of the most comprehensive clean monolingual datasets across a variety of languages. Quite important for cross-lingual reseach

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

thanks
viewer "fake_news_english" error
Author: emanuelevivoli | Pull request: no | State: closed

When I visit the [Huggingface - viewer](https://huggingface.co/datasets/viewer/) web site, under the dataset "fake_news_english" I've got this error:

> ImportError: To be able to use this dataset, you need to install the following dependencies['openpyxl'] using 'pip install # noqa: requires this pandas optional dependency for reading xlsx files' for instance'

as well as the error Traceback.
Adding the NorNE dataset for Norwegian POS and NER
Author: versae | Pull request: yes | State: closed

NorNE is a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names.

See #1720.
load_dataset ignoring features
Author: GuillemGSubies | Pull request: no | State: closed

First of all, I'm sorry if it is a repeated issue or the changes are already in master, I searched and I didn't find anything. 

I'm using datasets 1.5.0

![image](https://user-images.githubusercontent.com/37592763/113114369-8f376580-920b-11eb-900d-94365b59f04b.png)

As you can see, when I load the dataset, the ClassLabels are ignored, I have to cast the dataset in order to make it work.

Code to reproduce:

```python
import datasets
data_location = "/data/prueba_multiclase"
features = datasets.Features(
        {"texto": datasets.Value("string"), "label": datasets.features.ClassLabel(names=["false", "true"])}
    )
dataset = datasets.load_dataset(
        "csv", data_files=data_location, delimiter="\t", features=features
    )
```

Dataset I used:


[prueba_multiclase.zip](https://github.com/huggingface/datasets/files/6235022/prueba_multiclase.zip) (it has to be unzipped)


Thank you! ❤️