NyssaW/github-issues huggingface dataset:
Batched map fails when removing all columns

Hi @lhoestq ,

I'm hijacking this issue, because I'm currently trying to do the approach you recommend:

> Currently the optimal setup for single-column computations is probably to do something like
> 
> ```python
> result = dataset.map(f, input_columns="my_col", remove_columns=dataset.column_names)
> ```

Here is my code: (see edit, in which I added a simplified version

```
This is the error:
```bash
pyarrow.lib.ArrowInvalid: Column 1 named tokens expected length 8964 but got length 1000
```
I wonder why this error occurs, when I delete every column? Can you give me a hint?

### Edit:
I preprocessed my dataset before (using map with the features argument) and saved it to disk. May this be part of the error?  I can iterate over the
complete dataset and print every sample before calling map. There seems to be no other problem with the dataset.

I tried to simplify the code that crashes:

```python
# works
log.debug(dataset.column_names)
log.debug(dataset)
for i, sample in enumerate(dataset):
    log.debug(i, sample)

# crashes
counted_dataset = dataset.map(
    lambda x: {"a": list(range(20))},
    input_columns=column,
    remove_columns=dataset.column_names,
    load_from_cache_file=False,
    num_proc=num_workers,
    batched=True,
)
```

```
pyarrow.lib.ArrowInvalid: Column 1 named tokens expected length 20 but got length 1000
```

Edit2: 

May this be a problem with a schema I set when preprocessing the dataset before? I tried to add the `features` argument to the function and then I get a new error:

```python
# crashes
counted_dataset = dataset.map(
    lambda x: {"a": list(range(20))},
    input_columns=column,
    remove_columns=dataset.column_names,
    load_from_cache_file=False,
    num_proc=num_workers,
    batched=True,
    features=datasets.Features(
        {
              "a": datasets.Sequence(datasets.Value("int32"))
         }
    )
)
```

```
 File "env/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1704, in _map_single
    writer.write_batch(batch)
  File "env/lib/python3.8/site-packages/datasets/arrow_writer.py", line 312, in write_batch
    col_type = schema.field(col).type if schema is not None else None
  File "pyarrow/types.pxi", line 1341, in pyarrow.lib.Schema.field
KeyError: 'Column tokens does not exist in schema'
```

_Originally posted by @villmow in https://github.com/huggingface/datasets/issues/2193#issuecomment-820230874_

Comments:
I found the problem. I called `set_format` on some columns before. This makes it crash. Here is a complete example to reproduce:

```python
from datasets import load_dataset
sst = load_dataset("sst")
sst.set_format("torch", columns=["label"], output_all_columns=True)
ds = sst["train"]

# crashes
ds.map(
    lambda x: {"a": list(range(20))},
    remove_columns=ds.column_names,
    load_from_cache_file=False,
    num_proc=1,
    batched=True,
)
```

Thanks for reporting and for providing this code to reproduce the issue, this is really helpful !

I merged a fix, it should work on `master` now :)
We'll do a new release soon !
fixed one instance of 'train' to 'test'

I believe this should be 'test' instead of 'train'

Comments:
Thanks ! good catch

Could you also update the metadata of this dataset ?
You can do so by running
```
datasets-cli test ./datasets/newsgroup --all_configs --save_infos --ignore_verifications
```
This should update the dataset_infos.json file that contains the size of all the splits for example.

Hi,
`dataset_infos.json` should be updated now.
Set test cache config

Currently, running the tests populates the default cache directory `"~/.cache"`.

This PR monkey-patches the config to set the cache directory within the temporary test directory, avoiding side effects.

Comments:
>  why a cache dir per test function does not work?

Probably because we end up with multiple `datasets_module` in the python path. This breaks the import of all the datasets/metrics modules.
If you want to use one modules cache per test, you may need remove the `datasets_module` that was added to the python path during the test.
Indeed if the module cache hasn't been initialized, then it's added to the python path by calling `init_dynamic_modules`:

https://github.com/huggingface/datasets/blob/ba76012a19193a35053b9e20243ff40c2b4204ab/src/datasets/load.py#L291-L291

@lhoestq, for the moment, this PR avoids populating the `~/.cache` dir during training, which is already an improvement, isn't it?

Yes we can merge it this way if you're fine with it !
This is a good improvement

I will eventually try to implement a `cache_dir` per test function in another PR, but I think I should first fix some side effects in tests: each test function should be atomic and able to have its own `cache_dir` without being affected by the `cache_dir` set in other test functions.

Yes this would be ideal !
Fix too long WindowsFileLock name

Fix WindowsFileLock name longer than allowed MAX_PATH by shortening the basename.

Comments:
Windows users should disable the max path length limit. It's a nightmare to handle it.
Also the lock path must not be changed in a random way. Otherwise from another process the lock path might not be the same and the locking mechanism won't work.

Do you agree with handling the case where MAX_PATH is not disabled? If not, we can close this PR.

If so, would it work a deterministic lock path instead of random?

I'd rather not handle this at all, since there will be other places in the code where the limit will break things
Add SLR70 - SLR80 and SLR86 to OpenSLR dataset

I would like to add SLR70, SLR71, SLR72, SLR73, SLR74, SLR75, SLR76, SLR77, SLR78, SLR79, SLR80 and SLR86 to OpenSLR dataset. The languages are:
Nigerian English, Chilean Spanish, Columbian Spanish, Peruvian Spanish, Puerto Rico Spanish, Venezuelan Spanish, Basque, Galician, Gujarati and Kannada.

Comments:
Fix infinite loop in WindowsFileLock

Raise exception to avoid infinite loop.

Comments:
How is it possible to get an infinite loop ? Can you add more details ?

Yes, in Windows, if the filename is too long, a `FileNotFoundError` is raised. The exception should be raised in this case. Otherwise, we get into an infinite loop.

If other process has the file locked, then `PermissionError` is raised. In this case, `pass` is OK.

Note that the filelock module comes from this project that hasn't changed in years - while still being used by ten of thousands of projects:
https://github.com/benediktschmitt/py-filelock

Unless we have proper tests for this, I wouldn't recommend to change it

I'm pretty sure many things from the library could break for windows users that haven't disabled the max path length limit.
Maybe it would be simpler to simply raise an error on startup. For exampe, for windows users the error could ask them to disable the limit if it's not been disabled yet ?
Added CUAD dataset

Dataset link : https://github.com/TheAtticusProject/cuad/

Working on README.md currently.

Closes #2084 and [#1](https://github.com/TheAtticusProject/cuad/issues/1). 

Comments:
1) Changed the language in a few places apart from those you mentioned in README
2) Reduced the size of dummy data folder by removing all other entries except the first
3) Updated YAML tags by using to the past version of `datasets-tagging` app. Will update the quick fix on that repository too in a while

@bhavitvyamalik  Thanks for adding the dataset on huggingface! Can you please add a metric also for the dataset using the squad_v2 metric file? 

@MohammedRakib you can check [#2257](https://github.com/huggingface/datasets/pull/2257)
Revert breaking change in cache_files property

#2025 changed the format of `Dataset.cache_files`.
Before it was formatted like
```python
[{"filename": "path/to/file.arrow", "start": 0, "end": 1337}]
```
and it was changed to
```python
["path/to/file.arrow"]
```
since there's no start/end offsets available anymore.

To make this less breaking, I'm setting the format back to a list of dicts:
```python
[{"filename": "path/to/file.arrow"}]
```

Comments:
Add datasets SLR35 and SLR36 to OpenSLR 

I would like to add [SLR35](https://openslr.org/35/) (18GB) and [SLR36](https://openslr.org/36/)  (22GB) which are Large Javanese and Sundanese ASR training data set collected by Google in collaboration with Reykjavik University and Universitas Gadjah Mada in Indonesia.

Comments:
Hi @lhoestq,
Could you please help me, I got this error message in all "ci/circleci: run_dataset_script_tests_pyarrow*" tests:
```
...
    """Wrapper classes for various types of tokenization."""
    
    from bleurt.lib import bert_tokenization
    import tensorflow.compat.v1 as tf
>   import sentencepiece as spm
E   ModuleNotFoundError: No module named 'sentencepiece'
...
```
I am not sure why I do get it. Thanks.


Hi ! This issue appeared on master since the last update of `BLEURT`.
I'm working on a fix. You can ignore this issue for this PR

> Hi ! This issue appeared on master since the last update of `BLEURT`.
> I'm working on a fix. You can ignore this issue for this PR

Thanks for the info

Merging since the CI is fixed on master
load_metric error: module 'datasets.utils.file_utils' has no attribute 'add_start_docstrings'

I'm having the same problem as [Notebooks issue 10](https://github.com/huggingface/notebooks/issues/10) on datasets 1.2.1, and it seems to be an issue with the datasets package.

```python
>>> from datasets import load_metric
>>> metric = load_metric("glue", "sst2")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/ext3/miniconda3/lib/python3.8/site-packages/datasets-1.2.1-py3.8.egg/datasets/load.py", line 502, in load_metric
  File "/ext3/miniconda3/lib/python3.8/site-packages/datasets-1.2.1-py3.8.egg/datasets/load.py", line 66, in import_main_class
  File "/ext3/miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ns4008/.cache/huggingface/modules/datasets_modules/metrics/glue/e4606ab9804a36bcd5a9cebb2cb65bb14b6ac78ee9e6d5981fa679a495dd55de/glue.py", line 105, in <module>
    @datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
AttributeError: module 'datasets.utils.file_utils' has no attribute 'add_start_docstrings'
```

Comments:
Hi @nsaphra, thanks for reporting.

This issue was fixed in `datasets` version 1.3.0. Could you please update `datasets` and tell me if the problem persists?
```shell
pip install -U datasets
```

There might be a bug in the conda version of `datasets` 1.2.1 where the datasets/metric scripts are downloaded from `master` instead of the `1.2.1` repo.

You can try setting the env var `HF_SCRIPTS_VERSION="1.2.1"` as a workaround. Let me know if that helps.

I just faced the same issue. I was using 1.2.1 from conda and received the same AttributeError complaining about 'add_start_docstrings'. Uninstalling the conda installed datasets and then installing the latest datasets (version 1.5.0) using pip install solved the issue for me. I don't like mixing up conda and pip installs in the same environments but this will have to do for now, until 1.5.0 is made available through conda.

Yep, seems to have fixed things! The conda package could really do with an update. Thanks!
Can't reach "https://storage.googleapis.com/illuin/fquad/train.json.zip" when trying to load fquad dataset

I'm trying to load the [fquad dataset](https://huggingface.co/datasets/fquad) by running: 

```Python
fquad = load_dataset("fquad")
```

which produces the following error:

```
Using custom data configuration default

Downloading and preparing dataset fquad/default (download: 3.14 MiB, generated: 6.62 MiB, post-processed: Unknown size, total: 9.76 MiB) to /root/.cache/huggingface/datasets/fquad/default/0.1.0/778dc2c85813d05ddd0c17087294d5f8f24820752340958070876b677af9f061...

---------------------------------------------------------------------------

ConnectionError                           Traceback (most recent call last)

<ipython-input-48-a2721797e23b> in <module>()
----> 1 fquad = load_dataset("fquad")

11 frames

/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    614             raise FileNotFoundError("Couldn't find file at {}".format(url))
    615         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
--> 616         raise ConnectionError("Couldn't reach {}".format(url))
    617 
    618     # Try a second time

ConnectionError: Couldn't reach https://storage.googleapis.com/illuin/fquad/train.json.zip
```

Does anyone know why that is and how to fix it? 

Comments:
Hi ! Apparently the data are not available from this url anymore. We'll replace it with the new url when it's available

I saw this on their website when we request to download the dataset:
![image](https://user-images.githubusercontent.com/19718818/114879600-fa458680-9e1e-11eb-9e05-f0963d68ff0f.png)

Can we still request them link for the dataset and make a PR? @lhoestq @yjernite 

I've contacted Martin (first author of the fquad paper) regarding a possible new url. Hopefully we can get one soon !

They now made a website to force people who want to use the dataset for commercial purposes to seek a commercial license from them ...

The script has been adopted to support manual download from the website, so I'm closing this issue.
Getting checksum error when trying to load lc_quad dataset

I'm having issues loading the [lc_quad](https://huggingface.co/datasets/fquad) dataset by running:

```Python
lc_quad = load_dataset("lc_quad")
```

which is giving me the following error:

``` 
Using custom data configuration default

Downloading and preparing dataset lc_quad/default (download: 3.69 MiB, generated: 19.77 MiB, post-processed: Unknown size, total: 23.46 MiB) to /root/.cache/huggingface/datasets/lc_quad/default/2.0.0/5a98fe174603f5dec6df07edf1c2b4d2317210d2ad61f5a393839bca4d64e5a7...

---------------------------------------------------------------------------

NonMatchingChecksumError                  Traceback (most recent call last)

<ipython-input-42-404ace83f73c> in <module>()
----> 1 lc_quad = load_dataset("lc_quad")

3 frames

/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/AskNowQA/LC-QuAD2.0/archive/master.zip']
```

Does anyone know why this could be and how I fix it? 

Comments:
Hi,

I've already opened a PR with the fix. If you are in a hurry, just build the project from source and run:
```bash
datasets-cli test datasets/lc_quad --save_infos --all_configs --ignore_verifications
```



Ah sorry, I tried searching but couldn't find any related PR. 

Thank you!
making labels consistent across the datasets

Hi
For accessing the labels one can type 
```
>>> a.features['label']
ClassLabel(num_classes=3, names=['entailment', 'neutral', 'contradiction'], names_file=None, id=None)
```
The labels however are not consistent with the actual labels sometimes, for instance in case of XNLI, the actual labels are 0,1,2, but if one try to access as above they are entailment, neutral,contradiction,
it would be great to have the labels consistent.

thanks 


Comments:
Hi ! The ClassLabel feature type encodes the labels as integers.
The integer corresponds to the index of the label name in the `names` list of the ClassLabel.
Here that means that the labels are 'entailment' (0), 'neutral' (1), 'contradiction' (2).

You can get the label names back by using `a.features['label'].int2str(i)`.


Hi! You can also easily reorder the label with the [`Dataset.align_labels_with_mapping`](https://huggingface.co/docs/datasets/master/en/process#align) method.
Fix ArrowWriter overwriting features in ArrowBasedBuilder

This should fix the issues with CSV loading experienced in #2153 and #2200.

The CSV builder is an ArrowBasedBuilder that had an issue with its ArrowWriter used to write the arrow file from the csv data.
The writer wasn't initialized with the features passed by the user. Therefore the writer was inferring the features from the arrow data, discarding the features passed by the user.

I fixed that and I updated the tests

Comments:
_prepare_split will overwrite DatasetBuilder.info.features

Hi, here is my issue:
I initialized a Csv datasetbuilder with specific features:
```
def get_dataset_features(data_args):
    features = {}
    if data_args.text_features:
        features.update({text_feature: hf_features.Value("string") for text_feature in data_args.text_features.strip().split(",")})
    if data_args.num_features:
        features.update({text_feature: hf_features.Value("float32") for text_feature in data_args.num_features.strip().split(",")})
    if data_args.label_classes:
        features["label"] = hf_features.ClassLabel(names=data_args.label_classes.strip().split(","))
    else:
        features["label"] = hf_features.Value("float32")
    return hf_features.Features(features)

datasets = load_dataset(extension,
                                data_files=data_files,
                                sep=data_args.delimiter,
                                header=data_args.header,
                                column_names=data_args.column_names.split(",") if data_args.column_names else None,
                                features=get_dataset_features(data_args=data_args))
```
The `features` is printout as below before `builder_instance.as_dataset` is called:
```
{'label': ClassLabel(num_classes=2, names=['unacceptable', 'acceptable'], names_file=None, id=None), 'notated': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'src_code': Value(dtype='string', id=None)}
````

But after the `builder_instance.as_dataset` is called for Csv dataset builder, the `features` is changed to:
```
{'label': Value(dtype='int64', id=None), 'notated': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'src_code': Value(dtype='string', id=None)}
```

After digged into the code, I releazed that in `ArrowBasedBuilder._prepare_split`, the DatasetBuilder's info's features will be overwrited by `ArrowWriter`'s `_features`. 
But `ArrowWriter` is initailized without passing `features`.
So my concern is:
It's this overwrite must be done, or, should it be an option to pass features in `_prepare_split` function?

Comments:
Hi ! This might be related to #2153 

You're right the ArrowWriter should be initialized with `features=self.info.features` ! Good catch
I'm opening a PR to fix this and also to figure out how it was not caught in the tests

EDIT: opened #2201

> Hi ! This might be related to #2153
> 
> You're right the ArrowWriter should be initialized with `features=self.info.features` ! Good catch
> I'm opening a PR to fix this and also to figure out how it was not caught in the tests
> 
> EDIT: opened #2201

Glad to hear that! Thank you for your fix, I'm new to huggingface, it's a fantastic project 😁
Fix backward compatibility in Dataset.load_from_disk

Fix backward compatibility when loading from disk an old dataset saved to disk with indices using key "_indices_data_files".

Related to #2195.

Comments:
Hi @lhoestq, could you please check if this makes sense? Thanks.

What about using `_indices_data_files` field in save_to_disk instead of `_indices_files` ?
This way future datasets can also be reloaded from older versions of the lib

`_indices_files` was introduced in a recent PR and was not released

Yes, I have seen it is not released yet...

You are right! It was your awesome PR on Tables which renamed this. If there is no particular reason for this renaming, yes, we could switch it back to the previous `_indices_data_files`. ;)
added file_permission in load_dataset

As discussed in #2065 I've added `file_permission` argument in `load_dataset`. 

Added mainly 2 things here:
1) Permission of downloaded datasets when converted to .arrow files  can be changed with argument `file_permission` argument in `load_dataset` (default is 0o644 only)
2) Incase the user uses `map` later on to generate another cache file of dataset, it ensures the permissions of newly generated file are similar to that of` *-train.arrow` file inside cache_dir for that dataset.

Comments:
From offline discussions: we want to make the permissions handling consistent with `transformers`. However from discussion  in https://github.com/huggingface/transformers/pull/11119 it looks like it might not be a good solution to provide this argument. Users should use umask for now, and we'll see how things evolve.

@bhavitvyamalik I'm closing the PR for now if you don't mind
`load_dataset` caches two arrow files?

Hi,

I am using datasets to load large json file of 587G.
I checked the cached folder and found that there are two arrow files created:
* `cache-ed205e500a7dc44c.arrow` - 355G
*  `json-train.arrow` - 582G

Why is the first file created?
If I delete it, would I still be able to `load_from_disk`?

Comments:
Hi ! Files that starts with `cache-*` are cached computation files, i.e. they are the cached results of map/filter/cast/etc. operations. For example if you used `map` on your dataset to transform it, then the resulting dataset is going to be stored and cached in a `cache-*` file. These files are used to avoid having to load the dataset in RAM, even after many transforms

Thanks @lhoestq! Hmm.. that's strange because I specifically turned off auto caching, and saved mapped result, using `save_to_disk`, to another location. At this location, the following file is created:`355G	cache-ed205e500a7dc44c.arrow`

To my observation, both `load_dataset` and `map` creates `cache-*` files, and I wonder what the `cache-*` file from `load_dataset` is for (as I believe the same information is stored in `json-train.arrow`.

This is a wrong report -- `cache-*` files are created only my `map`, not by `load_dataset`.
KeyError: '_indices_files' in `arrow_dataset.py`

After pulling the latest master, I'm getting a crash when `load_from_disk` tries to load my local dataset.

Trace:
```
Traceback (most recent call last):
  File "load_data.py", line 11, in <module>
    dataset = load_from_disk(SRC)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/load.py", line 784, in load_from_disk
    return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/dataset_dict.py", line 692, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 634, in load_from_disk
    if state["_indices_files"]:
KeyError: '_indices_files'
```

I believe this is the line causing the error since there may not be a `_indices_files` key in the older versions:
https://github.com/huggingface/datasets/blob/b70141e3c5149430951773aaa0155555c5fb3e76/src/datasets/arrow_dataset.py#L634

May I suggest using `state.get()` instead of directly indexing the dictionary?

@lhoestq 

Comments:
Thanks for reporting @samsontmr.

It seems a backward compatibility issue...

Thanks @samsontmr this should be fixed on master now

Feel free to reopen if you're still having issues
Refactorize tests to use Dataset as context manager

Refactorize Dataset tests to use Dataset as context manager.

Comments:
I find very interesting that idea of using a fixture instead!

Let me rework a little bit this PR, @lhoestq.

@lhoestq, as this is a big refactoring, I had many problems to solve the conflicts with the master branch...

Therefore, I think it is better to merge this as it is, and then to make other PRs with additional refactorings, before I get conflicts again with the master branch...

There are still some conflicts that prevent merging.
Moreover I noticed that you added one fixture per method of the Dataset object to be mocked. The code of all these fixtures is pretty much the same, feel free to factorize them into one fixture.

Also feel free to create another branch from `master` if you don't want to fix the conflicts of this branch.
Let me know if I can help you on this

@lhoestq, yes, the new conflicts appeared after today merge commits on master...

I am definitely going to split this PR into smaller ones in order to avoid having to resolve many conflicts after each commit on master. There are lots of conflicts and these are painful to resolve.
News_commentary Dataset Translation Pairs are of Incorrect Language Specified Pairs

I used load_dataset to load the news_commentary dataset for "ar-en" translation pairs but found translations from Arabic to Hindi.  

```
train_ds = load_dataset("news_commentary", "ar-en", split='train[:98%]')
val_ds = load_dataset("news_commentary", "ar-en", split='train[98%:]')

# filtering out examples that are not ar-en translations but ar-hi
val_ds = val_ds.filter(lambda example, indice: indice not in chain(range(1312,1327) ,range(1384,1399), range(1030,1042)), with_indices=True)
```

* I'm fairly new to using datasets so I might be doing something wrong

Comments:
Hi @anassalamah,

Could you please try with this:
```python
train_ds = load_dataset("news_commentary", lang1="ar", lang2="en", split='train[:98%]')
val_ds = load_dataset("news_commentary", lang1="ar", lang2="en", split='train[98%:]')
```

Hello @albertvillanova, 

Thanks for the suggestion. I didn't know you could do that. however, it didn't resolve the issue

![image](https://user-images.githubusercontent.com/8571003/114169966-ec819400-993a-11eb-8a67-930f9a9b2290.png)
save_to_disk doesn't work when we use concatenate_datasets function before creating the final dataset_object.

As you can see, it saves the entire dataset.

@lhoestq 

You can  check by going through the following example,

```
from datasets import load_from_disk,concatenate_datasets

loaded_data=load_from_disk('/home/gsir059/HNSW-ori/my_knowledge_dataset')
n=20
kb_list=[loaded_data.shard(n, i, contiguous=True) for i in range(n)]
final_dataset=concatenate_datasets([kb_list[1],kb_list[2]])
final_dataset.save_to_disk('/home/gsir059/haha/k.arrow')
```

Comments:
Hi ! We refactored save_to_disk in #2025 so this doesn't happen.
Feel free to try it on master for now
We'll do a new release soon
Duplicate data in Timit dataset

I ran a simple code to list all texts in Timit dataset and the texts were all the same.
Is this dataset corrupted?
**Code:**
timit = load_dataset("timit_asr")
print(*timit['train']['text'], sep='\n')
**Result:**
Would such an act of refusal be useful?
Would such an act of refusal be useful?
Would such an act of refusal be useful?
Would such an act of refusal be useful?
...
...
Would such an act of refusal be useful?

Comments:
Hi ! Thanks for reporting
If I recall correctly this has been recently fixed #1995
Can you try to upgrade your local version of `datasets` ?
```
pip install --upgrade datasets
```

Hi Ihoestq,

Thank you. It works after upgrading the datasets
Fix s3fs tests for py36 and py37+

Recently several changes happened:
1. latest versions of `fsspec` require python>3.7 for async features
2. `s3fs` added a dependency on `aiobotocore`, which is not compatible with the `moto` s3 mock context manager

This PR fixes both issues, by pinning `fsspec` and `s3fs` for python 3.6, and by using `moto` in server mode to support running the tests on python>=3.7 with the latest version of `fsspec` and `s3fs`.

cc @philschmid 

Comments:
Set default in-memory value depending on the dataset size

Set a default value for `in_memory` depending on the size of the dataset to be loaded.

Close #2179.

TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~

Comments:
I ping @krandiash to keep him up to date.

TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~

@lhoestq I have a question, regarding:
> Also maybe we should add a warning if someone tries to specify cache_file_name= in map, filter etc. on a dataset that is in memory, since the computation is not going to be cached in this case.

- It might be the case that the user has an in-memory dataset and might want to use `map` and cache it, by passing `cache_file_name=`
- This is indeed allowed by the library and works as expected: the dataset is cached.

Why adding a warning?

Yes right, I meant if `load_from_cache_file` is set to True and `cache_file_name ` is None. my bad :p
Load small datasets in-memory instead of using memory map

Currently all datasets are loaded using memory mapping by default in `load_dataset`.
However this might not be necessary for small datasets. If a dataset is small enough, then it can be loaded in-memory and:
- its memory footprint would be small so it's ok
- in-memory computations/queries would be faster
- the caching on-disk would be disabled, making computations even faster (no I/O bound because of the disk)
- but running the same computation a second time would recompute everything since there would be no cached results on-disk. But this is probably fine since computations would be fast anyway + users should be able to provide a cache filename if needed.

Therefore, maybe the default behavior of `load_dataset` should be to load small datasets in-memory and big datasets using memory mapping.

Comments:
Fix cast memory usage by using map on subtables

The `cast` operation on a pyarrow Table may create new arrays in memory.
This is an issue since users expect memory mapped datasets to not fill up the RAM.

To fix that I used `map` to write a new arrow file on disk when cast is used.
To make things more convenient I introduced the `arrow` formatting of a dataset, to make it return pyarrow tables instead of python dicts. This way one can use pyarrow transforms directly when using `map`.

edit: we'll use the same mechanism for `filter`

Comments:
I addressed your comments about the docstrings and the output validation :)

I updated the bleurt mocking method and bleurt test is passing now.
I also ran the slow tests and they are passing for bleurt.

Thanks @lhoestq and @albertvillanova !
add social thumbnial

# What does this PR do?

I added OpenGraph/ Twitter Card support to the docs to create nice social thumbnails.

![Bildschirmfoto 2021-04-07 um 08 36 50](https://user-images.githubusercontent.com/32632186/113821698-bac2ce80-977c-11eb-81aa-d8f16355857e.png)

To be able to add these I needed to install `sphinxext-opengraph`. I came across this [issue](https://github.com/readthedocs/readthedocs.org/issues/1758) on the readthedocs repo saying that since someone has built this plugin they are not integrating and providing documentation to it. That's why I added it for creating the documentation. The repository can be found [here](https://github.com/wpilibsuite/sphinxext-opengraph/tree/main).

P.S. It seemed that `make style` never ran for `docs/` i hope the changes are okay otherwise I'll revert it. 

Comments:
Converting a Value to a ClassLabel

Hi!

In the docs for `cast`, it's noted that `For non-trivial conversion, e.g. string <-> ClassLabel you should use map() to update the Dataset.`

Would it be possible to have an example  that demonstrates such a string <-> ClassLabel conversion using `map`? Thanks!

Comments:
Hi @nelson-liu!
Here is what I do to convert a string to class label:

```python
from datasets import load_dataset, features


dset = load_dataset(...)
col_name = "the string column name"

class_names  = dset.unique(col_name)
class_feature = features.ClassLabel(names=sorted(class_names))
dset = dset.map(lambda str_value: {col_name: class_feature.str2int(str_value)}, input_columns=col_name)

dset = dset.cast(features.Features({
    ...
    col_name: class_feature
})
```


Hi! You can use `Dataset.class_encode_column` for this. And in the next release of `datasets` (this feature is only available on `master`), you'll also be able to use `cast` to do the conversion. 

An example of conversion via `cast`: 
```python
from datasets import Dataset, Features, ClassLabel
d = Dataset.from_dict({"a": ["no", "yes", "no"]})
d = d.cast(Features({"a": ClassLabel(names=["yes", "no"])}))
```
Pin docutils for better doc

The latest release of docutils make the navbar in the documentation weird and the Markdown wrongly interpreted:

![image](https://user-images.githubusercontent.com/35901082/113711773-5be55280-96b3-11eb-9b3b-9794f17709aa.png)

We had the same problem in Transformers and solved it by pinning docutils (a dep of sphinx).

You can see the version after the change [here](https://32769-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html).


Comments:
Add OpenSLR dataset

OpenSLR (https://openslr.org/) is a site devoted to hosting speech and language resources, such as training corpora for speech recognition, and software related to speech recognition. There are around 80 speech datasets listed in OpenSLR, currently this PR includes only 9 speech datasets SLR41, SLR42, SLR43, SLR44, SLR63, SLR64, SLR65, SLR66 and SLR69 (Javanese, Khmer, Nepali and Sundanese, Malayalam, Marathi, Tamil, Telugu and Catalan). I can add other speech datasets gradually next time.

Comments:
Pin fsspec lower than 0.9.0

Today's release of `fsspec` 0.9.0 implied a new release of `s3fs` 0.6.0 but this version breaks the CI (see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/5312/workflows/490f3240-cd1c-4dd1-bb60-b416771c5584/jobs/32734) for example)

I'm pinning `fsspec` until this has been resolved

Comments:
Split type not preserved when reloading the dataset

A minimal reproducible example:
```python
>>> from datasets import load_dataset, Dataset
>>> dset = load_dataset("sst", split="train")
>>> dset.save_to_disk("sst")
>>> type(dset.split)
<class 'datasets.splits.NamedSplit'>
>>> dset = Dataset.load_from_disk("sst")
>>> type(dset.split)  # NamedSplit expected
<class 'str'>
```

It seems like this bug was introduced in #2025.

Comments:
Regarding Test Sets for the GEM datasets

@yjernite Hi, are the test sets for the GEM datasets scheduled to be [added soon](https://gem-benchmark.com/shared_task)? 

e.g.

```
from datasets import load_dataset
DATASET_NAME="common_gen"
data = load_dataset("gem", DATASET_NAME)
```

The test set doesn't have the target or references.

```
data['test'][0]
{'concept_set_id': 0, 'concepts': ['drill', 'field', 'run', 'team'], 'gem_id': 'common_gen-test-0', 'gem_parent_id': 'common_gen-test-0', 'references': [], 'target': ''}
```



Comments:
Hi @vyraun ! The test references for CommonGen are not publicly available: you can reach out to the original dataset authors if you would like to ask for them, but we will not be releasing them as part of GEM (March 31st was the release date for the test set inputs, references are incidentally released for some of the test sets but shouldn't really be used for benchmark submissions)

cc @sebastiangehrmann

Oh okay, thanks @yjernite !
Concat only unique fields in DatasetInfo.from_merge

I thought someone from the community with less experience would be interested in fixing this issue, but that wasn't the case.

Fixes #2103 

Comments:
Hi @mariosasko,
Just came across this PR and I was wondering if we can use
`description = "\n\n".join(OrderedDict.fromkeys([info.description for info in dataset_infos]))`

This will obviate the need for `unique` and is almost as fast as `set`. We could have used `dict` inplace of `OrderedDict` but it's available 3.7+ onwards

Hi,

let's see what @lhoestq thinks. Although my approach adds more code, it's more readable IMO.

Yeah, that's true. Your approach is more readable.
visualization for cc100 is broken 

Hi
visualization through dataset viewer for cc100 is broken
https://huggingface.co/datasets/viewer/

thanks a lot


Comments:
This looks like an issue with the cc100 dataset itself but not sure
Did you try loading cc100 on your machine ?

Hi
loading works fine, but the viewer only is broken
thanks

On Wed, Apr 7, 2021 at 12:17 PM Quentin Lhoest ***@***.***>
wrote:

> This looks like an issue with the cc100 dataset itself but not sure
> Did you try loading cc100 on your machine ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2162#issuecomment-814793809>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMRUO33JSOYGT6RETWLTHQWNLANCNFSM42IUOR6Q>
> .
>


Hi! This visualization tool is deprecated now. The viewer at https://huggingface.co/datasets/cc100 works fine, so I'm closing this issue.
any possibility to download part of large datasets only?

Hi
Some of the datasets I need like cc100 are very large, and then I wonder if I can download first X samples of the shuffled/unshuffled data without going through first downloading the whole data then sampling? thanks

Comments:
Not yet but it’s on the short/mid-term roadmap (requested by many indeed).

oh, great, really awesome feature to have, thank you very much for the great, fabulous work

We'll work on dataset streaming soon. This should allow you to only load the examples you need ;)

thanks a lot Quentin, this would be really really a great feature to have

On Wed, Apr 7, 2021 at 12:14 PM Quentin Lhoest ***@***.***>
wrote:

> We'll work on dataset streaming soon. This should allow you to only load
> the examples you need ;)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2161#issuecomment-814791922>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMROD62QAKIJMAKWISTTHQWBVANCNFSM42IUI5JQ>
> .
>


Is streaming completed? On the 1.8.0 docs it is mentioned (https://huggingface.co/docs/datasets/dataset_streaming.html), but when following the example I get the following error:

```
>>> dataset2 = load_dataset("amazon_us_reviews", "Pet_Products_v1_00", split='train', streaming=True)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-1eedab26cff1> in <module>()
----> 1 en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)

3 frames
/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    339                 if value is not None:
    340                     if not hasattr(builder_config, key):
--> 341                         raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
    342                     setattr(builder_config, key, value)
    343 

ValueError: BuilderConfig OscarConfig(name='unshuffled_deduplicated_en', version=1.0.0, data_dir=None, data_files=None, description='Unshuffled and deduplicated, English OSCAR dataset') doesn't have a 'streaming' key.
```

UPDATE: Managed to get streaming working by building from source and installing the additional `datasets[streaming]` package:

```
!pip install git+https://github.com/huggingface/datasets.git
!pip install datasets[streaming]
```

Hi ! Streaming is available on `master` only right now. We'll make a new release 1.9.0 on Monday :)
data_args.preprocessing_num_workers almost freezes 

Hi @lhoestq 

I am running this code from huggingface transformers https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py 

to speed up tokenization, since I am running on multiple datasets, I am using data_args.preprocessing_num_workers = 4 with opus100 corpus but this moves on till a point and then this freezes almost for sometime during  tokenization steps and then this is back again, overall to me taking more time than normal case, I appreciate your advice on how I can use this option properly to speed up.

thanks

Comments:
Hi.
I cannot always reproduce this issue, and on later runs I did not see it so far. Sometimes also I set 8 processes but I see less being showed, is this normal, here only 5 are shown for 8 being set, thanks

```
#3:  11%|███████████████▊                                                                                                                                  | 172/1583 [00:46<06:21,  3.70ba/s]
#4:   9%|█████████████▏                                                                                                                                    | 143/1583 [00:46<07:46,  3.09ba/s]
#7:   6%|█████████                                                                                                                                          | 98/1583 [00:45<11:34,  2.14ba/s]
#5:   8%|███████████▍                                                                                                                                      | 124/1583 [00:46<09:03,  2.68ba/s]
#6:   7%|██████████▏  
```

closing since I cannot reproduce it again, thanks
adding ccnet dataset

## Adding a Dataset
- **Name:**  ccnet

- **Description:** 
Common Crawl

- **Paper:** 
https://arxiv.org/abs/1911.00359

- **Data:** 
https://github.com/facebookresearch/cc_net

- **Motivation:**
this is one of the most comprehensive clean monolingual datasets across a variety of languages. Quite important for cross-lingual reseach

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

thanks

Comments:
closing since I think this is cc100, just the name has been changed. thanks
viewer "fake_news_english" error

When I visit the [Huggingface - viewer](https://huggingface.co/datasets/viewer/) web site, under the dataset "fake_news_english" I've got this error:

> ImportError: To be able to use this dataset, you need to install the following dependencies['openpyxl'] using 'pip install # noqa: requires this pandas optional dependency for reading xlsx files' for instance'

as well as the error Traceback.



Comments:
Thanks for reporting !
The viewer doesn't have all the dependencies of the datasets. We may add openpyxl to be able to show this dataset properly

This viewer tool is deprecated now and the new viewer at https://huggingface.co/datasets/fake_news_english works fine, so I'm closing this issue
Adding the NorNE dataset for Norwegian POS and NER

NorNE is a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names.

See #1720.

Comments:
Awesome!
load_dataset ignoring features

First of all, I'm sorry if it is a repeated issue or the changes are already in master, I searched and I didn't find anything. 

I'm using datasets 1.5.0

![image](https://user-images.githubusercontent.com/37592763/113114369-8f376580-920b-11eb-900d-94365b59f04b.png)

As you can see, when I load the dataset, the ClassLabels are ignored, I have to cast the dataset in order to make it work.

Code to reproduce:

```python
import datasets
data_location = "/data/prueba_multiclase"
features = datasets.Features(
        {"texto": datasets.Value("string"), "label": datasets.features.ClassLabel(names=["false", "true"])}
    )
dataset = datasets.load_dataset(
        "csv", data_files=data_location, delimiter="\t", features=features
    )
```

Dataset I used:


[prueba_multiclase.zip](https://github.com/huggingface/datasets/files/6235022/prueba_multiclase.zip) (it has to be unzipped)


Thank you! ❤️ 


Comments:
Hi ! Thanks for reporting. I opened a PR to fix this issue: #2201

Nice question which helped me a lot! I have wasted a lot of time to the `DatasetDict` creation from a csv file. Hope the document of this module add some simple examples.

Hi :) We're indeed working on tutorials that we will add to the docs !
Add support for axis in concatenate datasets

Add support for `axis` (0 or 1) in `concatenate_datasets`.

Close #853.

Comments:
@lhoestq I am going to implement the consolidation step you mentioned in #1870.

@lhoestq I was thinking that the order of the TableBlocks is not relevant, isn't it?

I mean, in order to consolidate _consecutive_ in-memory table blocks, in this case:
```
blocks = [in_memory_1, memory_mapped, in_memory_2]
```
I could reorder the list:
```
blocks = [in_memory_1, in_memory_2, memory_mapped]
```
so that the first 2 can be consolidated into a single one:
```
blocks = [in_memory_3, memory_mapped]
```

I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item

> I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item

OK, therefore I leave `_consolidate_blocks` as it is, which currently keeps the order of the blocks (no shuffling).

Thank you guys for implementing this. Minor thing I noticed in the [documentation](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.concatenate_datasets): it says "Converts a list of Dataset with **the same schema** into a single Dataset". With the addition of the axis parameter, perhaps this should be reworded, no?
Telugu subset missing for xtreme tatoeba dataset

from nlp import load_dataset
train_dataset = load_dataset('xtreme', 'tatoeba.tel')['validation']
ValueError: BuilderConfig tatoeba.tel not found.

but language tel is actually included in xtreme:
https://github.com/google-research/xtreme/blob/master/utils_preprocess.py
def tatoeba_preprocess(args):
  lang3_dict = {
    'afr':'af', 'ara':'ar', 'bul':'bg', 'ben':'bn',
    'deu':'de', 'ell':'el', 'spa':'es', 'est':'et',
    'eus':'eu', 'pes':'fa', 'fin':'fi', 'fra':'fr',
    'heb':'he', 'hin':'hi', 'hun':'hu', 'ind':'id',
    'ita':'it', 'jpn':'ja', 'jav':'jv', 'kat':'ka',
    'kaz':'kk', 'kor':'ko', 'mal':'ml', 'mar':'mr',
    'nld':'nl', 'por':'pt', 'rus':'ru', 'swh':'sw',
    'tam':'ta', **_'tel':'te'_**, 'tha':'th', 'tgl':'tl', <----here
    'tur':'tr', 'urd':'ur', 'vie':'vi', 'cmn':'zh',
    'eng':'en',
  }

Comments:
Good catch ! Thanks for reporting

I just opened #2180 to fix this

Fixed in #2180