helmo/github-issues huggingface dataset:
Refactorize tests to use Dataset as context manager
State: closed | Author: albertvillanova (MEMBER)
Labels: name: refactoring

Refactorize Dataset tests to use Dataset as context manager.

Comments: I find very interesting that idea of using a fixture instead!

Let me rework a little bit this PR, @lhoestq.

@lhoestq, as this is a big refactoring, I had many problems to solve the conflicts with the master branch...

Therefore, I think it is better to merge this as it is, and then to make other PRs with additional refactorings, before I get conflicts again with the master branch...

There are still some conflicts that prevent merging.
Moreover I noticed that you added one fixture per method of the Dataset object to be mocked. The code of all these fixtures is pretty much the same, feel free to factorize them into one fixture.

Also feel free to create another branch from `master` if you don't want to fix the conflicts of this branch.
Let me know if I can help you on this

@lhoestq, yes, the new conflicts appeared after today merge commits on master...

I am definitely going to split this PR into smaller ones in order to avoid having to resolve many conflicts after each commit on master. There are lots of conflicts and these are painful to resolve.
News_commentary Dataset Translation Pairs are of Incorrect Language Specified Pairs
State: closed | Author: anassalamah (NONE)

I used load_dataset to load the news_commentary dataset for "ar-en" translation pairs but found translations from Arabic to Hindi.  

```
train_ds = load_dataset("news_commentary", "ar-en", split='train[:98%]')
val_ds = load_dataset("news_commentary", "ar-en", split='train[98%:]')

# filtering out examples that are not ar-en translations but ar-hi
val_ds = val_ds.filter(lambda example, indice: indice not in chain(range(1312,1327) ,range(1384,1399), range(1030,1042)), with_indices=True)
```

* I'm fairly new to using datasets so I might be doing something wrong

Comments: Hi @anassalamah,

Could you please try with this:
```python
train_ds = load_dataset("news_commentary", lang1="ar", lang2="en", split='train[:98%]')
val_ds = load_dataset("news_commentary", lang1="ar", lang2="en", split='train[98%:]')
```

Hello @albertvillanova, 

Thanks for the suggestion. I didn't know you could do that. however, it didn't resolve the issue

![image](https://user-images.githubusercontent.com/8571003/114169966-ec819400-993a-11eb-8a67-930f9a9b2290.png)
save_to_disk doesn't work when we use concatenate_datasets function before creating the final dataset_object.
State: closed | Author: shamanez (NONE)

As you can see, it saves the entire dataset.

@lhoestq 

You can  check by going through the following example,

```
from datasets import load_from_disk,concatenate_datasets

loaded_data=load_from_disk('/home/gsir059/HNSW-ori/my_knowledge_dataset')
n=20
kb_list=[loaded_data.shard(n, i, contiguous=True) for i in range(n)]
final_dataset=concatenate_datasets([kb_list[1],kb_list[2]])
final_dataset.save_to_disk('/home/gsir059/haha/k.arrow')
```

Comments: Hi ! We refactored save_to_disk in #2025 so this doesn't happen.
Feel free to try it on master for now
We'll do a new release soon
Duplicate data in Timit dataset
State: closed | Author: thanh-p (NONE)

I ran a simple code to list all texts in Timit dataset and the texts were all the same.
Is this dataset corrupted?
**Code:**
timit = load_dataset("timit_asr")
print(*timit['train']['text'], sep='\n')
**Result:**
Would such an act of refusal be useful?
Would such an act of refusal be useful?
Would such an act of refusal be useful?
Would such an act of refusal be useful?
...
...
Would such an act of refusal be useful?

Comments: Hi ! Thanks for reporting
If I recall correctly this has been recently fixed #1995
Can you try to upgrade your local version of `datasets` ?
```
pip install --upgrade datasets
```

Hi Ihoestq,

Thank you. It works after upgrading the datasets
GEM: new challenge sets
State: closed | Author: yjernite (MEMBER)

This PR updates the GEM dataset to:
- remove extraneous fields in WikiAuto after https://github.com/huggingface/datasets/pull/2171 fixed the source
- add context and services to Schema Guided Dialog
- Add new or update challenge sets for MLSUM ES and DE, XSUM, and SGD

Comments: cc @sebastiangehrmann
Fix s3fs tests for py36 and py37+
State: closed | Author: lhoestq (MEMBER)

Recently several changes happened:
1. latest versions of `fsspec` require python>3.7 for async features
2. `s3fs` added a dependency on `aiobotocore`, which is not compatible with the `moto` s3 mock context manager

This PR fixes both issues, by pinning `fsspec` and `s3fs` for python 3.6, and by using `moto` in server mode to support running the tests on python>=3.7 with the latest version of `fsspec` and `s3fs`.

cc @philschmid
Set default in-memory value depending on the dataset size
State: closed | Author: albertvillanova (MEMBER)
Labels: name: enhancement

Set a default value for `in_memory` depending on the size of the dataset to be loaded.

Close #2179.

TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~

Comments: I ping @krandiash to keep him up to date.

TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~

@lhoestq I have a question, regarding:
> Also maybe we should add a warning if someone tries to specify cache_file_name= in map, filter etc. on a dataset that is in memory, since the computation is not going to be cached in this case.

- It might be the case that the user has an in-memory dataset and might want to use `map` and cache it, by passing `cache_file_name=`
- This is indeed allowed by the library and works as expected: the dataset is cached.

Why adding a warning?

Yes right, I meant if `load_from_cache_file` is set to True and `cache_file_name ` is None. my bad :p
Load small datasets in-memory instead of using memory map
State: closed | Author: lhoestq (MEMBER)
Labels: name: enhancement

name: generic discussion

Currently all datasets are loaded using memory mapping by default in `load_dataset`.
However this might not be necessary for small datasets. If a dataset is small enough, then it can be loaded in-memory and:
- its memory footprint would be small so it's ok
- in-memory computations/queries would be faster
- the caching on-disk would be disabled, making computations even faster (no I/O bound because of the disk)
- but running the same computation a second time would recompute everything since there would be no cached results on-disk. But this is probably fine since computations would be fast anyway + users should be able to provide a cache filename if needed.

Therefore, maybe the default behavior of `load_dataset` should be to load small datasets in-memory and big datasets using memory mapping.
Fix cast memory usage by using map on subtables
State: closed | Author: lhoestq (MEMBER)
Labels: name: enhancement

The `cast` operation on a pyarrow Table may create new arrays in memory.
This is an issue since users expect memory mapped datasets to not fill up the RAM.

To fix that I used `map` to write a new arrow file on disk when cast is used.
To make things more convenient I introduced the `arrow` formatting of a dataset, to make it return pyarrow tables instead of python dicts. This way one can use pyarrow transforms directly when using `map`.

edit: we'll use the same mechanism for `filter`

Comments: I addressed your comments about the docstrings and the output validation :)

I updated the bleurt mocking method and bleurt test is passing now.
I also ran the slow tests and they are passing for bleurt.

Thanks @lhoestq and @albertvillanova !
add social thumbnial
State: closed | Author: philschmid (CONTRIBUTOR)

# What does this PR do?

I added OpenGraph/ Twitter Card support to the docs to create nice social thumbnails.

![Bildschirmfoto 2021-04-07 um 08 36 50](https://user-images.githubusercontent.com/32632186/113821698-bac2ce80-977c-11eb-81aa-d8f16355857e.png)

To be able to add these I needed to install `sphinxext-opengraph`. I came across this [issue](https://github.com/readthedocs/readthedocs.org/issues/1758) on the readthedocs repo saying that since someone has built this plugin they are not integrating and providing documentation to it. That's why I added it for creating the documentation. The repository can be found [here](https://github.com/wpilibsuite/sphinxext-opengraph/tree/main).

P.S. It seemed that `make style` never ran for `docs/` i hope the changes are okay otherwise I'll revert it.
Converting a Value to a ClassLabel
State: closed | Author: nelson-liu (NONE)
Labels: name: enhancement

Hi!

In the docs for `cast`, it's noted that `For non-trivial conversion, e.g. string <-> ClassLabel you should use map() to update the Dataset.`

Would it be possible to have an example  that demonstrates such a string <-> ClassLabel conversion using `map`? Thanks!

Comments: Hi @nelson-liu!
Here is what I do to convert a string to class label:

```python
from datasets import load_dataset, features


dset = load_dataset(...)
col_name = "the string column name"

class_names  = dset.unique(col_name)
class_feature = features.ClassLabel(names=sorted(class_names))
dset = dset.map(lambda str_value: {col_name: class_feature.str2int(str_value)}, input_columns=col_name)

dset = dset.cast(features.Features({
    ...
    col_name: class_feature
})
```


Hi! You can use `Dataset.class_encode_column` for this. And in the next release of `datasets` (this feature is only available on `master`), you'll also be able to use `cast` to do the conversion. 

An example of conversion via `cast`: 
```python
from datasets import Dataset, Features, ClassLabel
d = Dataset.from_dict({"a": ["no", "yes", "no"]})
d = d.cast(Features({"a": ClassLabel(names=["yes", "no"])}))
```
Pin docutils for better doc
State: closed | Author: sgugger (CONTRIBUTOR)

The latest release of docutils make the navbar in the documentation weird and the Markdown wrongly interpreted:

![image](https://user-images.githubusercontent.com/35901082/113711773-5be55280-96b3-11eb-9b3b-9794f17709aa.png)

We had the same problem in Transformers and solved it by pinning docutils (a dep of sphinx).

You can see the version after the change [here](https://32769-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html).
Add OpenSLR dataset
State: closed | Author: cahya-wirawan (CONTRIBUTOR)

OpenSLR (https://openslr.org/) is a site devoted to hosting speech and language resources, such as training corpora for speech recognition, and software related to speech recognition. There are around 80 speech datasets listed in OpenSLR, currently this PR includes only 9 speech datasets SLR41, SLR42, SLR43, SLR44, SLR63, SLR64, SLR65, SLR66 and SLR69 (Javanese, Khmer, Nepali and Sundanese, Malayalam, Marathi, Tamil, Telugu and Catalan). I can add other speech datasets gradually next time.
Pin fsspec lower than 0.9.0
State: closed | Author: lhoestq (MEMBER)

Today's release of `fsspec` 0.9.0 implied a new release of `s3fs` 0.6.0 but this version breaks the CI (see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/5312/workflows/490f3240-cd1c-4dd1-bb60-b416771c5584/jobs/32734) for example)

I'm pinning `fsspec` until this has been resolved
Updated WER metric implementation to avoid memory issues
State: closed | Author: diego-fustes (NONE)

This is in order to fix this issue:

https://github.com/huggingface/datasets/issues/2078



Comments: Hi ! Thanks for suggesting this fix 
Unfortunately it looks like it's already been fixed by #2111 

Feel free to share your thoughts about this PR !

I'm closing this one if you don't mind.
Split type not preserved when reloading the dataset
State: closed | Author: mariosasko (COLLABORATOR)

A minimal reproducible example:
```python
>>> from datasets import load_dataset, Dataset
>>> dset = load_dataset("sst", split="train")
>>> dset.save_to_disk("sst")
>>> type(dset.split)
<class 'datasets.splits.NamedSplit'>
>>> dset = Dataset.load_from_disk("sst")
>>> type(dset.split)  # NamedSplit expected
<class 'str'>
```

It seems like this bug was introduced in #2025.
Regarding Test Sets for the GEM datasets
State: closed | Author: vyraun (NONE)
Labels: name: Dataset discussion

@yjernite Hi, are the test sets for the GEM datasets scheduled to be [added soon](https://gem-benchmark.com/shared_task)? 

e.g.

```
from datasets import load_dataset
DATASET_NAME="common_gen"
data = load_dataset("gem", DATASET_NAME)
```

The test set doesn't have the target or references.

```
data['test'][0]
{'concept_set_id': 0, 'concepts': ['drill', 'field', 'run', 'team'], 'gem_id': 'common_gen-test-0', 'gem_parent_id': 'common_gen-test-0', 'references': [], 'target': ''}
```



Comments: Hi @vyraun ! The test references for CommonGen are not publicly available: you can reach out to the original dataset authors if you would like to ask for them, but we will not be releasing them as part of GEM (March 31st was the release date for the test set inputs, references are incidentally released for some of the test sets but shouldn't really be used for benchmark submissions)

cc @sebastiangehrmann

Oh okay, thanks @yjernite !
Concat only unique fields in DatasetInfo.from_merge
State: closed | Author: mariosasko (COLLABORATOR)

I thought someone from the community with less experience would be interested in fixing this issue, but that wasn't the case.

Fixes #2103 

Comments: Hi @mariosasko,
Just came across this PR and I was wondering if we can use
`description = "\n\n".join(OrderedDict.fromkeys([info.description for info in dataset_infos]))`

This will obviate the need for `unique` and is almost as fast as `set`. We could have used `dict` inplace of `OrderedDict` but it's available 3.7+ onwards

Hi,

let's see what @lhoestq thinks. Although my approach adds more code, it's more readable IMO.

Yeah, that's true. Your approach is more readable.
visualization for cc100 is broken 
State: closed | Author: dorost1234 (NONE)
Labels: name: nlp-viewer

Hi
visualization through dataset viewer for cc100 is broken
https://huggingface.co/datasets/viewer/

thanks a lot


Comments: This looks like an issue with the cc100 dataset itself but not sure
Did you try loading cc100 on your machine ?

Hi
loading works fine, but the viewer only is broken
thanks

On Wed, Apr 7, 2021 at 12:17 PM Quentin Lhoest ***@***.***>
wrote:

> This looks like an issue with the cc100 dataset itself but not sure
> Did you try loading cc100 on your machine ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2162#issuecomment-814793809>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMRUO33JSOYGT6RETWLTHQWNLANCNFSM42IUOR6Q>
> .
>


Hi! This visualization tool is deprecated now. The viewer at https://huggingface.co/datasets/cc100 works fine, so I'm closing this issue.
any possibility to download part of large datasets only?
State: closed | Author: dorost1234 (NONE)

Hi
Some of the datasets I need like cc100 are very large, and then I wonder if I can download first X samples of the shuffled/unshuffled data without going through first downloading the whole data then sampling? thanks

Comments: Not yet but it’s on the short/mid-term roadmap (requested by many indeed).

oh, great, really awesome feature to have, thank you very much for the great, fabulous work

We'll work on dataset streaming soon. This should allow you to only load the examples you need ;)

thanks a lot Quentin, this would be really really a great feature to have

On Wed, Apr 7, 2021 at 12:14 PM Quentin Lhoest ***@***.***>
wrote:

> We'll work on dataset streaming soon. This should allow you to only load
> the examples you need ;)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2161#issuecomment-814791922>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMROD62QAKIJMAKWISTTHQWBVANCNFSM42IUI5JQ>
> .
>


Is streaming completed? On the 1.8.0 docs it is mentioned (https://huggingface.co/docs/datasets/dataset_streaming.html), but when following the example I get the following error:

```
>>> dataset2 = load_dataset("amazon_us_reviews", "Pet_Products_v1_00", split='train', streaming=True)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-1eedab26cff1> in <module>()
----> 1 en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)

3 frames
/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    339                 if value is not None:
    340                     if not hasattr(builder_config, key):
--> 341                         raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
    342                     setattr(builder_config, key, value)
    343 

ValueError: BuilderConfig OscarConfig(name='unshuffled_deduplicated_en', version=1.0.0, data_dir=None, data_files=None, description='Unshuffled and deduplicated, English OSCAR dataset') doesn't have a 'streaming' key.
```

UPDATE: Managed to get streaming working by building from source and installing the additional `datasets[streaming]` package:

```
!pip install git+https://github.com/huggingface/datasets.git
!pip install datasets[streaming]
```

Hi ! Streaming is available on `master` only right now. We'll make a new release 1.9.0 on Monday :)
data_args.preprocessing_num_workers almost freezes 
State: closed | Author: dorost1234 (NONE)

Hi @lhoestq 

I am running this code from huggingface transformers https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py 

to speed up tokenization, since I am running on multiple datasets, I am using data_args.preprocessing_num_workers = 4 with opus100 corpus but this moves on till a point and then this freezes almost for sometime during  tokenization steps and then this is back again, overall to me taking more time than normal case, I appreciate your advice on how I can use this option properly to speed up.

thanks

Comments: Hi.
I cannot always reproduce this issue, and on later runs I did not see it so far. Sometimes also I set 8 processes but I see less being showed, is this normal, here only 5 are shown for 8 being set, thanks

```
#3:  11%|███████████████▊                                                                                                                                  | 172/1583 [00:46<06:21,  3.70ba/s]
#4:   9%|█████████████▏                                                                                                                                    | 143/1583 [00:46<07:46,  3.09ba/s]
#7:   6%|█████████                                                                                                                                          | 98/1583 [00:45<11:34,  2.14ba/s]
#5:   8%|███████████▍                                                                                                                                      | 124/1583 [00:46<09:03,  2.68ba/s]
#6:   7%|██████████▏  
```

closing since I cannot reproduce it again, thanks
adding ccnet dataset
State: closed | Author: dorost1234 (NONE)
Labels: name: dataset request

## Adding a Dataset
- **Name:**  ccnet

- **Description:** 
Common Crawl

- **Paper:** 
https://arxiv.org/abs/1911.00359

- **Data:** 
https://github.com/facebookresearch/cc_net

- **Motivation:**
this is one of the most comprehensive clean monolingual datasets across a variety of languages. Quite important for cross-lingual reseach

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

thanks

Comments: closing since I think this is cc100, just the name has been changed. thanks
viewer "fake_news_english" error
State: closed | Author: emanuelevivoli (NONE)
Labels: name: nlp-viewer

When I visit the [Huggingface - viewer](https://huggingface.co/datasets/viewer/) web site, under the dataset "fake_news_english" I've got this error:

> ImportError: To be able to use this dataset, you need to install the following dependencies['openpyxl'] using 'pip install # noqa: requires this pandas optional dependency for reading xlsx files' for instance'

as well as the error Traceback.



Comments: Thanks for reporting !
The viewer doesn't have all the dependencies of the datasets. We may add openpyxl to be able to show this dataset properly

This viewer tool is deprecated now and the new viewer at https://huggingface.co/datasets/fake_news_english works fine, so I'm closing this issue
Adding the NorNE dataset for Norwegian POS and NER
State: closed | Author: versae (CONTRIBUTOR)

NorNE is a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names.

See #1720.

Comments: Awesome!
load_dataset ignoring features
State: closed | Author: GuillemGSubies (NONE)
Labels: name: bug

First of all, I'm sorry if it is a repeated issue or the changes are already in master, I searched and I didn't find anything. 

I'm using datasets 1.5.0

![image](https://user-images.githubusercontent.com/37592763/113114369-8f376580-920b-11eb-900d-94365b59f04b.png)

As you can see, when I load the dataset, the ClassLabels are ignored, I have to cast the dataset in order to make it work.

Code to reproduce:

```python
import datasets
data_location = "/data/prueba_multiclase"
features = datasets.Features(
        {"texto": datasets.Value("string"), "label": datasets.features.ClassLabel(names=["false", "true"])}
    )
dataset = datasets.load_dataset(
        "csv", data_files=data_location, delimiter="\t", features=features
    )
```

Dataset I used:


[prueba_multiclase.zip](https://github.com/huggingface/datasets/files/6235022/prueba_multiclase.zip) (it has to be unzipped)


Thank you! ❤️ 


Comments: Hi ! Thanks for reporting. I opened a PR to fix this issue: #2201

Nice question which helped me a lot! I have wasted a lot of time to the `DatasetDict` creation from a csv file. Hope the document of this module add some simple examples.

Hi :) We're indeed working on tutorials that we will add to the docs !