helmo/github-issues huggingface dataset:
Implement Dataset from CSV
State: closed | Author: albertvillanova (MEMBER)

Implement `Dataset.from_csv`.

Analogue to #1943.

If finally, the scripts should be used instead, at least we can reuse the tests here. 

Comments: @lhoestq question about public API: `keep_in_memory` or just `in_memory`?

For consistence I'd say `keep_in_memory`, but no strong opinion.

@lhoestq done!
AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
State: closed | Author: dorost1234 (NONE)

Hi
I am trying to concatenate a list of huggingface datastes as:

` train_dataset = datasets.concatenate_datasets(train_datasets)
`
Here is the `train_datasets` when I print:

```
[Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 120361
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2670
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 6944
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 38140
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 173711
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 1655
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 4274
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2019
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2109
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 11963
})]
```

I am getting the following error:

`AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
`

I was wondering if you could help me with this issue, thanks a lot 

Comments: sorry my mistake, datasets were overwritten closing now, thanks a lot
Add Turkish News Category Dataset (270K - Lite Version)
State: closed | Author: yavuzKomecoglu (CONTRIBUTOR)

This PR adds the Turkish News Categories Dataset (270K - Lite Version) dataset which is a text classification dataset by me,  @basakbuluz and @serdarakyol. 
This dataset contains the same news from the current [interpress_news_category_tr dataset](https://huggingface.co/datasets/interpress_news_category_tr) but contains less information, OCR errors are reduced, can be easily separated, and can be divided into 10 classes ("kültürsanat", "ekonomi", "siyaset", "eğitim", "dünya", "spor", "teknoloji", "magazin", "sağlık", "gündem") were rearranged.

@SBrandeis @lhoestq, can you please review this PR?


Comments: I updated your suggestions. Thank you very much for your support. @lhoestq 

> Thanks for changing to ClassLabel :)
> This is all good now !
> 
> However I can see changes in other files than the ones for interpress_news_category_tr_lite, can you please fix that ?
> To do so you can create another branch and another PR to only include the interpress_news_category_tr_lite files.
> 
> Maybe this happened because of a git rebase ? Once you've already pushed your code, please use git merge instead of rebase in order to avoid this.

Thanks for the feedback.
New PR https://github.com/huggingface/datasets/pull/1967
Implement Dataset from JSON and JSON Lines
State: closed | Author: albertvillanova (MEMBER)

Implement `Dataset.from_jsonl`.

Comments: Thanks @lhoestq. I was trying to follow @thomwolf suggestion about integrating that script but as `from_json` method...
> Note that I don't think this is necessary a breaking change, we can still keep the old scripts around

Do you think there is a better way of doing it?

I was trying to implement more or less the same logic as in the script, but I confess I assumed the target was in-memory only...

Basically, I was trying to reimplement `Json(datasets.ArrowBasedBuilder)._generate_tables`, and no writing to arrow file (I assumed only in-memory usage). I started with the first "else" clause... 

I was planning to remove my `_cast_table_to_info_features` and use `paj.read_json(parse_options=...)` instead (like in the script).

@lhoestq I am wondering why `keep_in_memory` has no effect for JSON...

What's the issue exactly ? Apparently it's correctly passed to as_dataset so I don't find the issue

Nevermind @lhoestq, I found where the problem was in my code... I push!


merging master into this branch should fix the CI issue :)


Oops I didn't refresh the page sorry ^^'

Looks all good !

Good job ! I think we can merge after the last changes regarding the error message and the docstring above :)

@lhoestq Done! And I have also added some tests for the `field` parameter.

Let me add some more tests for dict of lists JSON file, please.

@lhoestq done! ;)

We can merge. Additional work will be done in another PR. ;)
CommonGen dataset page shows an error OSError: [Errno 28] No space left on device
State: closed | Author: yuchenlin (CONTRIBUTOR)
Labels: name: nlp-viewer

The page of the CommonGen data https://huggingface.co/datasets/viewer/?dataset=common_gen  shows 
![image](https://user-images.githubusercontent.com/10104354/108959311-1865e600-7629-11eb-868c-cf4cb27034ea.png)


Comments: Facing the same issue for [Squad](https://huggingface.co/datasets/viewer/?dataset=squad) and [TriviaQA](https://huggingface.co/datasets/viewer/?dataset=trivia_qa) datasets as well.

We just fixed the issue, thanks for reporting !
add CoVoST2
State: closed | Author: patil-suraj (CONTRIBUTOR)

This PR adds the CoVoST2 dataset for speech translation and ASR.
https://github.com/facebookresearch/covost#covost-2

The dataset requires manual download as the download page requests an email address and the URLs are temporary.

The dummy data is a bit bigger because of the mp3 files and 36 configs.

Comments: @patrickvonplaten 
I removed the mp3 files, dummy_data is much smaller now!
Add Stanford Sentiment Treebank (SST)
State: closed | Author: patpizio (CONTRIBUTOR)
Labels: name: dataset request

I am going to add SST:

- **Name:** The Stanford Sentiment Treebank
- **Description:** The first corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language
- **Paper:** [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
- **Data:** https://nlp.stanford.edu/sentiment/index.html
- **Motivation:** Already requested in #353, SST is a popular dataset for Sentiment Classification

What's the difference with the [_SST-2_](https://huggingface.co/datasets/viewer/?dataset=glue&config=sst2) dataset included in GLUE? Essentially, SST-2 is a version of SST where:
- the labels were mapped from real numbers in [0.0, 1.0] to a binary label: {0, 1}
- the labels of the *sub-sentences* were included only in the training set
- the labels in the test set are obfuscated

So there is a lot more information in the original SST. The tricky bit is, the data is scattered into many text files and, for one in particular, I couldn't find the original encoding ([*but I'm not the only one*](https://groups.google.com/g/word2vec-toolkit/c/QIUjLw6RqFk/m/_iEeyt428wkJ) 🎵). The only solution I found was to manually replace all the è, ë, ç and so on into an `utf-8` copy of the text file. I uploaded the result in my Dropbox and I am using that as the main repo for the dataset.

Also, the _sub-sentences_ are built at run-time from the information encoded in several text files, so generating the examples is a bit more cumbersome than usual. Luckily, the dataset is not enormous.

I plan to divide the dataset in 2 configs: one with just whole sentences with their labels, the other with sentences _and their sub-sentences_ with their labels. Each config will be split in train, validation and test. Hopefully this makes sense, we may discuss it in the PR I'm going to submit.





Comments: Dataset added in release [1.5.0](https://github.com/huggingface/datasets/releases/tag/1.5.0), I think I can close this.
Use arrow ipc file format
State: closed | Author: lhoestq (MEMBER)

According to the [documentation](https://arrow.apache.org/docs/format/Columnar.html?highlight=arrow1#ipc-file-format), it's identical to the streaming format except that it contains the memory offsets of each sample:

> We define a “file format” supporting random access that is build with the stream format. The file starts and ends with a magic string ARROW1 (plus padding). What follows in the file is identical to the stream format. At the end of the file, we write a footer containing a redundant copy of the schema (which is a part of the streaming format) plus memory offsets and sizes for each of the data blocks in the file. This enables random access any record batch in the file. See File.fbs for the precise details of the file footer.

Since it stores more metadata regarding the positions of the examples in the file, it should enable better example retrieval performances. However from the discussion in https://github.com/huggingface/datasets/issues/1803 it looks like it's not the case unfortunately. Maybe in the future this will allow speed gains.

I think it's still a good idea to start using it anyway for these reasons:
- in the future we may have speed gains
- it contains the arrow streaming format data
- it's compatible with the pyarrow Dataset implementation (it allows to load remote dataframes for example) if we want to use it in the future
- it's also the format used by arrow feather if we want to use it in the future
- it's roughly the same size as the streaming format
- it's easy to have backward compatibility with the streaming format



Comments: Should we close this PR?

Yes, this one was mostly related to https://github.com/huggingface/datasets/issues/4542 but now I think the TF support is not needed at the moment.

What about enabling the Arrow IPC format through an environment variable/config? Would be very helpful to better interop. with other libraries (e.g. [polars](https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.scan_ipc.html)) + supporting reads through pyarrow datasets.
Fix builder config creation with data_dir
State: closed | Author: lhoestq (MEMBER)

The data_dir parameter wasn't taken into account to create the config_id, therefore the resulting builder config was considered not custom. However a builder config that is non-custom must not have a name that collides with the predefined builder config names. Therefore it resulted in a `ValueError("Cannot name a custom BuilderConfig the same as an available...")`

I fixed that by commenting the line that used to ignore the data_dir when creating the config.

It was previously ignored before the introduction of config id because we didn't want to change the config name. Now it's fine to take it into account for the config id.

Now creating a config with a data_dir works again @patrickvonplaten
add m_lama (multilingual lama) dataset
State: closed | Author: pdufter (CONTRIBUTOR)

Add a multilingual (machine translated and automatically generated) version of the LAMA benchmark. For details see the paper https://arxiv.org/pdf/2102.00894.pdf 

Comments: Hi, it seems I am somewhat stuck here. The failed test `ci/circleci: run_dataset_script_tests_pyarrow_1_WIN` seems to be caused by some broken connection (`ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host`). Any help on this is appreciated. 

Edit: Seems to be resolved now.

I guess the `dummy_data.zip` is too large. I can reduce the languages that are contained there, but when testing it, it obviously throws an error, as not all files can be found. I guess I can either i) change the default value regarding which languages are loaded or ii) let the `_generate_examples` silently skip any language for which it cannot find files. Both solutions are not really pretty - is there another way around this?

Thanks for the review and the constructive comments :) ! I tried to address them, and reduced the number of lines in the dummy data to 1 to reduce its size.
updated the wino_bias dataset
State: closed | Author: JieyuZhao (CONTRIBUTOR)

Updated the wino_bias.py script.
- updated the data_url
- added different configurations for different data splits
- added the coreference_cluster to the data features

Comments: Hi @JieyuZhao ! Have you had a chance to add the different configurations ?
Thanks again for your help on this !

> Hi @JieyuZhao ! Have you had a chance to add the different configurations ?
> Thanks again for your help on this !

Hi @lhoestq  Yes, I've updated the code. Now the configuration will have dev/test splits.

> Cool thanks !
> This looks perfect this way.
> 
> Now we just need to update the dataset_infos.json (it contains the metadata of the dataset) and add dummy data to be able to test this script automatically.
> 
> To update the dataset_infos.json you just need delete the current one at `./datasets/wino_biais/dataset_infos.json`, and then run this command:
> 
> ```
> datasets-cli test ./datasets/wino_biais --save_infos --all_configs --ignore_verifications
> ```
> 
> To add the dummy data there's also a tool to add them automatically.
> First delete the folder at `./datasets/wino_biais/dummy` and then run
> 
> ```
> datasets-cli dummy_data ./datasets/wino_biais --auto_generate --match_text_files "*conll" --n_lines 15
> ```
> 
> Let me know if you have questions :)
> Also don't forget to run `make style` to format the code properly.

Thanks for the instruction! I've updated the metadata and the dummy data and also do the formatting. Please let me know if more is needed. :)
Improve typing and style and fix some inconsistencies
State: closed | Author: mariosasko (COLLABORATOR)

This PR:
* improves typing (mostly more consistent use of `typing.Optional`)
* `DatasetDict.cleanup_cache_files` now correctly returns a dict 
* replaces `dict()` with the corresponding literal
* uses `dict_to_copy.copy()` instead of `dict(dict_to_copy)` for shallow copying

Comments: @lhoestq Thanks for the quick review.

I merged master to this branch to re-run the CI before merging :)
Updating old cards
State: closed | Author: mcmillanmajora (CONTRIBUTOR)

Updated the cards for [Allocine](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/allocine), [CNN/DailyMail](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/cnn_dailymail), and [SNLI](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/snli). For the most part, the information was just rearranged or rephrased, but the social impact statements are new.
Fix: Wiki_dpr - add missing scalar quantizer
State: closed | Author: lhoestq (MEMBER)

All the prebuilt wiki_dpr indexes already use SQ8, I forgot to update the wiki_dpr script after building them. Now it's finally done.

The scalar quantizer SQ8 doesn't reduce the performance of the index as shown in retrieval experiments on RAG.
The quantizer reduces the size of the index a lot but increases index building time.
Anonymous Dataset Addition (i.e Anonymous PR?)
State: closed | Author: PierreColombo (CONTRIBUTOR)

Hello,
Thanks a lot for your librairy.
We plan to submit a paper on OpenReview using the Anonymous setting. Is it possible to add a new dataset without breaking the anonimity, with a link to the paper ? 
Cheers 
@eusip

Comments: Hi !
I guess you can add a dataset without the fields that must be kept anonymous, and then update those when the anonymity period is over.
You can also make the PR from an anonymous org.
Pinging @yjernite just to make sure it's ok

Hello,
I would prefer to do the reverse: adding a link to an anonymous paper without the people names/institution in the PR. Would it be conceivable ?
Cheers


Sure, I think it's ok on our side

Yup, sounds good!
Fix save_to_disk with relative path
State: closed | Author: lhoestq (MEMBER)

As noticed in #1919 and #1920 the target directory was not created using `makedirs` so saving to it raises `FileNotFoundError`. For absolute paths it works but not for the good reason. This is because the target path was the same as the temporary path where in-memory data are written as an intermediary step.

I added the `makedirs` call using `fs.makedirs` in order to support remote filesystems.
I also fixed the issue with the target path being the temporary path.

I added a test case for relative paths as well for save_to_disk.

Thanks to @M-Salti for reporting and investigating
How to update the "wino_bias" dataset
State: open | Author: JieyuZhao (CONTRIBUTOR)

Hi all,

Thanks for the efforts to collect all the datasets! But I think there is a problem with the wino_bias dataset. The current link is not correct. How can I update that?

Thanks!

Comments: Hi @JieyuZhao !

You can edit the dataset card of wino_bias to update the URL via a Pull Request. This would be really appreciated :)

The dataset card is the README.md file you can find at https://github.com/huggingface/datasets/tree/master/datasets/wino_bias
Also the homepage url is also mentioned in the wino_bias.py so feel free to update it there as well.

You can create a Pull Request directly from the github interface by editing the files you want and submit a PR, or from a local clone of the repository.

Thanks for noticing !
Standardizing datasets dtypes
State: closed | Author: justin-yan (CONTRIBUTOR)

This PR follows up on discussion in #1900 to have an explicit set of basic dtypes for datasets.

This moves away from str(pyarrow.DataType) as the method of choice for creating dtypes, favoring an explicit mapping to a list of supported Value dtypes.

I believe in practice this should be backward compatible, since anyone previously using Value() would only have been able to use dtypes that had an identically named pyarrow factory function, which are all explicitly supported here, with `float32` and `float64` acting as the official datasets dtypes, which resolves the tension between `double` being the pyarrow dtype and `float64` being the pyarrow type factory function.

Comments: @lhoestq - apologies for the multiple PRs, my previous one (#1905) got mangled due to some merge conflicts that I had trouble resolving so I just cherry-picked my changes onto a fresh branch here.
Fix save_to_disk issue
State: closed | Author: M-Salti (CONTRIBUTOR)

Fixes #1919 


Comments: So I was curious why the issue reported at #1919 wasn't caught in [this test](https://github.com/huggingface/datasets/blob/248104c4bdb2e01c036b7578867199191fbff181/tests/test_arrow_dataset.py#L209), so I did some digging.
I tried to save to a temporary directory (just like in the test), like this:
```python
with tempfile.TemporaryDirectory() as requested_tempdir:
    squad.save_to_disk(requested_tempdir)     # no error
```
and it executes succesfuly without problems.
So why does it work, but this doesn't?
```python
squad.save_to_disk("./squad")   # error
```
It's because `save_to_disk` also creates a temporary directory (let's call it `tempdir`), and since `tempdir` and `requested_tempdir` share the same parents, the `Path.joinpath` method [(here)](https://github.com/huggingface/datasets/blob/248104c4bdb2e01c036b7578867199191fbff181/src/datasets/arrow_dataset.py#L469) will keep `requested_tempdir` as it is and the *train* directory will be created under `requested_tempdir` and hence no errors will arise.

But in the second case (where we are saving to a local dir), the *train* directory is created under *squad* which in turn is created under `tempdir`, not under `.` (current dir).

So, all of this probably doesn't help solving the issue but it might help creating a better test, and it also makes me wonder why are we saving to a temporary dir in `save_to_disk` anyway? I mean, won't it be removed with all its contents upon execution completion? what's the point then? 

CLosing in favor of #1923
Failure to save with save_to_disk
State: closed | Author: M-Salti (CONTRIBUTOR)

When I try to save a dataset locally using the `save_to_disk` method I get the error:

```bash
FileNotFoundError: [Errno 2] No such file or directory: '/content/squad/train/squad-train.arrow'
```

To replicate:

1. Install `datasets` from master
2. Run this code:

    ```python
    from datasets import load_dataset
    squad = load_dataset("squad")   # or any other dataset
    squad.save_to_disk("squad")     # error here
    ```

The problem is that the method is not creating a directory with the name `dataset_path` for saving the dataset in (i.e. it's not creating the *train* and *validation* directories in this case). After creating the directory the problem resolves.
I'll open a PR soon doing that and linking this issue.


Comments: Hi thanks for reporting and for proposing a fix :)

I just merged a fix, feel free to try it from the master branch !

Closing since this has been fixed by #1923
UnicodeDecodeError: windows 10 machine
State: closed | Author: yosiasz (NONE)

Windows 10
Php 3.6.8

when running

```
import datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])
```
I get the following error

```
file "C:\PYTHON\3.6.8\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 58: character maps to <undefined>
```

Comments: upgraded to php 3.9.2 and it works!
Add keep_linebreaks parameter to text loader
State: closed | Author: lhoestq (MEMBER)

As asked in #870 and https://github.com/huggingface/transformers/issues/10269 there should be a parameter to keep the linebreaks when loading a text dataset.
cc @sgugger @jncasey

Comments: Just so I understand how it can be used in practice, do you have an example showing how to load a text dataset with this option?

Sure ! Here is an example:
```python
from datasets import load_dataset

load_dataset("text", keep_linebreaks=True, data_files=...)
```

I'll update the documentation to explain this

Perfect!
Update: WMT - use mirror links
State: closed | Author: lhoestq (MEMBER)

As asked in #1892 I created mirrors of the data hosted on statmt.org and updated the wmt scripts.
Now downloading the wmt datasets is blazing fast :)

cc @stas00 @patrickvonplaten 

Comments: So much better - thank you for doing that, @lhoestq!

Also fixed the `uncorpus` urls for wmt19 ru-en and zh-en for https://github.com/huggingface/datasets/issues/1893

Thanks!
Can this be merged sooner? 
I manually update it and it works well.