jamesLeeeeeee/datasets-github-issues huggingface dataset:
[WIP] Adding Support for Reading Pandas Category
Author: justin-yan | Pull request: yes | State: closed

@lhoestq - continuing our conversation from https://github.com/huggingface/datasets/issues/1906#issuecomment-784247014

The goal of this PR is to support `Dataset.from_pandas(df)` where the dataframe contains a Category.

Just the 4 line change below actually does seem to work:

```
>>> from datasets import Dataset
>>> import pandas as pd
>>> df = pd.DataFrame(pd.Series(["a", "b", "c", "a"], dtype="category"))
>>> ds = Dataset.from_pandas(df)
>>> ds.to_pandas()
   0
0  a
1  b
2  c
3  a
>>> ds.to_pandas().dtypes
0    category
dtype: object
```

save_to_disk, etc. all seem to work as well.  The main things that are theoretically "incorrect" if we leave this are:

```
>>> ds.features.type
StructType(struct<0: int64>)
```
there are a decent number of references to this property in the library, but I can't find anything that seems to actually break as a result of this being int64 vs. dictionary?  I think the gist of my question is: a) do we *need* to change the dtype of Classlabel and have get_nested_type return a pyarrow.DictionaryType instead of int64? and b) do you *want* it to change?  The biggest challenge I see to implementing this correctly is that the data will need to be passed in along with the pyarrow schema when instantiating the Classlabel (I *think* this is unavoidable, since the type itself doesn't contain the actual label values) which could be a fairly intrusive change - e.g. `from_arrow_schema`'s interface would need to change to include optional arrow data?  Once we start going down this path of modifying the public interfaces I am admittedly feeling a little bit outside of my comfort zone

Additionally I think `int2str`, `str2int`, and `encode_example` probably won't work - but I can't find any usages of them in the library itself.
add CoVoST2
Author: patil-suraj | Pull request: yes | State: closed

This PR adds the CoVoST2 dataset for speech translation and ASR.
https://github.com/facebookresearch/covost#covost-2

The dataset requires manual download as the download page requests an email address and the URLs are temporary.

The dummy data is a bit bigger because of the mp3 files and 36 configs.
Add Stanford Sentiment Treebank (SST)
Author: patpizio | Pull request: no | State: closed

I am going to add SST:

- **Name:** The Stanford Sentiment Treebank
- **Description:** The first corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language
- **Paper:** [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
- **Data:** https://nlp.stanford.edu/sentiment/index.html
- **Motivation:** Already requested in #353, SST is a popular dataset for Sentiment Classification

What's the difference with the [_SST-2_](https://huggingface.co/datasets/viewer/?dataset=glue&config=sst2) dataset included in GLUE? Essentially, SST-2 is a version of SST where:
- the labels were mapped from real numbers in [0.0, 1.0] to a binary label: {0, 1}
- the labels of the *sub-sentences* were included only in the training set
- the labels in the test set are obfuscated

So there is a lot more information in the original SST. The tricky bit is, the data is scattered into many text files and, for one in particular, I couldn't find the original encoding ([*but I'm not the only one*](https://groups.google.com/g/word2vec-toolkit/c/QIUjLw6RqFk/m/_iEeyt428wkJ) 🎵). The only solution I found was to manually replace all the è, ë, ç and so on into an `utf-8` copy of the text file. I uploaded the result in my Dropbox and I am using that as the main repo for the dataset.

Also, the _sub-sentences_ are built at run-time from the information encoded in several text files, so generating the examples is a bit more cumbersome than usual. Luckily, the dataset is not enormous.

I plan to divide the dataset in 2 configs: one with just whole sentences with their labels, the other with sentences _and their sub-sentences_ with their labels. Each config will be split in train, validation and test. Hopefully this makes sense, we may discuss it in the PR I'm going to submit.
Use arrow ipc file format
Author: lhoestq | Pull request: yes | State: closed

According to the [documentation](https://arrow.apache.org/docs/format/Columnar.html?highlight=arrow1#ipc-file-format), it's identical to the streaming format except that it contains the memory offsets of each sample:

> We define a “file format” supporting random access that is build with the stream format. The file starts and ends with a magic string ARROW1 (plus padding). What follows in the file is identical to the stream format. At the end of the file, we write a footer containing a redundant copy of the schema (which is a part of the streaming format) plus memory offsets and sizes for each of the data blocks in the file. This enables random access any record batch in the file. See File.fbs for the precise details of the file footer.

Since it stores more metadata regarding the positions of the examples in the file, it should enable better example retrieval performances. However from the discussion in https://github.com/huggingface/datasets/issues/1803 it looks like it's not the case unfortunately. Maybe in the future this will allow speed gains.

I think it's still a good idea to start using it anyway for these reasons:
- in the future we may have speed gains
- it contains the arrow streaming format data
- it's compatible with the pyarrow Dataset implementation (it allows to load remote dataframes for example) if we want to use it in the future
- it's also the format used by arrow feather if we want to use it in the future
- it's roughly the same size as the streaming format
- it's easy to have backward compatibility with the streaming format
Fix builder config creation with data_dir
Author: lhoestq | Pull request: yes | State: closed

The data_dir parameter wasn't taken into account to create the config_id, therefore the resulting builder config was considered not custom. However a builder config that is non-custom must not have a name that collides with the predefined builder config names. Therefore it resulted in a `ValueError("Cannot name a custom BuilderConfig the same as an available...")`

I fixed that by commenting the line that used to ignore the data_dir when creating the config.

It was previously ignored before the introduction of config id because we didn't want to change the config name. Now it's fine to take it into account for the config id.

Now creating a config with a data_dir works again @patrickvonplaten
Improve typing and style and fix some inconsistencies
Author: mariosasko | Pull request: yes | State: closed

This PR:
* improves typing (mostly more consistent use of `typing.Optional`)
* `DatasetDict.cleanup_cache_files` now correctly returns a dict 
* replaces `dict()` with the corresponding literal
* uses `dict_to_copy.copy()` instead of `dict(dict_to_copy)` for shallow copying
Updating old cards
Author: mcmillanmajora | Pull request: yes | State: closed

Updated the cards for [Allocine](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/allocine), [CNN/DailyMail](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/cnn_dailymail), and [SNLI](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/snli). For the most part, the information was just rearranged or rephrased, but the social impact statements are new.
Fix: Wiki_dpr - add missing scalar quantizer
Author: lhoestq | Pull request: yes | State: closed

All the prebuilt wiki_dpr indexes already use SQ8, I forgot to update the wiki_dpr script after building them. Now it's finally done.

The scalar quantizer SQ8 doesn't reduce the performance of the index as shown in retrieval experiments on RAG.
The quantizer reduces the size of the index a lot but increases index building time.
Fix: Wiki_dpr - fix when with_embeddings is False or index_name is "no_index"
Author: lhoestq | Pull request: yes | State: closed

Fix the bugs noticed in #1915 

There was a bug when `with_embeddings=False` where the configuration name was the same as if `with_embeddings=True`, which led the dataset builder to do bad verifications (for example it used to expect to download the embeddings for `with_embeddings=False`).

Another issue was that setting `index_name="no_index"` didn't set `with_index` to False.

I fixed both of them and added dummy data for those configurations for testing.
Fix save_to_disk with relative path
Author: lhoestq | Pull request: yes | State: closed

As noticed in #1919 and #1920 the target directory was not created using `makedirs` so saving to it raises `FileNotFoundError`. For absolute paths it works but not for the good reason. This is because the target path was the same as the temporary path where in-memory data are written as an intermediary step.

I added the `makedirs` call using `fs.makedirs` in order to support remote filesystems.
I also fixed the issue with the target path being the temporary path.

I added a test case for relative paths as well for save_to_disk.

Thanks to @M-Salti for reporting and investigating
Standardizing datasets dtypes
Author: justin-yan | Pull request: yes | State: closed

This PR follows up on discussion in #1900 to have an explicit set of basic dtypes for datasets.

This moves away from str(pyarrow.DataType) as the method of choice for creating dtypes, favoring an explicit mapping to a list of supported Value dtypes.

I believe in practice this should be backward compatible, since anyone previously using Value() would only have been able to use dtypes that had an identically named pyarrow factory function, which are all explicitly supported here, with `float32` and `float64` acting as the official datasets dtypes, which resolves the tension between `double` being the pyarrow dtype and `float64` being the pyarrow type factory function.
Failure to save with save_to_disk
Author: M-Salti | Pull request: no | State: closed

When I try to save a dataset locally using the `save_to_disk` method I get the error:

```bash
FileNotFoundError: [Errno 2] No such file or directory: '/content/squad/train/squad-train.arrow'
```

To replicate:

1. Install `datasets` from master
2. Run this code:

    ```python
    from datasets import load_dataset
    squad = load_dataset("squad")   # or any other dataset
    squad.save_to_disk("squad")     # error here
    ```

The problem is that the method is not creating a directory with the name `dataset_path` for saving the dataset in (i.e. it's not creating the *train* and *validation* directories in this case). After creating the directory the problem resolves.
I'll open a PR soon doing that and linking this issue.
UnicodeDecodeError: windows 10 machine
Author: yosiasz | Pull request: no | State: closed

Windows 10
Php 3.6.8

when running

```
import datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])
```
I get the following error

```
file "C:\PYTHON\3.6.8\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 58: character maps to <undefined>
```
Saving processed dataset running infinitely
Author: ayubSubhaniya | Pull request: no | State: open

I have a text dataset of size 220M.

For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 3hrs. I used map() with batch size 1024 and multi-process with 96 processes.

filter() function was way to slow, so I used a hack to use pyarrow filter table function, which is damm fast. Mentioned [here](https://github.com/huggingface/datasets/issues/1796)

```dataset._data = dataset._data.filter(...)```
It took 1 hr for the filter.

Then i use `save_to_disk()` on processed dataset and it is running forever.

I have been waiting since 8 hrs, it has not written a single byte. 

Infact it has actually read from disk more than 100GB, screenshot below shows the stats using `iotop`. 
Second process is the one.
<img width="1672" alt="Screenshot 2021-02-19 at 6 36 53 PM" src="https://user-images.githubusercontent.com/20911334/108508197-7325d780-72e1-11eb-8369-7c057d137d81.png">


I am not able to figure out, whether this is some issue with dataset library or that it is due to my hack for filter() function.
DBPedia14 Dataset Checksum bug?
Author: francisco-perez-sorrosal | Pull request: no | State: closed

Hi there!!!

I've been using successfully the DBPedia dataset (https://huggingface.co/datasets/dbpedia_14) with my codebase in the last couple of weeks, but in the last couple of days now I get this error:

```
Traceback (most recent call last):
  File "./conditional_classification/basic_pipeline.py", line 178, in <module>
    main()
  File "./conditional_classification/basic_pipeline.py", line 128, in main
    corpus.load_data(limit_train_examples_per_class=args.data_args.train_examples_per_class,
  File "/home/fp/dev/conditional_classification/conditional_classification/datasets_base.py", line 83, in load_data
    datasets = load_dataset(self.name, split=dataset_split)
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/load.py", line 609, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/builder.py", line 526, in download_and_prepare
    self._download_and_prepare(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/builder.py", line 586, in _download_and_prepare
    verify_checksums(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbQ2Vic1kxMmZZQ1k']
```

I've seen this has happened before in other datasets as reported in #537.

I've tried clearing my cache and call again `load_dataset` but still is not working. My same codebase is successfully downloading and using other datasets (e.g. AGNews) without any problem, so I guess something has happened specifically to the DBPedia dataset in the last few days. 

Can you please check if there's a problem with the checksums? 

Or this is related to any other stuff? I've seen that the path in the cache for the dataset is `/home/fp/.cache/huggingface/datasets/d_bpedia14/dbpedia_14/2.0.0/a70413e39e7a716afd0e90c9e53cb053691f56f9ef5fe317bd07f2c368e8e897...` and includes `d_bpedia14` instead maybe of `dbpedia_14`. Was this maybe a bug introduced recently?

Thanks!
Feature Request: Support for Pandas `Categorical`
Author: justin-yan | Pull request: no | State: open

```
from datasets import Dataset
import pandas as pd
import pyarrow

df = pd.DataFrame(pd.Series(["a", "b", "c", "a"], dtype="category"))
pyarrow.Table.from_pandas(df)
Dataset.from_pandas(df)
# Throws NotImplementedError
# TODO(thom) this will need access to the dictionary as well (for labels). I.e. to the py_table
```

I'm curious if https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L796 could be built out in a way similar to `Sequence`?

e.g. a `Map` class (or whatever name the maintainers might prefer) that can accept:

```
index_type = generate_from_arrow_type(pa_type.index_type)
value_type = generate_from_arrow_type(pa_type.value_type)
```

and then additional code points to modify:

- FeatureType: https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L694
- A branch to handle Map in get_nested_type: https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L719
- I don't quite understand what `encode_nested_example` does but perhaps a branch there? https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L755
- Similarly, I don't quite understand why `Sequence` is used this way in `generate_from_dict`, but perhaps a branch here? https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L775

I couldn't find other usages of `Sequence` outside of defining specific datasets, so I'm not sure if that's a comprehensive set of touchpoints.
Standardizing datasets.dtypes
Author: justin-yan | Pull request: yes | State: closed

This PR was further branched off of jdy-str-to-pyarrow-parsing, so it depends on https://github.com/huggingface/datasets/pull/1900 going first for the diff to be up-to-date (I'm not sure if there's a way for me to use jdy-str-to-pyarrow-parsing as a base branch while having it appear in the pull requests here).

This moves away from `str(pyarrow.DataType)` as the method of choice for creating dtypes, favoring an explicit mapping to a list of supported Value dtypes.

I believe in practice this should be backward compatible, since anyone previously using Value() would only have been able to use dtypes that had an identically named pyarrow factory function, which are all explicitly supported here.
Fix to_pandas for boolean ArrayXD
Author: lhoestq | Pull request: yes | State: closed

As noticed in #1887 the conversion of a dataset with a boolean ArrayXD feature types fails because of the underlying ListArray conversion to numpy requires `zero_copy_only=False`.

zero copy is available for all primitive types except booleans
see https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy
and https://issues.apache.org/jira/browse/ARROW-2871?jql=text%20~%20%22boolean%20to_numpy%22

cc @SBrandeis
Initial commit for the addition of TIMIT dataset
Author: vrindaprabhu | Pull request: yes | State: closed

Below points needs to be addressed:

- Creation of dummy dataset is failing
- Need to check on the data representation
- License is not creative commons. Copyright: Portions © 1993 Trustees of the University of Pennsylvania

Also the links (_except the download_) point to the ami corpus! ;-)

@patrickvonplaten  Requesting your comments, will be happy to address them!
Issue #1895: Bugfix for string_to_arrow timestamp[ns] support
Author: justin-yan | Pull request: yes | State: closed

Should resolve https://github.com/huggingface/datasets/issues/1895

The main part of this PR adds additional parsing in `string_to_arrow` to convert the timestamp dtypes that result from `str(pa_type)` back into the pa.DataType TimestampType.

While adding unit-testing, I noticed that support for the double/float types also don't invert correctly, so I added them, which I believe would hypothetically make this section of `Value` redundant:

```
    def __post_init__(self):
        if self.dtype == "double":  # fix inferred type
            self.dtype = "float64"
        if self.dtype == "float":  # fix inferred type
            self.dtype = "float32"
```

However, since I think Value.dtype is part of the public interface, removing that would result in a backward-incompatible change, so I didn't muck with that.

The rest of the PR consists of docstrings that I added while developing locally so I could keep track of which functions were supposed to be inverses of each other, and thought I'd include them initially in case you want to keep them around, but I'm happy to delete or remove any of them at your request!
ALT dataset has repeating instances in all splits
Author: 10-zin | Pull request: no | State: closed

The [ALT](https://huggingface.co/datasets/alt) dataset has all the same instances within each split :/
Seemed like a great dataset for some experiments I wanted to carry out, especially since its medium-sized, and has all splits.

Would be great if this could be fixed :)

Added a snapshot of the contents from `explore-datset` feature, for quick reference.

![image](https://user-images.githubusercontent.com/33179372/108206321-442a2d00-714c-11eb-882f-b4b6e708ef9c.png)
Fix PandasArrayExtensionArray conversion to native type
Author: lhoestq | Pull request: yes | State: closed

To make the conversion to csv work in #1887 , we need  PandasArrayExtensionArray used for multidimensional numpy arrays to be converted to pandas native types.
However previously pandas.core.internals.ExtensionBlock.to_native_types would fail with an PandasExtensionArray because
1. the PandasExtensionArray.isna method was wrong
2. the conversion of a PandasExtensionArray to a numpy array with dtype=object was returning a multidimensional array while pandas excepts a 1D array in this case (more info [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.extensions.ExtensionArray.html#pandas.api.extensions.ExtensionArray))

I fixed these two issues and now the conversion to native types works, and so is the export to csv.
cc @SBrandeis
Bug Report: timestamp[ns] not recognized
Author: justin-yan | Pull request: no | State: closed

Repro:

```
from datasets import Dataset
import pandas as pd
import pyarrow

df = pd.DataFrame(pd.date_range("2018-01-01", periods=3, freq="H"))
pyarrow.Table.from_pandas(df)
Dataset.from_pandas(df)
# Throws ValueError: Neither timestamp[ns] nor timestamp[ns]_ seems to be a pyarrow data type.
```

The factory function seems to be just "timestamp": https://arrow.apache.org/docs/python/generated/pyarrow.timestamp.html#pyarrow.timestamp

It seems like https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L36-L43 could have a little bit of additional structure for handling these cases?  I'd be happy to take a shot at opening a PR if I could receive some guidance on whether parsing something like `timestamp[ns]` and resolving it to timestamp('ns') is the goal of this method.

Alternatively, if I'm using this incorrectly (e.g. is the expectation that we always provide a schema when timestamps are involved?), that would be very helpful to know as well!

```
$ pip list  # only the relevant libraries/versions
datasets                      1.2.1
pandas                        1.0.3
pyarrow                       3.0.0
```
benchmarking against MMapIndexedDataset
Author: sshleifer | Pull request: no | State: open

I am trying to benchmark my datasets based implementation against fairseq's [`MMapIndexedDataset`](https://github.com/pytorch/fairseq/blob/master/fairseq/data/indexed_dataset.py#L365) and finding that, according to psrecord, my `datasets` implem uses about 3% more CPU memory and runs 1% slower for `wikitext103` (~1GB of tokens).

Questions:
1) Is this (basically identical) performance expected? 
2) Is there a scenario where this library will outperform `MMapIndexedDataset`? (maybe more examples/larger examples?)
3) Should I be using different benchmarking tools than `psrecord`/how do you guys do benchmarks?

Thanks in advance! Sam
`list_datasets()` returns a list of strings, not objects
Author: pminervini | Pull request: yes | State: closed

Here and there in the docs there is still stuff like this:

```python
>>> datasets_list = list_datasets()
>>> print(', '.join(dataset.id for dataset in datasets_list))
```

However, my understanding is that `list_datasets()` returns a list of strings rather than a list of objects.
Replace flatten_nested
Author: albertvillanova | Pull request: yes | State: closed

Replace `flatten_nested` with `NestedDataStructure.flatten`.

This is a first step towards having all NestedDataStructure logic as a separated concern, independent of the caller/user of the data structure.

Eventually, all checks (whether the underlying data is list, dict, etc.) will be only inside this class.

I have also generalized the flattening, and now it handles multiple levels of nesting.