kardwalker/my_dataset huggingface dataset:
GitHub issue #1956: [distributed env] potentially unsafe parallel execution
State: closed | Created: 2021-02-27T20:38:45Z

```
metric = load_metric('glue', 'mrpc', num_process=num_process, process_id=rank)
```

presumes that there is only one set of parallel processes running - and will intermittently fail if you have multiple sets running as they will surely overwrite each other. Similar to https://github.com/huggingface/datasets/issues/1942 (but for a different reason).
That's why dist environments use some unique to a group identifier so that each group is dealt with separately. 

e.g. the env-way of pytorch dist syncing is done with a unique per set `MASTER_ADDRESS+MASTER_PORT`

So ideally this interface should ask for a shared secret to do the right thing.

I'm not reporting an immediate need, but am only flagging that this will hit someone down the road.

This problem can be remedied by adding a new optional `shared_secret` option, which can then be used to differentiate different groups of processes. and this secret should be part of the file lock name and the experiment.

Thank you
GitHub issue #1955: typos + grammar
State: closed | Created: 2021-02-27T20:21:43Z

This PR proposes a few typo + grammar fixes, and rewrites some sentences in an attempt to improve readability.

N.B. When referring to the library `datasets` in the docs it is typically used as a singular, and it definitely is a singular when written as "`datasets` library", that is "`datasets` library is ..." and not "are ...".
GitHub issue #1952: Handle timeouts
State: closed | Created: 2021-02-26T15:02:07Z

As noticed in https://github.com/huggingface/datasets/issues/1939, timeouts were not properly handled when loading a dataset.
This caused the connection to hang indefinitely when working in a firewalled environment cc @stas00 

I added a default timeout, and included an option to our offline environment for tests to be able to simulate both connection errors and timeout errors (previously it was simulating connection errors only).

Now networks calls don't hang indefinitely.
The default timeout is set to 10sec (we might reduce it).
GitHub issue #1951: Add cross-platform support for datasets-cli
State: closed | Created: 2021-02-26T14:56:25Z

One thing I've noticed while going through the codebase is the usage of `scripts` in `setup.py`. This [answer](https://stackoverflow.com/a/28119736/14095927) on SO explains it nicely why it's better to use `entry_points` instead of `scripts`. To add cross-platform support to the CLI, this PR replaces `scripts` with `entry_points` in `setup.py` and moves datasets-cli to src/datasets/commands/datasets_cli.py. All *.md and *.rst files are updated accordingly. The same changes were made in the transformers repo to add cross-platform ([link to PR](https://github.com/huggingface/transformers/pull/4131)).
GitHub issue #1949: Enable Fast Filtering using Arrow Dataset
State: open | Created: 2021-02-26T02:53:37Z

Hi @lhoestq,

As mentioned in Issue #1796, I would love to work on enabling fast filtering/mapping. Can you please share the expectations? It would be great if you could point me to the relevant methods/files involved. Or the docs or maybe an overview of `arrow_dataset.py`. I only ask this because I am having trouble getting started ;-;

Any help would be appreciated.

Thanks,
Gunjan
GitHub issue #1948: dataset loading logger level
State: closed | Created: 2021-02-25T18:33:37Z

on master I get this with `--dataset_name wmt16 --dataset_config ro-en`:

```
WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-2e01bead8cf42e26.arrow
WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-ac3bebaf4f91f776.arrow
WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-810c3e61259d73a9.arrow
```

why are those WARNINGs? Should be INFO, no?

warnings should only be used when a user needs to pay attention to something, this is just informative - I'd even say it should be DEBUG, but definitely not WARNING.

Thank you.
GitHub issue #1947: Update documentation with not in place transforms and update DatasetDict
State: closed | Created: 2021-02-25T16:23:18Z

In #1883 were added the not in-place transforms `flatten`, `remove_columns`, `rename_column` and `cast`.

I added them to the documentation and added a paragraph on how to use them

You can preview the documentation [here](https://28862-250213286-gh.circle-artifacts.com/0/docs/_build/html/processing.html#renaming-removing-casting-and-flattening-columns)

I also added these methods to the DatasetDict class.
GitHub issue #1945: AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
State: closed | Created: 2021-02-25T13:09:45Z

Hi
I am trying to concatenate a list of huggingface datastes as:

` train_dataset = datasets.concatenate_datasets(train_datasets)
`
Here is the `train_datasets` when I print:

```
[Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 120361
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2670
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 6944
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 38140
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 173711
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 1655
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 4274
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2019
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2109
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 11963
})]
```

I am getting the following error:

`AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
`

I was wondering if you could help me with this issue, thanks a lot
GitHub issue #1944: Add Turkish News Category Dataset (270K - Lite Version)
State: closed | Created: 2021-02-25T09:45:22Z

This PR adds the Turkish News Categories Dataset (270K - Lite Version) dataset which is a text classification dataset by me,  @basakbuluz and @serdarakyol. 
This dataset contains the same news from the current [interpress_news_category_tr dataset](https://huggingface.co/datasets/interpress_news_category_tr) but contains less information, OCR errors are reduced, can be easily separated, and can be divided into 10 classes ("kültürsanat", "ekonomi", "siyaset", "eğitim", "dünya", "spor", "teknoloji", "magazin", "sağlık", "gündem") were rearranged.

@SBrandeis @lhoestq, can you please review this PR?
GitHub issue #1942: [experiment] missing default_experiment-1-0.arrow
State: closed | Created: 2021-02-25T03:02:15Z

the original report was pretty bad and incomplete - my apologies!

Please see the complete version here: https://github.com/huggingface/datasets/issues/1942#issuecomment-786336481

------------

As mentioned here https://github.com/huggingface/datasets/issues/1939 metrics don't get cached, looking at my local `~/.cache/huggingface/metrics` - there are many `*.arrow.lock` files but zero metrics files.

w/o the network I get:
```
FileNotFoundError: [Errno 2] No such file or directory: '~/.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow
```
there is just `~/.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow.lock`

I did run the same `run_seq2seq.py` script on the instance with network and it worked just fine, but only the lock file was left behind.

this is with master.

Thank you.
GitHub issue #1940: Side effect when filtering data due to `does_function_return_dict` call in `Dataset.map()`
State: closed | Created: 2021-02-24T19:18:56Z

Hi there!

In my codebase I have a function to filter rows in a dataset, selecting only a certain number of examples per class. The function passes a extra argument to maintain a counter of the number of dataset rows/examples already selected per each class, which are the ones I want to keep in the end:

```python
      def fill_train_examples_per_class(example, per_class_limit: int, counter: collections.Counter):
          label = int(example['label'])
          current_counter = counter.get(label, 0)
          if current_counter < per_class_limit:
              counter[label] = current_counter + 1
              return True
          return False
```

At some point I invoke it through the `Dataset.filter()` method in the `arrow_dataset.py` module like this:

```python
...
kwargs = {"per_class_limit": train_examples_per_class_limit, "counter": Counter()}
datasets['train'] = datasets['train'].filter(fill_train_examples_per_class,  num_proc=1, fn_kwargs=kwargs)
...
```

The problem is that, passing a stateful container (the counter,) provokes a side effect in the new filtered dataset obtained. This is due to the fact that at some point in `filter()`, the `map()`'s function `does_function_return_dict` is invoked in  line [1290](https://github.com/huggingface/datasets/blob/96578adface7e4bc1f3e8bafbac920d72ca1ca60/src/datasets/arrow_dataset.py#L1290). 

When this occurs, the state of the counter is initially modified by the effects of the function call on the 1 or 2 rows selected in lines 1288 and 1289 of the same file (which are marked as `test_inputs` & `test_indices` respectively in lines 1288 and 1289. This happens out of the control of the user (which for example can't reset the state of the counter before continuing the execution,) provoking in the end an undesired side effect in the results obtained. 

In my case, the resulting dataset -despite of the counter results are ok- lacks an instance of the classes 0 and 1 (which happen to be the classes of the first two examples of my dataset.) The rest of the classes I have in my dataset, contain the right number of examples as they were not affected by the effects of `does_function_return_dict` call.

I've debugged my code extensively and made a workaround myself hardcoding the necessary stuff (basically putting `update_data=True` in line 1290,) and then I obtain the results I expected without the side effect.

Is there a way to avoid that call to `does_function_return_dict` in map()'s line 1290 ? (e.g. extracting the required information that `does_function_return_dict` returns without making the testing calls to the user function on dataset rows 0 & 1) 

Thanks in advance,

Francisco Perez-Sorrosal
GitHub issue #1939: [firewalled env] OFFLINE mode
State: closed | Created: 2021-02-24T17:13:42Z

This issue comes from a need to be able to run `datasets` in a firewalled env, which currently makes the software hang until it times out, as it's unable to complete the network calls.

I propose the following approach to solving this problem, using the example of `run_seq2seq.py` as a sample program. There are 2 possible ways to going about it.

## 1. Manual

manually prepare data and metrics files, that is transfer to the firewalled instance the dataset and the metrics and run:

```
DATASETS_OFFLINE=1 run_seq2seq.py  --train_file xyz.csv --validation_file xyz.csv ...
```

`datasets` must not make any network calls and if there is a logic to do that and something is missing it should assert that this or that action requires network and therefore it can't proceed.

## 2. Automatic

In some clouds one can prepare a datastorage ahead of time with a normal networked environment but which doesn't have gpus and then one switches to the gpu instance which is firewalled, but it can access all the cached data. This is the ideal situation, since in this scenario we don't have to do anything manually, but simply run the same application twice:

1. on the non-firewalled instance:
```
run_seq2seq.py  --dataset_name wmt16 --dataset_config ro-en ...
```

which should download and cached everything.

2. and then immediately after on the firewalled instance, which shares the same filesystem
```
DATASETS_OFFLINE=1 run_seq2seq.py  --dataset_name wmt16 --dataset_config ro-en ...
```

and the metrics and datasets should be cached by the invocation number 1 and any network calls be skipped and if the logic is missing data it should assert and not try to fetch any data from online.

## Common Issues

1. for example currently `datasets` tries to look up online datasets if the files contain json or csv, despite the paths already provided

```
     if dataset and path in _PACKAGED_DATASETS_MODULES:
```

2. it has an issue with metrics. e.g. I had to manually copy `rouge/rouge.py` from the `datasets` repo to the current dir - or it was hanging.

I had to comment out `head_hf_s3(...)` calls to make things work. So all those `try: head_hf_s3(...)` shouldn't be tried with `DATASETS_OFFLINE=1`

Here is the corresponding issue for `transformers`: https://github.com/huggingface/transformers/issues/10379

Thanks.
GitHub issue #1937: CommonGen dataset page shows an error OSError: [Errno 28] No space left on device
State: closed | Created: 2021-02-24T06:47:33Z

The page of the CommonGen data https://huggingface.co/datasets/viewer/?dataset=common_gen  shows 
![image](https://user-images.githubusercontent.com/10104354/108959311-1865e600-7629-11eb-868c-cf4cb27034ea.png)
GitHub issue #1936: [WIP] Adding Support for Reading Pandas Category
State: closed | Created: 2021-02-23T18:32:54Z

@lhoestq - continuing our conversation from https://github.com/huggingface/datasets/issues/1906#issuecomment-784247014

The goal of this PR is to support `Dataset.from_pandas(df)` where the dataframe contains a Category.

Just the 4 line change below actually does seem to work:

```
>>> from datasets import Dataset
>>> import pandas as pd
>>> df = pd.DataFrame(pd.Series(["a", "b", "c", "a"], dtype="category"))
>>> ds = Dataset.from_pandas(df)
>>> ds.to_pandas()
   0
0  a
1  b
2  c
3  a
>>> ds.to_pandas().dtypes
0    category
dtype: object
```

save_to_disk, etc. all seem to work as well.  The main things that are theoretically "incorrect" if we leave this are:

```
>>> ds.features.type
StructType(struct<0: int64>)
```
there are a decent number of references to this property in the library, but I can't find anything that seems to actually break as a result of this being int64 vs. dictionary?  I think the gist of my question is: a) do we *need* to change the dtype of Classlabel and have get_nested_type return a pyarrow.DictionaryType instead of int64? and b) do you *want* it to change?  The biggest challenge I see to implementing this correctly is that the data will need to be passed in along with the pyarrow schema when instantiating the Classlabel (I *think* this is unavoidable, since the type itself doesn't contain the actual label values) which could be a fairly intrusive change - e.g. `from_arrow_schema`'s interface would need to change to include optional arrow data?  Once we start going down this path of modifying the public interfaces I am admittedly feeling a little bit outside of my comfort zone

Additionally I think `int2str`, `str2int`, and `encode_example` probably won't work - but I can't find any usages of them in the library itself.
GitHub issue #1935: add CoVoST2
State: closed | Created: 2021-02-23T16:28:16Z

This PR adds the CoVoST2 dataset for speech translation and ASR.
https://github.com/facebookresearch/covost#covost-2

The dataset requires manual download as the download page requests an email address and the URLs are temporary.

The dummy data is a bit bigger because of the mp3 files and 36 configs.
GitHub issue #1934: Add Stanford Sentiment Treebank (SST)
State: closed | Created: 2021-02-23T12:53:16Z

I am going to add SST:

- **Name:** The Stanford Sentiment Treebank
- **Description:** The first corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language
- **Paper:** [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
- **Data:** https://nlp.stanford.edu/sentiment/index.html
- **Motivation:** Already requested in #353, SST is a popular dataset for Sentiment Classification

What's the difference with the [_SST-2_](https://huggingface.co/datasets/viewer/?dataset=glue&config=sst2) dataset included in GLUE? Essentially, SST-2 is a version of SST where:
- the labels were mapped from real numbers in [0.0, 1.0] to a binary label: {0, 1}
- the labels of the *sub-sentences* were included only in the training set
- the labels in the test set are obfuscated

So there is a lot more information in the original SST. The tricky bit is, the data is scattered into many text files and, for one in particular, I couldn't find the original encoding ([*but I'm not the only one*](https://groups.google.com/g/word2vec-toolkit/c/QIUjLw6RqFk/m/_iEeyt428wkJ) 🎵). The only solution I found was to manually replace all the è, ë, ç and so on into an `utf-8` copy of the text file. I uploaded the result in my Dropbox and I am using that as the main repo for the dataset.

Also, the _sub-sentences_ are built at run-time from the information encoded in several text files, so generating the examples is a bit more cumbersome than usual. Luckily, the dataset is not enormous.

I plan to divide the dataset in 2 configs: one with just whole sentences with their labels, the other with sentences _and their sub-sentences_ with their labels. Each config will be split in train, validation and test. Hopefully this makes sense, we may discuss it in the PR I'm going to submit.
GitHub issue #1933: Use arrow ipc file format
State: closed | Created: 2021-02-23T10:38:24Z

According to the [documentation](https://arrow.apache.org/docs/format/Columnar.html?highlight=arrow1#ipc-file-format), it's identical to the streaming format except that it contains the memory offsets of each sample:

> We define a “file format” supporting random access that is build with the stream format. The file starts and ends with a magic string ARROW1 (plus padding). What follows in the file is identical to the stream format. At the end of the file, we write a footer containing a redundant copy of the schema (which is a part of the streaming format) plus memory offsets and sizes for each of the data blocks in the file. This enables random access any record batch in the file. See File.fbs for the precise details of the file footer.

Since it stores more metadata regarding the positions of the examples in the file, it should enable better example retrieval performances. However from the discussion in https://github.com/huggingface/datasets/issues/1803 it looks like it's not the case unfortunately. Maybe in the future this will allow speed gains.

I think it's still a good idea to start using it anyway for these reasons:
- in the future we may have speed gains
- it contains the arrow streaming format data
- it's compatible with the pyarrow Dataset implementation (it allows to load remote dataframes for example) if we want to use it in the future
- it's also the format used by arrow feather if we want to use it in the future
- it's roughly the same size as the streaming format
- it's easy to have backward compatibility with the streaming format
GitHub issue #1932: Fix builder config creation with data_dir
State: closed | Created: 2021-02-23T10:26:02Z

The data_dir parameter wasn't taken into account to create the config_id, therefore the resulting builder config was considered not custom. However a builder config that is non-custom must not have a name that collides with the predefined builder config names. Therefore it resulted in a `ValueError("Cannot name a custom BuilderConfig the same as an available...")`

I fixed that by commenting the line that used to ignore the data_dir when creating the config.

It was previously ignored before the introduction of config id because we didn't want to change the config name. Now it's fine to take it into account for the config id.

Now creating a config with a data_dir works again @patrickvonplaten
GitHub issue #1929: Improve typing and style and fix some inconsistencies
State: closed | Created: 2021-02-22T22:47:41Z

This PR:
* improves typing (mostly more consistent use of `typing.Optional`)
* `DatasetDict.cleanup_cache_files` now correctly returns a dict 
* replaces `dict()` with the corresponding literal
* uses `dict_to_copy.copy()` instead of `dict(dict_to_copy)` for shallow copying
GitHub issue #1928: Updating old cards
State: closed | Created: 2021-02-22T19:26:04Z

Updated the cards for [Allocine](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/allocine), [CNN/DailyMail](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/cnn_dailymail), and [SNLI](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/snli). For the most part, the information was just rearranged or rephrased, but the social impact statements are new.
GitHub issue #1926: Fix: Wiki_dpr - add missing scalar quantizer
State: closed | Created: 2021-02-22T15:32:05Z

All the prebuilt wiki_dpr indexes already use SQ8, I forgot to update the wiki_dpr script after building them. Now it's finally done.

The scalar quantizer SQ8 doesn't reduce the performance of the index as shown in retrieval experiments on RAG.
The quantizer reduces the size of the index a lot but increases index building time.
GitHub issue #1925: Fix: Wiki_dpr - fix when with_embeddings is False or index_name is "no_index"
State: closed | Created: 2021-02-22T15:23:46Z

Fix the bugs noticed in #1915 

There was a bug when `with_embeddings=False` where the configuration name was the same as if `with_embeddings=True`, which led the dataset builder to do bad verifications (for example it used to expect to download the embeddings for `with_embeddings=False`).

Another issue was that setting `index_name="no_index"` didn't set `with_index` to False.

I fixed both of them and added dummy data for those configurations for testing.
GitHub issue #1923: Fix save_to_disk with relative path
State: closed | Created: 2021-02-22T10:27:19Z

As noticed in #1919 and #1920 the target directory was not created using `makedirs` so saving to it raises `FileNotFoundError`. For absolute paths it works but not for the good reason. This is because the target path was the same as the temporary path where in-memory data are written as an intermediary step.

I added the `makedirs` call using `fs.makedirs` in order to support remote filesystems.
I also fixed the issue with the target path being the temporary path.

I added a test case for relative paths as well for save_to_disk.

Thanks to @M-Salti for reporting and investigating
GitHub issue #1921: Standardizing datasets dtypes
State: closed | Created: 2021-02-20T22:04:01Z

This PR follows up on discussion in #1900 to have an explicit set of basic dtypes for datasets.

This moves away from str(pyarrow.DataType) as the method of choice for creating dtypes, favoring an explicit mapping to a list of supported Value dtypes.

I believe in practice this should be backward compatible, since anyone previously using Value() would only have been able to use dtypes that had an identically named pyarrow factory function, which are all explicitly supported here, with `float32` and `float64` acting as the official datasets dtypes, which resolves the tension between `double` being the pyarrow dtype and `float64` being the pyarrow type factory function.
GitHub issue #1919: Failure to save with save_to_disk
State: closed | Created: 2021-02-20T14:18:10Z

When I try to save a dataset locally using the `save_to_disk` method I get the error:

```bash
FileNotFoundError: [Errno 2] No such file or directory: '/content/squad/train/squad-train.arrow'
```

To replicate:

1. Install `datasets` from master
2. Run this code:

    ```python
    from datasets import load_dataset
    squad = load_dataset("squad")   # or any other dataset
    squad.save_to_disk("squad")     # error here
    ```

The problem is that the method is not creating a directory with the name `dataset_path` for saving the dataset in (i.e. it's not creating the *train* and *validation* directories in this case). After creating the directory the problem resolves.
I'll open a PR soon doing that and linking this issue.
GitHub issue #1917: UnicodeDecodeError: windows 10 machine
State: closed | Created: 2021-02-19T22:13:05Z

Windows 10
Php 3.6.8

when running

```
import datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])
```
I get the following error

```
file "C:\PYTHON\3.6.8\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 58: character maps to <undefined>
```
GitHub issue #1911: Saving processed dataset running infinitely
State: open | Created: 2021-02-19T13:09:19Z

I have a text dataset of size 220M.

For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 3hrs. I used map() with batch size 1024 and multi-process with 96 processes.

filter() function was way to slow, so I used a hack to use pyarrow filter table function, which is damm fast. Mentioned [here](https://github.com/huggingface/datasets/issues/1796)

```dataset._data = dataset._data.filter(...)```
It took 1 hr for the filter.

Then i use `save_to_disk()` on processed dataset and it is running forever.

I have been waiting since 8 hrs, it has not written a single byte. 

Infact it has actually read from disk more than 100GB, screenshot below shows the stats using `iotop`. 
Second process is the one.
<img width="1672" alt="Screenshot 2021-02-19 at 6 36 53 PM" src="https://user-images.githubusercontent.com/20911334/108508197-7325d780-72e1-11eb-8369-7c057d137d81.png">


I am not able to figure out, whether this is some issue with dataset library or that it is due to my hack for filter() function.
GitHub issue #1907: DBPedia14 Dataset Checksum bug?
State: closed | Created: 2021-02-18T22:25:48Z

Hi there!!!

I've been using successfully the DBPedia dataset (https://huggingface.co/datasets/dbpedia_14) with my codebase in the last couple of weeks, but in the last couple of days now I get this error:

```
Traceback (most recent call last):
  File "./conditional_classification/basic_pipeline.py", line 178, in <module>
    main()
  File "./conditional_classification/basic_pipeline.py", line 128, in main
    corpus.load_data(limit_train_examples_per_class=args.data_args.train_examples_per_class,
  File "/home/fp/dev/conditional_classification/conditional_classification/datasets_base.py", line 83, in load_data
    datasets = load_dataset(self.name, split=dataset_split)
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/load.py", line 609, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/builder.py", line 526, in download_and_prepare
    self._download_and_prepare(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/builder.py", line 586, in _download_and_prepare
    verify_checksums(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbQ2Vic1kxMmZZQ1k']
```

I've seen this has happened before in other datasets as reported in #537.

I've tried clearing my cache and call again `load_dataset` but still is not working. My same codebase is successfully downloading and using other datasets (e.g. AGNews) without any problem, so I guess something has happened specifically to the DBPedia dataset in the last few days. 

Can you please check if there's a problem with the checksums? 

Or this is related to any other stuff? I've seen that the path in the cache for the dataset is `/home/fp/.cache/huggingface/datasets/d_bpedia14/dbpedia_14/2.0.0/a70413e39e7a716afd0e90c9e53cb053691f56f9ef5fe317bd07f2c368e8e897...` and includes `d_bpedia14` instead maybe of `dbpedia_14`. Was this maybe a bug introduced recently?

Thanks!
GitHub issue #1906: Feature Request: Support for Pandas `Categorical`
State: open | Created: 2021-02-18T19:46:05Z

```
from datasets import Dataset
import pandas as pd
import pyarrow

df = pd.DataFrame(pd.Series(["a", "b", "c", "a"], dtype="category"))
pyarrow.Table.from_pandas(df)
Dataset.from_pandas(df)
# Throws NotImplementedError
# TODO(thom) this will need access to the dictionary as well (for labels). I.e. to the py_table
```

I'm curious if https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L796 could be built out in a way similar to `Sequence`?

e.g. a `Map` class (or whatever name the maintainers might prefer) that can accept:

```
index_type = generate_from_arrow_type(pa_type.index_type)
value_type = generate_from_arrow_type(pa_type.value_type)
```

and then additional code points to modify:

- FeatureType: https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L694
- A branch to handle Map in get_nested_type: https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L719
- I don't quite understand what `encode_nested_example` does but perhaps a branch there? https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L755
- Similarly, I don't quite understand why `Sequence` is used this way in `generate_from_dict`, but perhaps a branch here? https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L775

I couldn't find other usages of `Sequence` outside of defining specific datasets, so I'm not sure if that's a comprehensive set of touchpoints.
GitHub issue #1905: Standardizing datasets.dtypes
State: closed | Created: 2021-02-18T19:15:31Z

This PR was further branched off of jdy-str-to-pyarrow-parsing, so it depends on https://github.com/huggingface/datasets/pull/1900 going first for the diff to be up-to-date (I'm not sure if there's a way for me to use jdy-str-to-pyarrow-parsing as a base branch while having it appear in the pull requests here).

This moves away from `str(pyarrow.DataType)` as the method of choice for creating dtypes, favoring an explicit mapping to a list of supported Value dtypes.

I believe in practice this should be backward compatible, since anyone previously using Value() would only have been able to use dtypes that had an identically named pyarrow factory function, which are all explicitly supported here.
GitHub issue #1904: Fix to_pandas for boolean ArrayXD
State: closed | Created: 2021-02-18T16:30:46Z

As noticed in #1887 the conversion of a dataset with a boolean ArrayXD feature types fails because of the underlying ListArray conversion to numpy requires `zero_copy_only=False`.

zero copy is available for all primitive types except booleans
see https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy
and https://issues.apache.org/jira/browse/ARROW-2871?jql=text%20~%20%22boolean%20to_numpy%22

cc @SBrandeis
GitHub issue #1903: Initial commit for the addition of TIMIT dataset
State: closed | Created: 2021-02-18T14:23:12Z

Below points needs to be addressed:

- Creation of dummy dataset is failing
- Need to check on the data representation
- License is not creative commons. Copyright: Portions © 1993 Trustees of the University of Pennsylvania

Also the links (_except the download_) point to the ami corpus! ;-)

@patrickvonplaten  Requesting your comments, will be happy to address them!
GitHub issue #1900: Issue #1895: Bugfix for string_to_arrow timestamp[ns] support
State: closed | Created: 2021-02-17T20:26:04Z

Should resolve https://github.com/huggingface/datasets/issues/1895

The main part of this PR adds additional parsing in `string_to_arrow` to convert the timestamp dtypes that result from `str(pa_type)` back into the pa.DataType TimestampType.

While adding unit-testing, I noticed that support for the double/float types also don't invert correctly, so I added them, which I believe would hypothetically make this section of `Value` redundant:

```
    def __post_init__(self):
        if self.dtype == "double":  # fix inferred type
            self.dtype = "float64"
        if self.dtype == "float":  # fix inferred type
            self.dtype = "float32"
```

However, since I think Value.dtype is part of the public interface, removing that would result in a backward-incompatible change, so I didn't muck with that.

The rest of the PR consists of docstrings that I added while developing locally so I could keep track of which functions were supposed to be inverses of each other, and thought I'd include them initially in case you want to keep them around, but I'm happy to delete or remove any of them at your request!
GitHub issue #1898: ALT dataset has repeating instances in all splits
State: closed | Created: 2021-02-17T12:51:42Z

The [ALT](https://huggingface.co/datasets/alt) dataset has all the same instances within each split :/
Seemed like a great dataset for some experiments I wanted to carry out, especially since its medium-sized, and has all splits.

Would be great if this could be fixed :)

Added a snapshot of the contents from `explore-datset` feature, for quick reference.

![image](https://user-images.githubusercontent.com/33179372/108206321-442a2d00-714c-11eb-882f-b4b6e708ef9c.png)
GitHub issue #1897: Fix PandasArrayExtensionArray conversion to native type
State: closed | Created: 2021-02-17T11:48:24Z

To make the conversion to csv work in #1887 , we need  PandasArrayExtensionArray used for multidimensional numpy arrays to be converted to pandas native types.
However previously pandas.core.internals.ExtensionBlock.to_native_types would fail with an PandasExtensionArray because
1. the PandasExtensionArray.isna method was wrong
2. the conversion of a PandasExtensionArray to a numpy array with dtype=object was returning a multidimensional array while pandas excepts a 1D array in this case (more info [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.extensions.ExtensionArray.html#pandas.api.extensions.ExtensionArray))

I fixed these two issues and now the conversion to native types works, and so is the export to csv.
cc @SBrandeis
GitHub issue #1895: Bug Report: timestamp[ns] not recognized
State: closed | Created: 2021-02-16T20:38:04Z

Repro:

```
from datasets import Dataset
import pandas as pd
import pyarrow

df = pd.DataFrame(pd.date_range("2018-01-01", periods=3, freq="H"))
pyarrow.Table.from_pandas(df)
Dataset.from_pandas(df)
# Throws ValueError: Neither timestamp[ns] nor timestamp[ns]_ seems to be a pyarrow data type.
```

The factory function seems to be just "timestamp": https://arrow.apache.org/docs/python/generated/pyarrow.timestamp.html#pyarrow.timestamp

It seems like https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L36-L43 could have a little bit of additional structure for handling these cases?  I'd be happy to take a shot at opening a PR if I could receive some guidance on whether parsing something like `timestamp[ns]` and resolving it to timestamp('ns') is the goal of this method.

Alternatively, if I'm using this incorrectly (e.g. is the expectation that we always provide a schema when timestamps are involved?), that would be very helpful to know as well!

```
$ pip list  # only the relevant libraries/versions
datasets                      1.2.1
pandas                        1.0.3
pyarrow                       3.0.0
```
GitHub issue #1894: benchmarking against MMapIndexedDataset
State: open | Created: 2021-02-16T20:04:58Z

I am trying to benchmark my datasets based implementation against fairseq's [`MMapIndexedDataset`](https://github.com/pytorch/fairseq/blob/master/fairseq/data/indexed_dataset.py#L365) and finding that, according to psrecord, my `datasets` implem uses about 3% more CPU memory and runs 1% slower for `wikitext103` (~1GB of tokens).

Questions:
1) Is this (basically identical) performance expected? 
2) Is there a scenario where this library will outperform `MMapIndexedDataset`? (maybe more examples/larger examples?)
3) Should I be using different benchmarking tools than `psrecord`/how do you guys do benchmarks?

Thanks in advance! Sam
GitHub issue #1881: `list_datasets()` returns a list of strings, not objects
State: closed | Created: 2021-02-15T14:20:15Z

Here and there in the docs there is still stuff like this:

```python
>>> datasets_list = list_datasets()
>>> print(', '.join(dataset.id for dataset in datasets_list))
```

However, my understanding is that `list_datasets()` returns a list of strings rather than a list of objects.
GitHub issue #1879: Replace flatten_nested
State: closed | Created: 2021-02-15T13:29:40Z

Replace `flatten_nested` with `NestedDataStructure.flatten`.

This is a first step towards having all NestedDataStructure logic as a separated concern, independent of the caller/user of the data structure.

Eventually, all checks (whether the underlying data is list, dict, etc.) will be only inside this class.

I have also generalized the flattening, and now it handles multiple levels of nesting.
GitHub issue #1878: Add LJ Speech dataset
State: closed | Created: 2021-02-15T13:10:42Z

This PR adds the LJ Speech dataset (https://keithito.com/LJ-Speech-Dataset/)
As requested by #1841 
The ASR format is based on #1767 

There are a couple of quirks that should be addressed:
- I tagged this dataset as `other-other-automatic-speech-recognition` and `other-other-text-to-speech` (as classified by paperswithcode). Since the number of speech datasets is about to grow, maybe these categories should be added to the main list? 
- Similarly to #1767 this dataset uses only a single dummy sample to reduce the zip size (`wav`s are quite heavy). Is there a plan to allow LFS or S3 usage for dummy data in the repo?
- The dataset is distributed under the Public Domain license, which is not used anywhere else in the repo, AFAIK. Do you think Public Domain is worth adding to the tagger app as well?

Pinging @patrickvonplaten to review
GitHub issue #1877: Allow concatenation of both in-memory and on-disk datasets
State: closed | Created: 2021-02-15T11:39:46Z

This is a prerequisite for the addition of the `add_item` feature (see #1870).
Currently there is one assumption that we would need to change: a dataset is either fully in memory (dataset._data_files is empty), or the dataset can be reloaded from disk (using the dataset._data_files).
This assumption is used for pickling for example:
- in-memory dataset can just be pickled/unpickled in-memory
- on-disk dataset can be unloaded to only keep the filepaths when pickling, and then reloaded from the disk when unpickling

Maybe let's have a design that allows a Dataset to have a Table that can be rebuilt from heterogenous sources like in-memory tables or on-disk tables ? This could also be further extended in the future

One idea would be to define a list of sources and each source implements a way to reload its corresponding pyarrow Table.
Then the dataset would be the concatenation of all these tables.

Depending on the source type, the serialization using pickle would be different. In-memory data would be copied while on-disk data would simply be replaced by the path to these data.

If you have some ideas you would like to share about the design/API feel free to do so :)

cc @albertvillanova
GitHub issue #1876:  load_dataset("multi_woz_v22") NonMatchingChecksumError
State: closed | Created: 2021-02-14T19:14:48Z

Hi, it seems that loading the multi_woz_v22 dataset gives a NonMatchingChecksumError.

To reproduce:

`dataset = load_dataset('multi_woz_v22','v2.2_active_only',split='train')`


This will give the following error:

```
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/dialog_acts.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_001.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_003.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_004.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_005.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_006.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_007.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_008.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_009.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_010.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_012.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_013.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_014.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_015.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_016.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/train/dialogues_017.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/dev/dialogues_001.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/dev/dialogues_002.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/test/dialogues_001.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/test/dialogues_002.json']
```
GitHub issue #1874: Adding Europarl Bilingual dataset
State: closed | Created: 2021-02-13T17:02:04Z

Implementation of Europarl bilingual dataset from described [here](https://opus.nlpl.eu/Europarl.php).

This dataset allows to use every language pair detailed in the original dataset. The loading script manages also the small errors contained in the original dataset (in very rare cases (1 over 10M) there are some keys that references to inexistent sentences).
I chose to follow the the style of a similar dataset available in this repository: `multi_para_crawl`.
GitHub issue #1873: add iapp_wiki_qa_squad
State: closed | Created: 2021-02-13T13:34:27Z

`iapp_wiki_qa_squad` is an extractive question answering dataset from Thai Wikipedia articles.
It is adapted from [the original iapp-wiki-qa-dataset](https://github.com/iapp-technology/iapp-wiki-qa-dataset)
to [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format, resulting in
5761/742/739 questions from 1529/191/192 articles.
GitHub issue #1868: Update oscar sizes
State: closed | Created: 2021-02-12T10:55:35Z

This commit https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383 removed empty lines from the oscar deduplicated datasets. This PR updates the size of each deduplicated dataset to fix possible `NonMatchingSplitsSizesError` errors. cc @cahya-wirawan
GitHub issue #1867: ERROR WHEN USING SET_TRANSFORM() 
State: closed | Created: 2021-02-12T10:38:31Z

Hi, I'm trying to use dataset.set_transform(encode) as @lhoestq told me in this issue: https://github.com/huggingface/datasets/issues/1825#issuecomment-774202797

However, when I try to use Trainer from transformers with such dataset, it throws an error:

```
TypeError: __init__() missing 1 required positional argument: 'transform'
[INFO|trainer.py:357] 2021-02-12 10:18:09,893 >> The following columns in the training set don't have a corresponding argument in `AlbertForMaskedLM.forward` and have been ignored: text.
Exception in device=TPU:0: __init__() missing 1 required positional argument: 'transform'
Traceback (most recent call last):
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 330, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 324, in _start_fn
    fn(gindex, *args)
  File "/home/alejandro_vaca/transformers/examples/language-modeling/run_mlm_wwm.py", line 368, in _mp_fn
    main()
  File "/home/alejandro_vaca/transformers/examples/language-modeling/run_mlm_wwm.py", line 332, in main
    data_collator=data_collator,
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 286, in __init__
    self._remove_unused_columns(self.train_dataset, description="training")
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 359, in _remove_unused_columns
    dataset.set_format(type=dataset.format["type"], columns=columns)
  File "/home/alejandro_vaca/datasets/src/datasets/fingerprint.py", line 312, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/alejandro_vaca/datasets/src/datasets/arrow_dataset.py", line 818, in set_format
    _ = get_formatter(type, **format_kwargs)
  File "/home/alejandro_vaca/datasets/src/datasets/formatting/__init__.py", line 112, in get_formatter
    return _FORMAT_TYPES[format_type](**format_kwargs)
TypeError: __init__() missing 1 required positional argument: 'transform'
```

The code I'm using:

```{python}

    def tokenize_function(examples):
        # Remove empty lines
        examples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]
        return tokenizer(examples["text"], padding=padding, truncation=True, max_length=data_args.max_seq_length)

    datasets.set_transform(tokenize_function)

    data_collator = DataCollatorForWholeWordMask(tokenizer=tokenizer, mlm_probability=data_args.mlm_probability)

    # Initialize our Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=datasets["train"] if training_args.do_train else None,
        eval_dataset=datasets["val"] if training_args.do_eval else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
```

I've installed from source, master branch.