Added metadata and correct splits for swda.

@lhoestq thank you for your comments! I went ahead and fixed the code 😃. Please let me know if I missed anything.

Switchboard Dialog Act Corpus

I made some changes following  @bhavitvyamalik recommendation in #1678:

* Contains all metadata.
* Used official implementation from the [/swda](https://github.com/cgpotts/swda) repo.
* Add official train and test splits used in [Stolcke et al. (2000)](https://web.stanford.edu/~jurafsky/ws97) and validation split used in [Probabilistic-RNN-DA-Classifier](https://github.com/NathanDuran/Probabilistic-RNN-DA-Classifier).


add Stuctured Argument Extraction for Korean dataset


datasets slicing with seed

Hi :)
The slicing API from https://huggingface.co/docs/datasets/splits.html doesn't shuffle the data.
You can shuffle and then take a subset of your dataset with
```python
# shuffle and take the first 100 examples
dataset = dataset.shuffle(seed=42).select(range(100))
```

You can find more information about shuffling and selecting rows in the documentation: https://huggingface.co/docs/datasets/processing.html#selecting-sorting-shuffling-splitting-rows

thank you so much

On Mon, Jan 18, 2021 at 3:17 PM Quentin Lhoest <notifications@github.com>
wrote:

> Hi :)
> The slicing API doesn't shuffle the data.
> You can shuffle and then take a subset of your dataset with
>
> # shuffle and take the first 100 examplesdataset = dataset.shuffle(seed=42).select(range(100))
>
> You can find more information about shuffling and selecting rows in the
> documentation:
> https://huggingface.co/docs/datasets/processing.html#selecting-sorting-shuffling-splitting-rows
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1747#issuecomment-762278134>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM3GZM5D5MDPLJGI4IG3UADS2Q7GPANCNFSM4WHLOZJQ>
> .
>

Hi
I need to slice a dataset with random seed, I looked into documentation here https://huggingface.co/docs/datasets/splits.html
I could not find a seed option, could you assist me please how I can get a slice for different seeds?
thank you.
@lhoestq


Fix release conda worflow

The current workflow yaml file is not valid according to https://github.com/huggingface/datasets/actions/runs/487638110


difference between wsc and wsc.fixed for superglue

From the description given in the dataset script for `wsc.fixed`:
```
This version fixes issues where the spans are not actually substrings of the text.
```

Hi
I see two versions of wsc in superglue, and I am not sure what is the differences and which one is the original one. could you help to discuss the differences? thanks @lhoestq


Add missing "brief" entries to reuters

@lhoestq I ran `make style` but CI code quality still failing and I don't have access to logs

It's also likely that due to the previous placement of the field initialization, much of the data about topics etc was simply wrong and carried over from previous entries. Model scores seem to improve significantly with this PR.

This brings the number of examples for ModApte to match the stated `Training set (9,603 docs)...Test Set (3,299 docs)`


Issue while Creating Custom Metric

Currently it's only possible to define the features for the two columns `references` and `predictions`.
The data for these columns can then be passed to `metric.add_batch` and `metric.compute`.
Instead of defining more columns `text`, `offset_mapping` and `ground` you must include them in either references and predictions.

For example
```python
features = datasets.Features({
    'predictions':datasets.Sequence(datasets.Value("int32")),
    "references": datasets.Sequence({
        "references_ids": datasets.Value("int32"),
        "offset_mapping": datasets.Value("int32"),
        'text': datasets.Value('string'),
        "ground": datasets.Value("int32")
    }),
})
```

Another option would be to simply have the two features like
```python
features = datasets.Features({
    'predictions':datasets.Sequence(datasets.Value("int32")),
    "references": datasets.Sequence(datasets.Value("int32")),
})
```
and keep `offset_mapping`, `text` and `ground` as as parameters for the computation (i.e. kwargs when calling `metric.compute`).

What is the metric you would like to implement ?

I'm asking since we consider allowing additional fields as requested in the `Comet` metric (see PR and discussion [here](https://github.com/huggingface/datasets/pull/1577)) and I'd like to know if it's something that can be interesting for users.

What do you think ?

Hi @lhoestq,

I am doing text segmentation and the metric is effectively dice score on character offsets. So I need to pass the actual spans and I want to be able to get the spans based on predictions using offset_mapping.

Including them in references seems like a good idea. I'll try it out and get back to you. If there's a better way to write a metric function for the same, please let me know.

Hi Team,

I am trying to create a custom metric for my training as follows, where f1 is my own metric:

```python
 def _info(self):
        # TODO: Specifies the datasets.MetricInfo object
        return datasets.MetricInfo(
            # This is the description that will appear on the metrics page.
            description=_DESCRIPTION,
            citation=_CITATION,
            inputs_description=_KWARGS_DESCRIPTION,
            # This defines the format of each prediction and reference
            features = datasets.Features({'predictions':datasets.Sequence(datasets.Value("int32")), "references": datasets.Sequence(datasets.Value("int32")),"offset_mapping":datasets.Sequence(datasets.Value("int32")),'text':datasets.Sequence(datasets.Value('string')),"ground":datasets.Sequence(datasets.Value("int32")),}),
            # Homepage of the metric for documentation
            homepage="http://metric.homepage",
            # Additional links to the codebase or references
            codebase_urls=["http://github.com/path/to/codebase/of/new_metric"],
            reference_urls=["http://path.to.reference.url/new_metric"]
        )

    def _compute(self,predictions,references,text,offset_mapping,spans):

        pred_spans = []

        for i,preds in enumerate(predictions):
            current_preds = []
            for j,token_preds in enumerate(preds):
                if (preds>0.5):
                    current_preds+=list(range(offset_mapping[i][j][0],offset_mapping[i][j][1]))
            pred_spans.append(current_spans)

        return {
            "Token Wise F1": f1_score(references,predictions,labels=[0,1]),
            "Offset Wise F1": np.mean([f1(preds,gold) for preds,fold in zip(pred_spans,ground)])
        }

```

I believe this is not correct. But that's not the issue I am facing right now. I get this error :
```python
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-144-ed7349b50821> in <module>()
----> 1 new_metric.compute(predictions=inputs["labels"],references=inputs["labels"], text=inputs["text"], offset_mapping=inputs["offset_mapping"],ground=inputs["ground"] )

2 frames
/usr/local/lib/python3.6/dist-packages/datasets/features.py in encode_batch(self, batch)
    802         encoded_batch = {}
    803         if set(batch) != set(self):
--> 804             print(batch)
    805             print(self)
    806             raise ValueError("Column mismatch between batch {} and features {}".format(set(batch), set(self)))

ValueError: Column mismatch between batch {'references', 'predictions'} and features {'ground', 'predictions', 'offset_mapping', 'text', 'references'}
```
On checking the features.py file, I see the call is made from add_batch() in metrics.py which only takes in predictions and references.

How do I make my custom metric work? Will it work with a trainer even if I am able to make this metric work?

Thanks,
Gunjan


Add GLUE Compat (compatible with transformers<3.5.0)

Maybe it would be simpler to just overwrite the order of the label classes of the `glue` dataset ?
```python
mnli = load_dataset("glue", "mnli", label_classes=["contradiction", "entailment", "neutral"])
```

Sounds good. Will close the issue if that works.

Link to our discussion on Slack (HF internal)
https://huggingface.slack.com/archives/C014N4749J9/p1609668119337400

The next step is to add a compatible option in the new `run_glue.py`

I duplicated `glue` and made the following changes:
1. Change the name to `glue_compat`.
2. Change the label assignments for MNLI and AX.


error when run fine_tuning on text_classification

dataset:sem_eval_2014_task_1
pretrained_model:bert-base-uncased

error description:
when i use these resoruce to train fine_tuning a text_classification on sem_eval_2014_task_1,there always be some problem(when i use other dataset ,there exist the error too). And i followed the colab code (url:https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb#scrollTo=TlqNaB8jIrJW).

the error is like this :
`File "train.py", line 69, in <module>
    trainer.train()
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/transformers/trainer.py", line 784, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
KeyError: 2`

this is my code :
```dataset_name = 'sem_eval_2014_task_1'
num_labels_size = 3
batch_size = 4
model_checkpoint = 'bert-base-uncased'
number_train_epoch = 5

def tokenize(batch):
return tokenizer(batch['premise'], batch['hypothesis'], truncation=True, )

def compute_metrics(pred):
labels = pred.label_ids
preds = pred.predictions.argmax(-1)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='micro')
acc = accuracy_score(labels, preds)
return {
'accuracy': acc,
'f1': f1,
'precision': precision,
'recall': recall
}

model = BertForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels_size)
tokenizer = BertTokenizerFast.from_pretrained(model_checkpoint, use_fast=True)

train_dataset = load_dataset(dataset_name, split='train')
test_dataset = load_dataset(dataset_name, split='test')

train_encoded_dataset = train_dataset.map(tokenize, batched=True)
test_encoded_dataset = test_dataset.map(tokenize, batched=True)

args = TrainingArguments(
output_dir='./results',
evaluation_strategy="epoch",
learning_rate=2e-5,
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=batch_size,
num_train_epochs=number_train_epoch,
weight_decay=0.01,
do_predict=True,
)
trainer = Trainer(
model=model,
args=args,
compute_metrics=compute_metrics,
train_dataset=train_encoded_dataset,
eval_dataset=test_encoded_dataset,
tokenizer=tokenizer
)

trainer.train()
trainer.evaluate()


add id_liputan6 dataset

id_liputan6 is a large-scale Indonesian summarization dataset. The articles were harvested from an online news portal, and obtain 215,827 document-summary pairs: https://arxiv.org/abs/2011.00679


fixes and improvements for the WebNLG loader

The dataset card is fantastic!

Looks good to me! Did you check that this still passes the slow tests with the existing dummy data?

Yes, I ran and passed all the tests specified in [this guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#automatically-add-code-metadata), including the slow ones.

I just added the `from pathlib import Path` at the top to fix the script

I ran the tests locally and they all pass, merging

- fixes test sets loading in v3.0
- adds additional fields for v3.0_ru
- adds info to the WebNLG data card


Conda support

Nice thanks :)
Note that in `datasets` the tags are simply the version without the `v`. For example `1.2.1`.

Yes I've always used tags only for versions

Will push a new version on anaconda cloud every time a tag starting with `v` is pushed (like `v1.2.2`).

Will appear here: https://anaconda.org/huggingface/datasets

Depends on `conda-forge` for now, so the following is required for installation:

```
conda install -c huggingface -c conda-forge datasets
```


update link in TLC to be github links

Base on this issue https://github.com/huggingface/datasets/issues/1064, I can now use the official links.


Adjust BrWaC dataset features name

I added this dataset some days ago, and today I used it to train some models and realized that the names of the features aren't so good.

Looking at the current features hierarchy, we have "paragraphs" with a list of "sentences" with a list of "sentences?!". But the actual hierarchy is a "text" with a list of "paragraphs" with a list of "sentences".

I confused myself trying to use the dataset with these names. So I think it's better to change it.


Update add new dataset template

Add new "dataset"? ;)
Lol, too used to Transformers ;-)

This PR fixes a few typos in the "Add new dataset template" and clarifies a bit what to do for the dummy data creation when the `auto_generate` flag can't work.


Fix empty token bug for `thainer` and `lst20`

add a condition to check if tokens exist before yielding in `thainer` and `lst20`


connection issue with glue, what is the data url for glue?

Hello @juliahane, which config of GLUE causes you trouble?
The URLs are defined in the dataset script source code: https://github.com/huggingface/datasets/blob/master/datasets/glue/glue.py

Hi
my codes sometimes fails due to connection issue with glue, could you tell me how I can have the URL datasets library is trying to read GLUE from to test the machines I am working on if there is an issue on my side or not
thanks


[GEM Dataset] Added TurkCorpus, an evaluation dataset for sentence simplification.

Thank you for the feedback! I updated the code.

We want to use TurkCorpus for validation and testing of the sentence simplification task.


Couldn't reach swda.py

Hi @yangp725,
The SWDA has been added very recently and has not been released yet, thus it is not available in the `1.2.0` version of 🤗`datasets`.
You can still access it by installing the latest version of the library (master branch), by following instructions in [this issue](https://github.com/huggingface/datasets/issues/1641#issuecomment-751571471).
Let me know if this helps !

Thanks @SBrandeis ,
Problem solved by downloading and installing the latest version datasets.

ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.0/datasets/swda/swda.py


Add MNIST dataset

This PR adds the MNIST dataset to the library.


Is there support for Deep learning datasets?

Hi @ZurMaD!
Thanks for your interest in 🤗 `datasets`. Support for image datasets is at an early stage, with CIFAR-10 added in #1617
MNIST is also on the way: #1730

If you feel like adding another image dataset, I would advise starting by reading the [ADD_NEW_DATASET.md](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) guide. New datasets are always very much appreciated 🚀

I looked around this repository and looking the datasets I think that there's no support for images-datasets. Or am I missing something? For example to add a repo like this https://github.com/DZPeru/fish-datasets


Add an entry to an arrow dataset

Hi @ameet-1997,
I think what you are looking for is the `concatenate_datasets` function: https://huggingface.co/docs/datasets/processing.html?highlight=concatenate#concatenate-several-datasets

For your use case, I would use the [`map` method](https://huggingface.co/docs/datasets/processing.html?highlight=concatenate#processing-data-with-map) to transform the SQuAD sentences and the `concatenate` the original and mapped dataset.

Let me know If this helps!

That's a great idea! Thank you so much!

When I try that solution, I get the following error when I try to concatenate `datasets` and `modified_dataset`. I have also attached the output I get when I print out those two variables. Am I missing something?

Code:
``` python
combined_dataset = concatenate_datasets([datasets, modified_dataset])
```

Error:
```
AttributeError: 'DatasetDict' object has no attribute 'features'
```

Output:
```
(Pdb) datasets
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'special_tokens_mask'],
        num_rows: 493
    })
})
(Pdb) modified_dataset
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'special_tokens_mask'],
        num_rows: 493
    })
})
```

The error is stemming from the fact that the attribute `datasets.features` does not exist. Would it not be possible to use `concatenate_datasets` in such a case? Is there an alternate solution?

You should do `combined_dataset = concatenate_datasets([datasets['train'], modified_dataset['train']])`

Didn't we talk about returning a Dataset instead of a DatasetDict with load_dataset and no split provided @lhoestq? Not sure it's the way to go but I'm wondering if it's not simpler for some use-cases.

> Didn't we talk about returning a Dataset instead of a DatasetDict with load_dataset and no split provided @lhoestq? Not sure it's the way to go but I'm wondering if it's not simpler for some use-cases.

My opinion is that users should always know in advance what type of objects they're going to get. Otherwise the development workflow on their side is going to be pretty chaotic with sometimes unexpected behaviors.
For instance is `split=` is not specified it's currently always returning a DatasetDict. And if `split="train"` is given for example it's always returning a Dataset.

Is it possible to add an entry to a dataset object?

**Motivation: I want to transform the sentences in the dataset and add them to the original dataset**

For example, say we have the following code:

``` python
from datasets import load_dataset

# Load a dataset and print the first examples in the training set
squad_dataset = load_dataset('squad')
print(squad_dataset['train'][0])
```

Is it possible to add an entry to `squad_dataset`? Something like the following?

``` python
squad_dataset.append({'text': "This is a new sentence"})
```

The motivation for doing this is that I want to transform the sentences in the squad dataset and add them to the original dataset.

If the above doesn't work, is there any other way of achieving the motivation mentioned above? Perhaps by creating a new arrow dataset by using the older one and the transformer sentences?


BLEURT score calculation raises UnrecognizedFlagError

Upgrading tensorflow to version 2.4.0 solved the issue.

I still have the same error even with TF 2.4.0.

And I have the same error with TF 2.4.1. I believe this issue should be reopened. Any ideas?!

I'm seeing the same issue with TF 2.4.1 when running the following in https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb:
```
!pip install git+https://github.com/google-research/bleurt.git
references = ["foo bar baz", "one two three"]
bleurt_metric = load_metric('bleurt')
predictions =  ["foo bar", "four five six"]
bleurt_metric.compute(predictions=predictions, references=references)
```

@aleSuglia @oscartackstrom - Are you getting the error when running your code in a Jupyter notebook ?

I tried reproducing this error again, and was unable to do so from the python command line console in a virtual environment similar to the one I originally used (and unfortunately no longer have access to) when I first got the error.
However, I've managed to reproduce the error by running the same code in a Jupyter notebook running a kernel from the same virtual environment.
This made me suspect that the problem is somehow related to the Jupyter notebook.

More environment details:
```
OS: Ubuntu Linux 18.04
conda==4.8.3
python==3.8.5
datasets==1.3.0
tensorflow==2.4.0
BLEURT==0.0.1
notebook==6.2.0
```

This happens when running the notebook on colab. The issue seems to be that colab populates sys.argv with arguments not handled by bleurt.

Running this before calling bleurt fixes it:
```
import sys
sys.argv = sys.argv[:1]
```

Not the most elegant solution. Perhaps it needs to be fixed in the bleurt code itself rather than huggingface?

This is the output of `print(sys.argv)` when running on colab:
```
['/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py', '-f', '/root/.local/share/jupyter/runtime/kernel-a857a78c-44d6-4b9d-b18a-030b858ee327.json']
```

I got the error when running it from the command line. It looks more like an error that should be fixed in the BLEURT codebase.

Seems to be a known issue in the bleurt codebase: https://github.com/google-research/bleurt/issues/24.

Calling the `compute` method for **bleurt** metric fails with an `UnrecognizedFlagError` for `FLAGS.bleurt_batch_size`.

My environment:
```
python==3.8.5
datasets==1.2.0
tensorflow==2.3.1
cudatoolkit==11.0.221
```

Test code for reproducing the error:
```
from datasets import load_metric
bleurt = load_metric('bleurt')
gen_text = "I am walking on the promenade today"
ref_text = "I am walking along the promenade on this sunny day"
bleurt.compute(predictions=[test_text], references=[test_text])
```

Error Output:
```
Using default BLEURT-Base checkpoint for sequence maximum length 128. You can use a bigger model for better results with e.g.: datasets.load_metric('bleurt', 'bleurt-large-512').
INFO:tensorflow:Reading checkpoint /home/ubuntu/.cache/huggingface/metrics/bleurt/default/downloads/extracted/9aee35580225730ac5422599f35c4986e4c49cafd08082123342b1019720dac4/bleurt-base-128.
INFO:tensorflow:Config file found, reading.
INFO:tensorflow:Will load checkpoint bert_custom
INFO:tensorflow:Performs basic checks...
INFO:tensorflow:... name:bert_custom
INFO:tensorflow:... vocab_file:vocab.txt
INFO:tensorflow:... bert_config_file:bert_config.json
INFO:tensorflow:... do_lower_case:True
INFO:tensorflow:... max_seq_length:128
INFO:tensorflow:Creating BLEURT scorer.
INFO:tensorflow:Loading model...
INFO:tensorflow:BLEURT initialized.
---------------------------------------------------------------------------
UnrecognizedFlagError                     Traceback (most recent call last)
<ipython-input-12-8b3f4322318a> in <module>
      2 gen_text = "I am walking on the promenade today"
      3 ref_text = "I am walking along the promenade on this sunny day"
----> 4 bleurt.compute(predictions=[gen_text], references=[ref_text])

~/anaconda3/envs/noved/lib/python3.8/site-packages/datasets/metric.py in compute(self, *args, **kwargs)
    396             references = self.data["references"]
    397             with temp_seed(self.seed):
--> 398                 output = self._compute(predictions=predictions, references=references, **kwargs)
    399
    400             if self.buf_writer is not None:

~/.cache/huggingface/modules/datasets_modules/metrics/bleurt/b1de33e1cbbcb1dbe276c887efa1fad68c6aff913885108078fa1ad408908778/bleurt.py in _compute(self, predictions, references)
    103
    104     def _compute(self, predictions, references):
--> 105         scores = self.scorer.score(references=references, candidates=predictions)
    106         return {"scores": scores}

~/anaconda3/envs/noved/lib/python3.8/site-packages/bleurt/score.py in score(self, references, candidates, batch_size)
    164     """
    165     if not batch_size:
--> 166       batch_size = FLAGS.bleurt_batch_size
    167
    168     candidates, references = list(candidates), list(references)

~/anaconda3/envs/noved/lib/python3.8/site-packages/tensorflow/python/platform/flags.py in __getattr__(self, name)
     83     # a flag.
     84     if not wrapped.is_parsed():
---> 85       wrapped(_sys.argv)
     86     return wrapped.__getattr__(name)
     87

~/anaconda3/envs/noved/lib/python3.8/site-packages/absl/flags/_flagvalues.py in __call__(self, argv, known_only)
    643     for name, value in unknown_flags:
    644       suggestions = _helpers.get_flag_suggestions(name, list(self))
--> 645       raise _exceptions.UnrecognizedFlagError(
    646           name, value, suggestions=suggestions)
    647

UnrecognizedFlagError: Unknown command line flag 'f'
```

Possible Fix:
Modify `_compute` method https://github.com/huggingface/datasets/blob/7e64851a12263dc74d41c668167918484c8000ab/metrics/bleurt/bleurt.py#L104
to receive a `batch_size` argument, for example:
```
def _compute(self, predictions, references, batch_size=1):
    scores = self.scorer.score(references=references, candidates=predictions, batch_size=batch_size)
    return {"scores": scores}
```


Offline loading

It's maybe a bit annoying to add but could we maybe have as well a version of the local data loading scripts in the package?
The `text`, `json`, `csv`. Thinking about people like in #1725 who are expecting to be able to work with local data without downloading anything.

Maybe we can add them to package_data or something?

Yes I mentioned this in #824 as well. I'm looking into it

Alright now `csv`, `json`, `text` and `pandas` are "packaged datasets", i.e. they're part of the `datasets` package, which makes them available in offline mode without any change in terms of API:
```python
from datasets import load_dataset

d = load_dataset("csv", data_files=["path/to/data.csv"])
```

Instead of loading the dataset script from the module cache, it's loaded from inside the `datasets` package.

I updated the test to still be able to fetch the dummy data files for those datasets from `datasets/{text|csv|pandas|json}/dummy` in the repo.

Alright now all test pass :)
(I don't thank you windows)

> LGTM! Since you're getting the local script's last modification date anyways do you think it might be a good idea to show it in the warning?

Yep good idea. I added the date in the warning. For example `(last modified on Mon Nov 30 11:01:56 2020)`

As discussed in #824 it would be cool to make the library work in offline mode.
Currently if there's not internet connection then modules (datasets or metrics) that have already been loaded in the past can't be loaded and it raises a ConnectionError.
This is because `prepare_module` fetches online for the latest version of the module.

To make it work in offline mode one suggestion was to reload the latest local version of the module.
I implemented that and I also raise a warning saying that the module that is loaded is the latest local version.
```python
logger.warning(
    f"Using the latest cached version of the module from {cached_module_path} since it "
    f"couldn't be found locally at {input_path} or remotely ({error_type_that_prevented_reaching_out_remote_stuff})."
)
```

I added tests to make sure it works as expected and I needed to do a few changes in the code to be able to test things properly. In particular I added a parameter `hf_modules_cache` to `init_dynamic_modules` for testing purposes. It makes it possible to have temporary modules caches for testing.

I also added a `offline` context utility that allows to test part of the code by making all the requests fail as if there was no internet.

Close #824, close #761.


load the local dataset

You should rephrase your question or give more examples and details on what you want to do.

it’s not possible to understand it and help you with only this information.

sorry for that.
i want to know how could i load the train set and the test set from the local ,which api or function should i use .

Did you try to follow the instructions in the documentation?
Here: https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files

thanks a lot
i find that the problem is i dont use vpn...
so i have to keep my net work even if i want to load the local data ?

your guidebook's example is like
>>>from datasets import load_dataset
>>> dataset = load_dataset('json', data_files='my_file.json')
but the first arg is path...
so how should i do if i want to load the local dataset for model training?
i will be grateful if you can help me handle this problem!
thanks a lot!


ADD S3 support for downloading and uploading processed datasets

I created the documentation for `FileSystem Integration for cloud storage` with loading and saving datasets to/from a filesystem with an example of using `datasets.filesystem.S3Filesystem`. I added a note on the `Saving a processed dataset on disk and reload` saying that it is also possible to use other filesystems and cloud storages such as S3 with a link to the newly created documentation page from me.
I Attach a screenshot of it here.
![screencapture-localhost-5500-docs-build-html-filesystems-html-2021-01-19-17_16_10](https://user-images.githubusercontent.com/32632186/105062131-8d6a5c80-5a7a-11eb-90b0-f6128b758605.png)

# What does this PR do?

This PR adds the functionality to load and save `datasets` from and to s3.
You can save `datasets` with either `Dataset.save_to_disk()` or `DatasetDict.save_to_disk`.
You can load `datasets` with either `load_from_disk` or `Dataset.load_from_disk()`, `DatasetDict.load_from_disk()`.

Loading `csv` or `json` datasets from s3 is not implemented.

To save/load datasets to s3 you either need to provide an `aws_profile`, which is set up on your machine, per default it uses the `default` profile or you have to pass an `aws_access_key_id` and `aws_secret_access_key`.

The implementation was done with the `fsspec` and `boto3`.

### Example `aws_profile` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")
```

</details>

### Example `aws_access_key_id` and `aws_secret_access_key` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk",
                            aws_access_key_id="fake_access_key",
                            aws_secret_access_key="fake_secret_key"
                           )

load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk",
                            aws_access_key_id="fake_access_key",
                            aws_secret_access_key="fake_secret_key"
                           )
```

</details>

If you want to load a dataset from a public s3 bucket you can pass `anon=True`

### Example  `anon=True` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

load_from_disk("s3://moto-mock-s3-bucketdatasets/sdk",anon=True)
```

</details>

### Full Example

```python
import datasets

dataset = datasets.load_dataset("imdb")
print(f"DatasetDict contains {len(dataset)} datasets")
print(f"train Dataset has the size of: {len(dataset['train'])}")

dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

remote_dataset = datasets.load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")
print(f"DatasetDict contains {len(remote_dataset)} datasets")
print(f"train Dataset has the size of: {len(remote_dataset['train'])}")
```

Related to #878

I would also adjust the documentation after the code would be reviewed, as long as I leave the PR in "draft" status. Something that we can consider is renaming the functions and changing the `_disk` maybe to `_filesystem`


could not run models on a offline server successfully

Transferred to `datasets` based on the stack trace.

Hi @lkcao !
Your issue is indeed related to `datasets`. In addition to installing the package manually, you will need to download the `text.py` script on your server. You'll find it (under `datasets/datasets/text`: https://github.com/huggingface/datasets/blob/master/datasets/text/text.py.
Then you can change the line 221 of `run_mlm_new.py` into:
```python
  datasets = load_dataset('/path/to/text.py', data_files=data_files)
```
Where `/path/to/text.py` is the path on the server where you saved the `text.py` script.

We're working on including the local dataset builders (csv, text, json etc.) directly in the `datasets` package so that they can be used offline

The local dataset builders (csv, text , json and pandas) are now part of the `datasets` package since #1726 :)
You can now use them offline
```python
datasets = load_dataset('text', data_files=data_files)
```

We'll do a new release soon

> The local dataset builders (csv, text , json and pandas) are now part of the `datasets` package since #1726 :)
> You can now use them offline
>
> ```python
> datasets = load_dataset('text', data_files=data_files)
> ```
>
> We'll do a new release soon

so the new version release now?

Yes it's been available since datasets 1.3.0 !

Hi, I really need your help about this.
I am trying to fine-tuning a RoBERTa on a remote server, which is strictly banning internet. I try to install all the packages by hand and try to run run_mlm.py on the server. It works well on colab, but when I try to run it on this offline server, it shows:
![image](https://user-images.githubusercontent.com/49967236/104276256-25a88600-546a-11eb-9776-8ec695dfa24e.png)

is there anything I can do? Is it possible to download all the things in cache and upload it to the server? Please help me out...


Added unfiltered versions of the Wiki-Auto training data for the GEM simplification task.

The current version of Wiki-Auto dataset contains a filtered version of the aligned dataset. The commit adds unfiltered versions of the data that can be useful the GEM task participants.


[Scientific papers] Mirror datasets zip

> Nice !
>
> Could you try to reduce the size of the dummy_data.zip files ? they're quite big (300KB)

Yes, I think it might make sense to enhance the tool a tiny bit to prevent this automatically

That's the lightest I can make it...it's long-range summarization so a single sample has ~11000 tokens.

Datasets were uploading to https://s3.amazonaws.com/datasets.huggingface.co/scientific_papers/1.1.1/arxiv-dataset.zip and https://s3.amazonaws.com/datasets.huggingface.co/scientific_papers/1.1.1/pubmed-dataset.zip respectively to escape google drive quota and enable faster download.


Adding the NorNE dataset for NER

Quick question, @lhoestq. In this specific dataset, two special types `GPE_LOC` and `GPE_ORG` can easily be altered depending on the task, choosing either the more general `GPE` tag or the more specific `LOC`/`ORG` tags, conflating them with the other annotations of the same type. However, I have not found an easy way to implement that. Using splits or configs does not seem appropriate.

About the `GPE_LOC` and `GPE_ORG`. The original NorNE paper in which they published the dataset, does an evaluation on three different NER tag sets, one considering `GPE_LOC` and `GPE_ORG` as they are, another changing them to be just `GPE`, and another one by changing it to become `LOC` and `ORG`. The called these sets, `norne-full`, `norne-7`, and `norne-9`. What I would like is to provide a way for the user of this dataset to get `norne-7` and `norne-9` without having to duplicate the code.

Ok I see !
I guess you can have three configurations  `norne-full`, `norne-7` and `norne-9`.
Each config can have different feature types. You can simply check for the `self.config.name` in the `_info(self)` method and pick the right ClassLabel names accordingly. And then in `_generate_examples` as well you can check for `self.config.name` to know how to process the labels to yield either GPE_LOC/GPE_ORG, GPE or LOC/ORG

But I'm already using the configurations for the different language
varieties. So you propose having something like `bokmaal`, `bokmaal-7`,
etc? Would there be a different way? If not, I'd be fine the corpus as it
is until we come up with a solution. Thanks in any case.

--
Sent using a cell-phone, so sorry for the typos and wrong auto-corrections.

On Tue, Jan 19, 2021, 4:56 PM Quentin Lhoest <notifications@github.com>
wrote:

> Ok I see !
> I guess you can have three configurations norne-full, norne-7 and norne-9.
> Each config can have different feature types. You can simply check for the
> self.config.name in the _info(self) method and pick the right ClassLabel
> names accordingly. And then in _generate_examples as well you can check
> for self.config.name to know how to process the labels to yield either
> GPE_LOC/GPE_ORG, GPE or LOC/ORG
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/1720#issuecomment-762936612>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABKLYOWNDBD76WZPJHFCWLS2WTTHANCNFSM4V6GSUQA>
> .
>

The first option about having configurations like `bokmaal-7`, `bokmaal-9` etc. would definitely work.

A second option would be to add a parameter `ner_tags_set` to `NorneConfig` and then one could load them with
```python
bokmaal_full = load_dataset("norne", "bokmaal", ner_tags_set="norne-full")
```
for example.

What do you think ?

Hi @versae have you had a chance to consider one of the two options for the config ?
I think both are ok but I have a small preference for the first one since it's simpler to implement.

Feel free to ping me if you have questions or if I can help :)

Hi @lhoestq. Agree, option 1 seems easier to implement. Just haven't had bandwidth to get to it yet. Hopefully starting next week I'll be able to update the PR.

Hi @versae ! Did you manage to add the configurations ? Let me know if we can help you on this

Hi @lhoestq, I do actually have to code ready, just need to generate the dummy data for it.

One thing I don't know how to do is to make `_info(self)` return the different NER tags in its `DatasetInfo` object depending on the specific config.

Closing this one and opening a new one with a cleaner commit log.

NorNE is a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names.


Fix column list comparison in transmit format

As noticed in #1718 the cache might not reload the cache files when new columns were added.
This is because of an issue in `transmit_format` where the column list comparison fails because the order was not deterministic. This causes the `transmit_format` to apply an unnecessary `set_format` transform with shuffled column names.

I fixed that by sorting the columns for the comparison and added a test.

To properly test that I added a third column `col_3` to the dummy_dataset used for tests.


Possible cache miss in datasets

Thanks for reporting !
I was able to reproduce thanks to your code and find the origin of the bug.
The cache was not reusing the same file because one object was not deterministic. It comes from a conversion from `set` to `list` in the `datasets.arrrow_dataset.transmit_format` function, where the resulting list would not always be in the same order and therefore the function that computes the hash used by the cache would not always return the same result.
I'm opening a PR to fix this.

Also we plan to do a new release in the coming days so you can expect the fix to be available soon.
Note that you can still specify `cache_file_name=` in the second `map()` call to name the cache file yourself if you want to.

Thanks for the fast reply, waiting for the fix :)

I tried to use `cache_file_names` and wasn't sure how, I tried to give it the following:
```
tokenized_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    num_proc=60,
    load_from_cache_file=True,
    cache_file_names={k: f'.cache/{str(k)}' for k in tokenized_datasets}
)
```

and got an error:
```
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/venv/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/venv/lib/python3.6/site-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1491, in _map_single
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
  File "/usr/lib/python3.6/tempfile.py", line 690, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/lib/python3.6/tempfile.py", line 401, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: '_00000_of_00060.cache/tmpsvszxtop'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 48, in <module>
    cache_file_names={k: f'.cache/{str(k)}' for k in tokenized_datasets}
  File "/venv/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map
    for k, dataset in self.items()
  File "/venv/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1317, in map
    transformed_shards = [r.get() for r in results]
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1317, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/venv/lib/python3.6/site-packages/multiprocess/pool.py", line 644, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '_00000_of_00060.cache/tmpsvszxtop'
```

The documentation says
```
cache_file_names (`Optional[Dict[str, str]]`, defaults to `None`): Provide the name of a cache file to use to store the
    results of the computation instead of the automatically generated cache file name.
    You have to provide one :obj:`cache_file_name` per dataset in the dataset dictionary.
```
What is expected is simply the name of a file, not a path. The file will be located in the cache directory of the `wikitext` dataset. You can try again with something like
```python
cache_file_names = {k: f'tokenized_and_grouped_{str(k)}' for k in tokenized_datasets}
```

Managed to get `cache_file_names` working and caching works well with it
Had to make a small modification for it to work:
```
cache_file_names = {k: f'tokenized_and_grouped_{str(k)}.arrow' for k in tokenized_datasets}
```

Another comment on `cache_file_names`, it doesn't save the produced cached files in the dataset's cache folder, it requires to give a path to an existing directory for it to work.
I can confirm that this is how it works in `datasets==1.1.3`

Oh yes indeed ! Maybe we need to update the docstring to mention that it is a path

I fixed the docstring. Hopefully this is less confusing now: https://github.com/huggingface/datasets/commit/42ccc0012ba8864e6db1392430100f350236183a

I upgraded to the latest version and I encountered some strange behaviour, the script I posted in the OP doesn't trigger recalculation, however, if I add the following change it does trigger partial recalculation, I am not sure if its something wrong on my machine or a bug:
```
from datasets import load_dataset
from transformers import AutoTokenizer

datasets = load_dataset('wikitext', 'wikitext-103-raw-v1')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)

column_names = datasets["train"].column_names
text_column_name = "text" if "text" in column_names else column_names[0]
def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
# CHANGE
print('hello')
# CHANGE

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
...
```
I am using datasets in the `run_mlm.py` script in the transformers examples and I found that if I change the script without touching any of the preprocessing. it still triggers recalculation which is very weird

Edit: accidently clicked the close issue button

This is because the `group_texts` line definition changes (it is defined 3 lines later than in the previous call). Currently if a function is moved elsewhere in a script we consider it to be different.

Not sure this is actually a good idea to keep this behavior though. We had this as a security in the early development of the lib but now the recursive hashing of objects is robust so we can probably remove that.
Moreover we're already ignoring the line definition for lambda functions.

I opened a PR to change this, let me know what you think.

Sounds great, thank you for your quick responses and help! Looking forward for the next release.

I am having a similar issue where only the grouped files are loaded from cache while the tokenized ones aren't. I can confirm both datasets are being stored to file, but only the grouped version is loaded from cache. Not sure what might be going on. But I've tried to remove all kinds of non deterministic behaviour, but still no luck. Thanks for the help!

```python
    # Datasets
    train = sorted(glob(args.data_dir + '*.{}'.format(args.ext)))
    if args.dev_split >= len(train):
        raise ValueError("Not enough dev files")
    dev = []
    state = random.Random(1001)
    for _ in range(args.dev_split):
        dev.append(train.pop(state.randint(0, len(train) - 1)))

    max_seq_length = min(args.max_seq_length, tokenizer.model_max_length)

    def tokenize_function(examples):
        return tokenizer(examples['text'], return_special_tokens_mask=True)

    def group_texts(examples):
        # Concatenate all texts from our dataset and generate chunks of max_seq_length
        concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
        total_length = len(concatenated_examples[list(examples.keys())[0]])
        # Truncate (not implementing padding)
        total_length = (total_length // max_seq_length) * max_seq_length
        # Split by chunks of max_seq_length
        result = {
            k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]
            for k, t in concatenated_examples.items()
        }
        return result

    datasets = load_dataset(
        'text', name='DBNL', data_files={'train': train[:10], 'dev': dev[:5]},
        cache_dir=args.data_cache_dir)
    datasets = datasets.map(tokenize_function,
        batched=True, remove_columns=['text'],
        cache_file_names={k: os.path.join(args.data_cache_dir, f'{k}-tokenized') for k in datasets},
        load_from_cache_file=not args.overwrite_cache)
    datasets = datasets.map(group_texts,
        batched=True,
        cache_file_names={k: os.path.join(args.data_cache_dir, f'{k}-grouped') for k in datasets},
        load_from_cache_file=not args.overwrite_cache)
```

And this is the log

```
04/26/2021 10:26:59 - WARNING - datasets.builder -   Using custom data configuration DBNL-f8d988ad33ccf2c1
04/26/2021 10:26:59 - WARNING - datasets.builder -   Reusing dataset text (/home/manjavacasema/data/.cache/text/DBNL-f8d988ad33ccf2c1/0.0.0/e16f44aa1b321ece1f87b07977cc5d70be93d69b20486d6dacd62e12cf25c9a5)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 21.07ba/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:01<00:00, 24.28ba/s]
04/26/2021 10:27:01 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/manjavacasema/data/.cache/train-grouped
04/26/2021 10:27:01 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/manjavacasema/data/.cache/dev-grouped
```

Hi,

I am using the datasets package and even though I run the same data processing functions, datasets always recomputes the function instead of using cache.
I have attached an example script that for me reproduces the problem.
In the attached example the second map function always recomputes instead of loading from cache.
Is this a bug or am I doing something wrong?
Is there a way for fix this and avoid all the recomputation?

Thanks

Edit:
transformers==3.5.1
datasets==1.2.0

```
from datasets import load_dataset
from transformers import AutoTokenizer

datasets = load_dataset('wikitext', 'wikitext-103-raw-v1')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)

column_names = datasets["train"].column_names
text_column_name = "text" if "text" in column_names else column_names[0]
def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=60,
    remove_columns=[text_column_name],
    load_from_cache_file=True,
)
max_seq_length = tokenizer.model_max_length
def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {
        k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
    # customize this part to your needs.
    total_length = (total_length // max_seq_length) * max_seq_length
    # Split by chunks of max_len.
    result = {
        k: [t[i: i + max_seq_length]
            for i in range(0, total_length, max_seq_length)]
        for k, t in concatenated_examples.items()
    }
    return result

tokenized_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    num_proc=60,
    load_from_cache_file=True,
)
print(tokenized_datasets)

print('finished')
```


SciFact dataset - minor changes

Hi Dave,
You are more than welcome to open a PR to make these changes! 🤗
You will find the relevant information about opening a PR in the [contributing guide](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md) and in the [dataset addition guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Pinging also @lhoestq for the Google cloud matter.

> I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?

Sure ! Also feel free to ping us for reviews or if we can help :)

> It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?

What makes you think that ?
Afaik there's no scifact on our google storage

> > I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?
>
> Sure ! Also feel free to ping us for reviews or if we can help :)
>
OK! We're organizing a [shared task](https://sdproc.org/2021/sharedtasks.html#sciver) based on the dataset, and I made some updates and changed the download URL - so the current code points to a dead URL. I'll update appropriately once the task is finalized and make a PR.

> > It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?
>
> What makes you think that ?
> Afaik there's no scifact on our google storage

You're right, I had the data cached on my machine somewhere.

I opened a PR about this: https://github.com/huggingface/datasets/pull/1780. Closing this issue, will continue there.

Hi,

SciFact dataset creator here. First of all, thanks for adding the dataset to Huggingface, much appreciated!

I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?

It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?

Thanks,

Dave


Add Hatexplain Dataset

Adding Hatexplain - the first benchmark hate speech dataset covering multiple aspects of the issue


add Korean intonation-aided intention identification dataset


Adding adversarialQA dataset

Oh that's a really cool one, we'll review/merge it soon!

In the meantime, do you have any specific positive/negative feedback on the process of adding a datasets Max?
Did you follow the instruction in the [detailed step-by-step](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md)?

Thanks Thom, been a while, hope all is well!

Yes, I followed the step by step instructions and found them pretty straightforward. The only things I wasn't sure of were what should go into the YAML tags field for the dataset card, and whether there was a list of options somewhere (maybe akin to the metrics?) of the possible supported tasks. I found the rest very intuitive and the automated metadata and dummy data generation very handy. Thanks!

Good point! pinging @yjernite here so he can improve this part!

@maxbartolo cool addition!

For the YAML tag, you should use the tagging app we provide to choose from a drop-down menu:
https://github.com/huggingface/datasets-tagging

The process is described toward the end of the [step-by-step guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card), do you have any suggestions for making it easier to find?

Otherwise, the dataset card is really cool, thanks for making it so complete!

@yjernite

Thanks, YAML tags added. I think my main issue was with the flow of the [step-by-step guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md). For example, the [card creator](https://huggingface.co/datasets/card-creator/) is introduced in Step 4, right after creating an empty directory for your dataset. The first field it requires are the YAML tags, which (at least for me) was the last step of the process.

I'd suggest having the guide structured in the same order as the creation process. For me it was something like:
- Step 1: Preparing your env
- Step 2: Write the loading/processing code
- Step 3: Automatically generate dummy data and `dataset_infos.json`
- Step 4: Tag the dataset
- Step 5: Write the dataset card using the [card creator](https://huggingface.co/datasets/card-creator/)
- Step 6: Open a Pull Request on the main HuggingFace repo and share your work!!

Thanks again!

Adding the adversarialQA dataset (https://adversarialqa.github.io/) from Beat the AI (https://arxiv.org/abs/2002.00293)


Installation using conda

Yes indeed the idea is to have the next release on conda cc @LysandreJik

Great! Did you guys have a timeframe in mind for the next release?

Thank you for all the great work in developing this library.

I think we can have `datasets` on conda by next week. Will see what I can do!

`datasets` has been added to the huggingface channel thanks to @LysandreJik :)
It depends on conda-forge though

```
conda install -c huggingface -c conda-forge datasets
```

Will a conda package for installing datasets be added to the huggingface conda channel? I have installed transformers using conda and would like to use the datasets library to use some of the scripts in the transformers/examples folder but am unable to do so at the moment as datasets can only be installed using pip and using pip in a conda environment is generally a bad idea in my experience.


Silicone

Hi @eusip,

> When should we expect to see our dataset appear in the search dropdown at huggingface.co?

when this PR is merged.

I've implemented all the changes requested by @lhoestq but I made the mistake of trying to change the remote branch name.

Hopefully the changes are seen on your end as both branches `silicone` and `main` should be up-to-date.

It looks like the PR includes changes about many other files than the ones for Silicone (+30,000 line changes)

Maybe you can try to create another branch and another PR ?

> It looks like the PR includes changes about many other files than the ones for Silicone (+30,000 line changes)
>
> Maybe you can try to create another branch and another PR ?

Sure. I will make a new pull request.

My collaborators and I within the Affective Computing team at Telecom Paris would like to push our spoken dialogue dataset for publication.


Fix windows path scheme in cached path

As noticed in #807 there's currently an issue with `cached_path` not raising `FileNotFoundError` on windows for absolute paths. This is due to the way we check for a path to be local or not. The check on the scheme using urlparse was incomplete.

I fixed this and added tests


IsADirectoryError when trying to download C4

I haven't tested C4 on my side so there so there may be a few bugs in the code/adjustments to make.
Here it looks like in c4.py, line 190 one of the `files_to_download` is `'/'` which is invalid.
Valid files are paths to local files or URLs to remote files.

**TLDR**:

I fail to download C4 and see a stacktrace originating in `IsADirectoryError` as an explanation for failure.

How can the problem be fixed?

**VERBOSE**:

I use Python version 3.7 and have the following dependencies listed in my project:

```
datasets==1.2.0
apache-beam==2.26.0
```

When running the following code, where `/data/huggingface/unpacked/` contains a single unzipped `wet.paths` file manually downloaded as per the instructions for C4:

```
from datasets import load_dataset

load_dataset("c4", "en", data_dir="/data/huggingface/unpacked", beam_runner='DirectRunner')
```

I get the following stacktrace:

```
/Users/fredriko/venv/misc/bin/python /Users/fredriko/source/misc/main.py
Downloading and preparing dataset c4/en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /Users/fredriko/.cache/huggingface/datasets/c4/en/2.3.0/8304cf264cc42bdebcb13fca4b9cb36368a96f557d36f9dc969bebbe2568b283...
Traceback (most recent call last):
  File "/Users/fredriko/source/misc/main.py", line 3, in <module>
    load_dataset("c4", "en", data_dir="/data/huggingface/unpacked", beam_runner='DirectRunner')
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 1066, in _download_and_prepare
    pipeline=pipeline,
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 582, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/fredriko/.cache/huggingface/modules/datasets_modules/datasets/c4/8304cf264cc42bdebcb13fca4b9cb36368a96f557d36f9dc969bebbe2568b283/c4.py", line 190, in _split_generators
    file_paths = dl_manager.download_and_extract(files_to_download)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 258, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 189, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 117, in _record_sizes_checksums
    self._recorded_sizes_checksums[str(url)] = get_size_checksum_dict(path)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 80, in get_size_checksum_dict
    with open(path, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/'

Process finished with exit code 1
```


Databases

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


<html dir="ltr" lang="en" class="focus-outline-visible"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">


Added generated READMEs for datasets that were missing one.

Looks like we need to trim the ones with too many configs, will look into it tomorrow!

This is it: we worked on a generator with Yacine @yjernite , and we generated dataset cards for all missing ones (161), with all the information we could gather from datasets repository, and using dummy_data to generate examples when possible.

Code is available here for the moment: https://github.com/madlag/datasets_readme_generator .
We will move it to a Hugging Face repository and to https://huggingface.co/datasets/card-creator/ later.


Error when downloading a large dataset on slow connection.

Hi ! Is this an issue you have with `openwebtext` specifically or also with other datasets ?

It looks like the downloaded file is corrupted and can't be extracted using `tarfile`.
Could you try loading it again with
```python
import datasets
datasets.load_dataset("openwebtext", download_mode="force_redownload")
```

I receive the following error after about an hour trying to download the `openwebtext` dataset.

The code used is:
```python
import datasets
datasets.load_dataset("openwebtext")
```

> Traceback (most recent call last):                                                                                [4/28]
>   File "<stdin>", line 1, in <module>
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/load.py", line 610, in load_dataset
>     ignore_verifications=ignore_verifications,
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/builder.py", line 515, in download_and_prepare
>     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/builder.py", line 570, in _download_and_prepare
>     split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
>   File "/home/lucadiliello/.cache/huggingface/modules/datasets_modules/datasets/openwebtext/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02/openwebtext.py", line 62, in _split_generators
>     dl_dir = dl_manager.download_and_extract(_URL)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 254, in download_and_extract
>     return self.extract(self.download(url_or_urls))
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 235, in extract
>     num_proc=num_proc,
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 225, in map_nested
>     return function(data_struct)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
>     tar_file.extractall(output_path_extracted)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 2000, in extractall
>     numeric_owner=numeric_owner)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 2042, in extract
>     numeric_owner=numeric_owner)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 2112, in _extract_member
>     self.makefile(tarinfo, targetpath)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 2161, in makefile
>     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 253, in copyfileobj
>     buf = src.read(remainder)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/lzma.py", line 200, in read
>     return self._buffer.read(size)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/_compression.py", line 68, in readinto
>     data = self.read(len(byte_view))
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/_compression.py", line 99, in read
>     raise EOFError("Compressed file ended before the "
> EOFError: Compressed file ended before the end-of-stream marker was reached


Add information about caching and verifications in "Load a Dataset" docs

documentation

Improvements or additions to documentation

Related to #215.

Missing improvements from @lhoestq's #1703.


Update XSUM Factuality DatasetCard

Update XSUM Factuality DatasetCard


Improvements regarding caching and fingerprinting

I few comments here for discussion:
- I'm not convinced yet the end user should really have to understand the difference between "caching" and 'fingerprinting", what do you think? I think fingerprinting should probably stay as an internal thing. Is there a case where we want cahing without fingerprinting or vice-versa?
- while I think the random fingerprint mechanism is smart, I have one question: when we disable caching or fingerprinting we also probably don't want the disk usage to grow so we should then try to keep only one cache file. Is it the case currently?
- the warning should be emitted only once per session if possible (we have a mechanism to do that in transformers, you should ask Lysandre/Sylvain)

About your points:
- Yes I agree, I just wanted to bring the discussion on this point. Until now fingerprinting hasn't been blocking for user experience. I'll probably remove the enable/disable fingerprinting function to keep things simple from the user's perspective.
- Right now every time a not in-place transform (i.e. map, filter) is applied, a new cache file is created. It is the case even if caching is disabled since disabling it only means that the cache file won't be reloaded. Therefore you're right that it might end up filling the disk with files that won't be reused. I like the idea of keeping only one cache file. Currently all the cache files are kept on disk until the user clears the cache. To be able to keep only one, we need to know if a dataset that has been transformed is still loaded or not. For example
```python
# case 1 - keep both cache files (dataset1 and dataset2)
dataset2 = dataset1.map(...)
# case 2 - keep only the new cache file
dataset1 = dataset1.map(...)
```
In python it doesn't seem trivial to detect such changes. One thing that we can actually do on the other hand is store the cache files in a temporary directory that is cleared when the session closes. I think that's a good a simple solution for this problem.
- Yes good idea ! I don't like spam either :)

> * To be able to keep only one, we need to know if a dataset that has been transformed is still loaded or not. For example
>
> ```python
> # case 1 - keep both cache files (dataset1 and dataset2)
> dataset2 = dataset1.map(...)
> # case 2 - keep only the new cache file
> dataset1 = dataset1.map(...)
> ```

I see what you mean. It's a tricky question. One option would be that if caching is deactivated we have a single memory mapped file and have copy act as a copy by reference instead of a copy by value. We will then probably want a `copy()` or `deepcopy()` functionality. Maybe we should think a little bit about it though.

- I like the idea of using a temporary directory per session!
- If the default behavior when caching is disabled is to re-use the same file, I'm a little worried about people making mistakes and having to re-download and process from scratch.
- So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?

> * If the default behavior when caching is disabled is to re-use the same file, I'm a little worried about people making mistakes and having to re-download and process from scratch.

We should distinguish between the caching from load_dataset (base dataset cache files) and the caching after dataset transforms such as map or filter (transformed dataset cache files). When disabling caching only the second type (for map and filter) doesn't reload from cache files.
Therefore nothing is re-downloaded. To re-download the dataset entirely the argument `download_mode="force_redownload"` must be used in `load_dataset`.
Do we have to think more about the naming to make things less confusing in your opinion ?

> * So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?

There's no such `in_place` parameter in map, what do you mean exactly ?

I updated the PR:
- I removed the enable/disable fingerprinting function
- if caching is disabled arrow files are written in a temporary directory that is deleted when session closes
- the warning that is showed when hashing a transform fails is only showed once
- I added the `set_caching_enabled` function to the docs and explained the caching mechanism and its relation with fingerprinting

I would love to have some feedback :)

> > * So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?
>
> There's no such `in_place` parameter in map, what do you mean exactly ?

Sorry, that wasn't clear at all. I was responding to your previous comment about case 1 / case 2. I don't think the behavior should depend on the command, but we could have:

```
# case 1 - keep both cache files (dataset1 and dataset2)
dataset2 = dataset1.map(...)
# case 2 - keep only the new cache file
dataset1 = dataset1.map(..., in_place=True)
```

Case 1 returns a new reference using the new cache file, case 2 returns the same reference

> Sorry, that wasn't clear at all. I was responding to your previous comment about case 1 / case 2. I don't think the behavior should depend on the command, but we could have:
>
> ```
> # case 1 - keep both cache files (dataset1 and dataset2)
> dataset2 = dataset1.map(...)
> # case 2 - keep only the new cache file
> dataset1 = dataset1.map(..., in_place=True)
> ```
>
> Case 1 returns a new reference using the new cache file, case 2 returns the same reference

Ok I see !
`in_place` is a parameter that is used in general to designate a transform so I would name that differently (maybe `overwrite` or something like that).
Not sure if it's possible to update an already existing arrow file that is memory-mapped, let me check real quick.
Also it's possible to call `dataset2.cleanup_cache_files()` to delete the other cache files if we create a new one after the transform. Or even to get the cache file with `dataset1.cache_files` and let the user remove them by hand.

EDIT: updating an arrow file in place is not part of the current API of pyarrow, so we would have to make new files.

This PR adds these features:
- Enable/disable caching
    If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
    It is equivalent to setting `load_from_cache` to `False` in dataset transforms.
```python
from datasets import set_caching_enabled

set_caching_enabled(False)
```
- Allow unpicklable functions in `map`
    If an unpicklable function is used, then it's not possible to hash it to update the dataset fingerprint that is used to name cache files. To workaround that, a random fingerprint is generated instead and a warning is raised.
```python
logger.warning(
    f"Transform {transform} couldn't be hashed properly, a random hash was used instead. "
    "Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. "
    "If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything."
)
```

and also (open to discussion, EDIT: actually NOT included):
- Enable/disable fingerprinting
    Fingerprinting allows to have one deterministic fingerprint per dataset state.
    A dataset fingerprint is updated after each transform.
    Re-running the same transforms on a dataset in a different session results in the same fingerprint.
    Disabling the fingerprinting mechanism makes all the fingerprints random.
    Since the caching mechanism uses fingerprints to name the cache files, then cache file names will be different.
    Therefore disabling fingerprinting will prevent the caching mechanism from reloading datasets files that have already been computed.
    Disabling fingerprinting may speed up the lib for users that don't care about this feature and don't want to use caching.
```python
from datasets import set_fingerprinting_enabled

set_fingerprinting_enabled(False)
```

Other details:
- I renamed the `fingerprint` decorator to `fingerprint_transform` since the name was clearly not explicit. This decorator is used on dataset transform functions to allow them to update fingerprints.
- I added some `ignore_kwargs` when decorating transforms with `fingerprint_transform`, to make the fingerprint update not sensible to kwargs like `load_from_cache` or `cache_file_name`.

Todo: tests for set_fingerprinting_enabled + documentation for all the above features


Fix importlib metdata import in py38

In Python 3.8 there's no need to install `importlib_metadata` since it already exists as `importlib.metadata` in the standard lib.


Some datasets miss dataset_infos.json or dummy_data.zip

Thanks for reporting.
We should indeed add all the missing dummy_data.zip and also the dataset_infos.json at least for lm1b, reclor and wikihow.

For c4 I haven't tested the script and I think we'll require some optimizations regarding beam datasets before processing it.

While working on dataset REAME generation script at https://github.com/madlag/datasets_readme_generator , I noticed that some datasets miss a dataset_infos.json :

```
c4
lm1b
reclor
wikihow
```

And some does not have a dummy_data.zip :

```
kor_nli
math_dataset
mlqa
ms_marco
newsgroup
qa4mre
qangaroo
reddit_tifu
super_glue
trivia_qa
web_of_science
wmt14
wmt15
wmt16
wmt17
wmt18
wmt19
xtreme
```

But it seems that some of those last do have a "dummy" directory .


Update Curiosity dialogs DatasetCard

Update Curiosity dialogs DatasetCard

There are some entries in the data fields section yet to be filled. There is little information regarding those fields.


Update DBRD dataset card and download URL

not sure why the CI was not triggered though

I've added the Dutch Bood Review Dataset (DBRD) during the recent sprint. This pull request makes two minor changes:

1.  I'm changing the download URL from Google Drive to the dataset's GitHub release package. This is now possible because of PR #1316.
2. I've updated the dataset card.

Cheers! 😄


Update Coached Conv Pref DatasetCard

Really cool!

Can you add some task tags for `dialogue-modeling` (under `sequence-modeling`) and `parsing` (under `structured-prediction`)?

Update Coached Conversation Preferance DatasetCard


Update DialogRE DatasetCard

Same as #1698, can you add a task tag for dialogue-modeling (under sequence-modeling) :) ?

Update the information in the dataset card for the Dialog RE dataset.