kardwalker/my_dataset huggingface dataset:
GitHub issue #1785: Not enough disk space (Needed: Unknown size) when caching on a cluster
State: closed | Created: 2021-01-27T21:30:59Z

I'm running some experiments where I'm caching datasets on a cluster and accessing it through multiple compute nodes. However, I get an error when loading the cached dataset from the shared disk.

The exact error thrown:

```bash
>>> load_dataset(dataset, cache_dir="/path/to/cluster/shared/path")
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
```


[`utils.has_sufficient_disk_space`](https://github.com/huggingface/datasets/blob/8a03ab7d123a76ee744304f21ce868c75f411214/src/datasets/utils/py_utils.py#L332) fails on each job because of how the cluster system is designed (`disk_usage(".").free` can't compute on the cluster's shared disk).


This is exactly where the error gets thrown:
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L502

```python
if not utils.has_sufficient_disk_space(self.info.size_in_bytes or 0, directory=self._cache_dir_root):
    raise IOError(
          "Not enough disk space. Needed: {} (download: {}, generated: {}, post-processed: {})".format(
          utils.size_str(self.info.size_in_bytes or 0),
          utils.size_str(self.info.download_size or 0),
          utils.size_str(self.info.dataset_size or 0),
          utils.size_str(self.info.post_processing_size or 0),
       )
    )

```

What would be a good way to circumvent this? my current fix is to manually comment out that part, but that is not ideal. 
Would it be possible to pass a flag to skip this check on disk space?
GitHub issue #1784: JSONDecodeError on JSON with multiple lines
State: closed | Created: 2021-01-27T00:19:22Z

Hello :),

I have been trying to load data using a JSON file. Based on the [docs](https://huggingface.co/docs/datasets/loading_datasets.html#json-files), the following format is supported:

```json
{"key1":11, "key2":12, "key3":13}
{"key1":21, "key2":22, "key3":23}
```
 But, when I try loading a dataset with the same format, I get a JSONDecodeError : `JSONDecodeError: Extra data: line 2 column 1 (char 7142)`. Now, this is expected when using `json` to load a JSON file. But I was wondering if there are any special arguments to pass when using `load_dataset` as the docs suggest that this format is supported.

When I convert the JSON file to a list of dictionaries format, I get AttributeError: `AttributeError: 'list' object has no attribute 'keys'`. So, I can't convert them to list of dictionaries either.

Please let me know :)

Thanks,
Gunjan
GitHub issue #1783: Dataset Examples Explorer
State: closed | Created: 2021-01-26T20:39:02Z

In the Older version of the Dataset, there are a useful Dataset Explorer that allow user to visualize the examples (training, test and validation) of a particular dataset, it is no longer there in current version.

Hope HuggingFace can re-enable the feature that at least allow viewing of  the first 20  examples of a particular dataset, or alternatively can extract 20 examples for each datasets and make those part of the Dataset Card Documentation.
GitHub issue #1781: AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 
State: closed | Created: 2021-01-26T04:18:35Z

I'm using Colab. And suddenly this morning, there is this error. Have a look below!

![screenshot-colab research google com-2021 01 26-08-15-36](https://user-images.githubusercontent.com/45964869/105799890-fdaf3b80-5fae-11eb-8f06-11b65cdccc30.png)
GitHub issue #1779: Ignore definition line number of functions for caching
State: closed | Created: 2021-01-25T16:42:29Z

As noticed in #1718 , when a function used for processing with `map` is moved inside its python file, then the change of line number causes the caching mechanism to consider it as a different function. Therefore in this case, it recomputes everything.

This is because we were not ignoring the line number definition for such functions (even though we're doing it for lambda functions).

For example this code currently prints False:
```python
from datasets.fingerprint import Hasher

# define once
def foo(x):
    return x

h = Hasher.hash(foo)

# define a second time elsewhere
def foo(x):
    return x

print(h == Hasher.hash(foo))
```

I changed this by ignoring the line number for all functions.
GitHub issue #1777: GPT2 MNLI training using run_glue.py
State: closed | Created: 2021-01-25T10:53:52Z

Edit: I'm closing this because I actually meant to post this in `transformers `not `datasets`

Running this on Google Colab,

```
!python run_glue.py \
  --model_name_or_path gpt2 \
  --task_name mnli \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 10 \
  --gradient_accumulation_steps 32\
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir models/gpt2/mnli/
```

I get the following error,

```
 "Asking to pad but the tokenizer does not have a padding token. "
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
```

Do I need to modify the trainer to work with GPT2 ?
GitHub issue #1776: [Question & Bug Report] Can we preprocess a dataset on the fly?
State: closed | Created: 2021-01-24T09:28:24Z

I know we can use `Datasets.map` to preprocess a dataset, but I'm using it with very large corpus which generates huge cache file (several TB cache from a 400 GB text file). I have no disk large enough to save it.  Can we preprocess a dataset on the fly without generating cache?

BTW, I tried raising `writer_batch_size`. Seems that argument doesn't have any effect when it's larger than `batch_size`, because you are saving all the batch instantly after it's processed. Please check the following code:

https://github.com/huggingface/datasets/blob/0281f9d881f3a55c89aeaa642f1ba23444b64083/src/datasets/arrow_dataset.py#L1532
GitHub issue #1772: Adding SICK dataset
State: closed | Created: 2021-01-24T02:15:31Z

Hi
It would be great to include SICK dataset.

## Adding a Dataset
- **Name:** SICK
- **Description:** a well known entailment dataset 
- **Paper:** http://marcobaroni.org/composes/sick.html
- **Data:** http://marcobaroni.org/composes/sick.html
- **Motivation:** this is an important NLI benchmark

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).



thanks
GitHub issue #1771: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/csv/csv.py
State: closed | Created: 2021-01-24T01:53:52Z

Hi,
When I load_dataset from local csv files, below error happened, looks raw.githubusercontent.com was blocked by the chinese government. But why it need to download csv.py? should it include when pip install the dataset?

```
Traceback (most recent call last):
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
    max_retries=download_config.max_retries,
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 617, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/csv/csv.py

```
GitHub issue #1770: how can I combine 2 dataset with different/same features?
State: closed | Created: 2021-01-24T01:26:06Z

to combine 2 dataset by one-one map like ds = zip(ds1, ds2):
ds1: {'text'}, ds2: {'text'}, combine ds:{'src', 'tgt'} 
or different feature:
ds1: {'src'}, ds2: {'tgt'}, combine ds:{'src', 'tgt'}
GitHub issue #1768: Mention kwargs in the Dataset Formatting docs
State: closed | Created: 2021-01-22T16:43:20Z

Hi,

This was discussed in Issue #1762 where the docs didn't mention that keyword arguments to `datasets.Dataset.set_format()` are allowed. 
To prevent people from having to check the code/method docs, I just added a couple of lines in the docs.

Please let me know your thoughts on this.

Thanks,
Gunjan

@lhoestq
GitHub issue #1767: Add Librispeech ASR
State: closed | Created: 2021-01-22T14:54:37Z

This PR adds the librispeech asr dataset: https://www.tensorflow.org/datasets/catalog/librispeech

There are 2 configs: "clean" and "other" whereas there are two "train" datasets for "clean", hence the name "train.100" and "train.360".

As suggested by @lhoestq, due to the enormous size of the dataset in `.arrow` format, the speech files are not directly prepared to a float32-array, but instead just the path to the array file is stored.
GitHub issue #1766: Issues when run two programs compute the same metrics
State: closed | Created: 2021-01-22T14:22:55Z

I got the following error when running two different programs that both compute sacreblue metrics. It seems that both read/and/write to the same location (.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow) where it caches the batches:

```
File "train_matching_min.py", line 160, in <module>ch_9_label
    avg_loss = valid(epoch, args.batch, args.validation, args.with_label)
  File "train_matching_min.py", line 93, in valid
    bleu += eval.compute()
  File "/u/tlhoang/projects/seal/match/models/eval.py", line 23, in compute
    return self.metric.compute()['score']
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/metric.py", line 387, in compute
    self._finalize()
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/metric.py", line 355, in _finalize
    self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 231, in read_files
    pa_table = self._read_files(files)
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 170, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 299, in _get_dataset_from_filename
    pa_table = f.read_all()
  File "pyarrow/ipc.pxi", line 481, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected to read 1819307375 metadata bytes, but only read 454396
```
GitHub issue #1765: Error iterating over Dataset with DataLoader
State: closed | Created: 2021-01-21T22:56:45Z

I have a Dataset that I've mapped a tokenizer over:

```
encoded_dataset.set_format(type='torch',columns=['attention_mask','input_ids','token_type_ids'])
encoded_dataset[:1]
```
```
{'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]),
 'input_ids': tensor([[  101,   178,  1198,  1400,  1714, 22233, 21365,  4515,  8618,  1113,
            102]]),
 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])}
```

When I try to iterate as in the docs, I get errors:

```
dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_sampler=32)
next(iter(dataloader))
```

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-45-05180ba8aa35> in <module>()
      1 dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_sampler=32)
----> 2 next(iter(dataloader))

3 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __init__(self, loader)
    411         self._timeout = loader.timeout
    412         self._collate_fn = loader.collate_fn
--> 413         self._sampler_iter = iter(self._index_sampler)
    414         self._base_seed = torch.empty((), dtype=torch.int64).random_(generator=loader.generator).item()
    415         self._persistent_workers = loader.persistent_workers

TypeError: 'int' object is not iterable


```
GitHub issue #1764: Connection Issues
State: closed | Created: 2021-01-21T20:56:09Z

Today, I am getting connection issues while loading a dataset and the metric.
```
Traceback (most recent call last):
  File "src/train.py", line 180, in <module>
    train_dataset, dev_dataset, test_dataset = create_race_dataset()
  File "src/train.py", line 130, in create_race_dataset
    train_dataset = load_dataset("race", "all", split="train")
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/load.py", line 591, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
    max_retries=download_config.max_retries,
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 617, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/race/race.py
```

Or

```
Traceback (most recent call last):
  File "src/train.py", line 105, in <module>
    rouge = datasets.load_metric("rouge")
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/load.py", line 500, in load_metric
    dataset=False,
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
    max_retries=download_config.max_retries,
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 617, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/metrics/rouge/rouge.py
```
GitHub issue #1763: PAWS-X: Fix csv Dictreader splitting data on quotes
State: closed | Created: 2021-01-21T18:21:01Z


```python
from datasets import load_dataset
# load english paws-x dataset 
datasets = load_dataset('paws-x', 'en')
print(len(datasets['train']))                     # outputs 49202 but official dataset has 49401 pairs
print(datasets['train'].unique('label'))     # outputs [1, 0, -1] but labels are binary [0,1]
```

changed `data = csv.DictReader(f, delimiter="\t")` to `data = csv.DictReader(f, delimiter="\t", quoting=csv.QUOTE_NONE)` in the dataloader to make csv module not split by quotes.

The results are as expected for all languages after the change.
GitHub issue #1762: Unable to format dataset to CUDA Tensors
State: closed | Created: 2021-01-21T15:31:23Z

Hi,

I came across this [link](https://huggingface.co/docs/datasets/torch_tensorflow.html) where the docs show show to convert a dataset to a particular format. I see that there is an option to convert it to tensors, but I don't see any option to convert it to CUDA tensors.

I tried this, but Dataset doesn't support assignment:
```
  columns=['input_ids', 'token_type_ids', 'attention_mask', 'start_positions','end_positions']

        samples.set_format(type='torch', columns = columns)
        for column in columns:
            samples[column].to(torch.device(self.config.device))
```
There should be an option to do so, or if there is already a way to do this, please let me know.

Thanks,
Gunjan
GitHub issue #1761: Add SILICONE benchmark
State: closed | Created: 2021-01-21T14:29:12Z

My collaborators and I within the Affective Computing team at Telecom Paris would like to re-submit our spoken dialogue dataset for publication.

This is a new pull request relative to the [previously closed request](https://github.com/huggingface/datasets/pull/1712) which was reviewed by @lhoestq.
GitHub issue #1760: More tags
State: closed | Created: 2021-01-21T13:50:10Z

Since the hub v2 is going to be released soon I figured it would be great to add the missing tags at least for some of the datasets of reference listed [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#write-the-loadingprocessing-code)
GitHub issue #1759: wikipedia dataset incomplete
State: closed | Created: 2021-01-21T11:47:15Z

Hey guys,

I am using the https://github.com/huggingface/datasets/tree/master/datasets/wikipedia dataset.
Unfortunately, I found out that there is an incompleteness for the German dataset.
 For reasons unknown to me, the number of inhabitants has been removed from many pages:
Thorey-sur-Ouche has 128 inhabitants according to the webpage (https://de.wikipedia.org/wiki/Thorey-sur-Ouche).
The pickle file however shows: französische Gemeinde mit  Einwohnern (Stand).
 Is it possible to fix this?

Best regards 
Chris
GitHub issue #1757: FewRel
State: closed | Created: 2021-01-20T23:56:03Z

## Adding a Dataset
- **Name:** FewRel
- **Description:**  Large-Scale Supervised Few-Shot Relation Classification Dataset
- **Paper:** @inproceedings{han2018fewrel,
               title={FewRel:A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation},
               author={Han, Xu and Zhu, Hao and Yu, Pengfei and Wang, Ziyun and Yao, Yuan and Liu, Zhiyuan and Sun, Maosong},
               booktitle={EMNLP},
               year={2018}}
- **Data:** https://github.com/ProKil/FewRel
- **Motivation:** relationship extraction dataset that's been used by some state of the art systems that should be incorporated.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1756: Ccaligned multilingual translation dataset
State: closed | Created: 2021-01-20T22:18:44Z

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- CCAligned consists of parallel or comparable web-document pairs in 137 languages aligned with English. These web-document pairs were constructed by performing language identification on raw web-documents, and ensuring corresponding language codes were corresponding in the URLs of web documents. This pattern matching approach yielded more than 100 million aligned documents paired with English. Recognizing that each English document was often aligned to mulitple documents in different target language, we can join on English documents to obtain aligned documents that directly pair two non-English documents (e.g., Arabic-French).
- **Paper:** *link to the dataset paper if available*
- https://www.aclweb.org/anthology/2020.emnlp-main.480.pdf
- **Data:** *link to the Github repository or current dataset location*
- http://www.statmt.org/cc-aligned/
- **Motivation:** *what are some good reasons to have this dataset*
- The authors says it's an high quality dataset.
- it's pretty large and includes many language pairs. It could be interesting training mt5 on this task.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1754: Use a config id in the cache directory names for custom configs
State: closed | Created: 2021-01-20T11:11:00Z

As noticed by @JetRunner there was some issues when trying to generate a dataset using a custom config that is based on an existing config.

For example in the following code the `mnli_custom` would reuse the cache used to create `mnli` instead of generating a new dataset with the new label classes:
```python
from datasets import load_dataset

mnli = load_dataset("glue", "mnli")
mnli_custom = load_dataset("glue", "mnli", label_classes=["contradiction", "entailment", "neutral"])
```

I fixed that by extending the cache directory definition of a dataset that is being generated.
Instead of using the config name in the cache directory name, I switched to using a `config_id`.

By default it is equal to the config name.
However the name of a config is not sufficent to have a unique identifier for the dataset being generated since it doesn't take into account:
- the config kwargs that can be used to overwrite attributes
- the custom features used to write the dataset
- the data_files for json/text/csv/pandas datasets

Therefore the config id is just the config name with an optional suffix based on these.

In particular taking into account the config kwargs fixes the issue with the `label_classes` above.

I completed the current test cases by adding the case that was missing: overwriting an already existing config.
GitHub issue #1753: fix comet citations
State: closed | Created: 2021-01-20T10:52:38Z

I realized COMET citations were not showing in the hugging face metrics page:

<img width="814" alt="Screenshot 2021-01-20 at 09 48 44" src="https://user-images.githubusercontent.com/17256847/105164848-8b9da900-5b0d-11eb-9e20-a38f559d2037.png">

This pull request is intended to fix that.

Thanks!
GitHub issue #1752: COMET metric citation
State: closed | Created: 2021-01-20T09:54:43Z

In my last pull request to add COMET metric, the citations where not following the usual "format". Because of that they where not correctly displayed on the website: 

<img width="814" alt="Screenshot 2021-01-20 at 09 48 44" src="https://user-images.githubusercontent.com/17256847/105158000-686efb80-5b05-11eb-8bb0-9c85fdac2938.png">

This pull request is only intended to fix that.
GitHub issue #1751: Updated README for the Social Bias Frames dataset
State: closed | Created: 2021-01-19T17:53:00Z

See the updated card at https://github.com/mcmillanmajora/datasets/tree/add-SBIC-card/datasets/social_bias_frames. I incorporated information from the [SBIC data statement](https://homes.cs.washington.edu/~msap/social-bias-frames/DATASTATEMENT.html), paper, and the corpus README file included with the dataset download.
GitHub issue #1749: Added metadata and correct splits for swda.
State: closed | Created: 2021-01-18T18:36:32Z

Switchboard Dialog Act Corpus

I made some changes following  @bhavitvyamalik recommendation in #1678:

* Contains all metadata.
* Used official implementation from the [/swda](https://github.com/cgpotts/swda) repo.
* Add official train and test splits used in [Stolcke et al. (2000)](https://web.stanford.edu/~jurafsky/ws97) and validation split used in [Probabilistic-RNN-DA-Classifier](https://github.com/NathanDuran/Probabilistic-RNN-DA-Classifier).
GitHub issue #1747: datasets slicing with seed 
State: closed | Created: 2021-01-18T14:08:55Z

Hi
I need to slice a dataset with random seed, I looked into documentation here https://huggingface.co/docs/datasets/splits.html 
I could not find a seed option, could you assist me please how I can get a slice for different seeds?
thank you.
@lhoestq
GitHub issue #1743: Issue while Creating Custom Metric
State: closed | Created: 2021-01-17T07:01:14Z

Hi Team,

I am trying to create a custom metric for my training as follows, where f1 is my own metric:

```python
 def _info(self):
        # TODO: Specifies the datasets.MetricInfo object
        return datasets.MetricInfo(
            # This is the description that will appear on the metrics page.
            description=_DESCRIPTION,
            citation=_CITATION,
            inputs_description=_KWARGS_DESCRIPTION,
            # This defines the format of each prediction and reference
            features = datasets.Features({'predictions':datasets.Sequence(datasets.Value("int32")), "references": datasets.Sequence(datasets.Value("int32")),"offset_mapping":datasets.Sequence(datasets.Value("int32")),'text':datasets.Sequence(datasets.Value('string')),"ground":datasets.Sequence(datasets.Value("int32")),}),
            # Homepage of the metric for documentation
            homepage="http://metric.homepage",
            # Additional links to the codebase or references
            codebase_urls=["http://github.com/path/to/codebase/of/new_metric"],
            reference_urls=["http://path.to.reference.url/new_metric"]
        )

    def _compute(self,predictions,references,text,offset_mapping,spans):

        pred_spans = []

        for i,preds in enumerate(predictions):
            current_preds = []
            for j,token_preds in enumerate(preds):
                if (preds>0.5):
                    current_preds+=list(range(offset_mapping[i][j][0],offset_mapping[i][j][1]))
            pred_spans.append(current_spans)
        
        return {
            "Token Wise F1": f1_score(references,predictions,labels=[0,1]),
            "Offset Wise F1": np.mean([f1(preds,gold) for preds,fold in zip(pred_spans,ground)])
        }

```

I believe this is not correct. But that's not the issue I am facing right now. I get this error :
```python
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-144-ed7349b50821> in <module>()
----> 1 new_metric.compute(predictions=inputs["labels"],references=inputs["labels"], text=inputs["text"], offset_mapping=inputs["offset_mapping"],ground=inputs["ground"] )

2 frames
/usr/local/lib/python3.6/dist-packages/datasets/features.py in encode_batch(self, batch)
    802         encoded_batch = {}
    803         if set(batch) != set(self):
--> 804             print(batch)
    805             print(self)
    806             raise ValueError("Column mismatch between batch {} and features {}".format(set(batch), set(self)))

ValueError: Column mismatch between batch {'references', 'predictions'} and features {'ground', 'predictions', 'offset_mapping', 'text', 'references'}
```
On checking the features.py file, I see the call is made from add_batch() in metrics.py which only takes in predictions and references.

How do I make my custom metric work? Will it work with a trainer even if I am able to make this metric work?

Thanks,
Gunjan
GitHub issue #1742: Add GLUE Compat (compatible with transformers<3.5.0)
State: closed | Created: 2021-01-17T05:54:25Z

Link to our discussion on Slack (HF internal)
https://huggingface.slack.com/archives/C014N4749J9/p1609668119337400

The next step is to add a compatible option in the new `run_glue.py`

I duplicated `glue` and made the following changes:
1. Change the name to `glue_compat`.
2. Change the label assignments for MNLI and AX.
GitHub issue #1741: error when run fine_tuning on text_classification
State: closed | Created: 2021-01-16T02:23:19Z

dataset:sem_eval_2014_task_1
pretrained_model:bert-base-uncased

error description:
when i use these resoruce to train fine_tuning a text_classification on sem_eval_2014_task_1,there always be some problem(when i use other dataset ,there exist the error too). And i followed the colab code (url:https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb#scrollTo=TlqNaB8jIrJW).


the error is like this :
`File "train.py", line 69, in <module>
    trainer.train()
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/transformers/trainer.py", line 784, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
KeyError: 2`

this is my code :
```dataset_name = 'sem_eval_2014_task_1'
num_labels_size = 3
batch_size = 4
model_checkpoint = 'bert-base-uncased'
number_train_epoch = 5

def tokenize(batch):
return tokenizer(batch['premise'], batch['hypothesis'], truncation=True, )

def compute_metrics(pred):
labels = pred.label_ids
preds = pred.predictions.argmax(-1)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='micro')
acc = accuracy_score(labels, preds)
return {
'accuracy': acc,
'f1': f1,
'precision': precision,
'recall': recall
}

model = BertForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels_size)
tokenizer = BertTokenizerFast.from_pretrained(model_checkpoint, use_fast=True)

train_dataset = load_dataset(dataset_name, split='train')
test_dataset = load_dataset(dataset_name, split='test')

train_encoded_dataset = train_dataset.map(tokenize, batched=True)
test_encoded_dataset = test_dataset.map(tokenize, batched=True)

args = TrainingArguments(
output_dir='./results',
evaluation_strategy="epoch",
learning_rate=2e-5,
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=batch_size,
num_train_epochs=number_train_epoch,
weight_decay=0.01,
do_predict=True,
)
trainer = Trainer(
model=model,
args=args,
compute_metrics=compute_metrics,
train_dataset=train_encoded_dataset,
eval_dataset=test_encoded_dataset,
tokenizer=tokenizer
)

trainer.train()
trainer.evaluate()
GitHub issue #1738: Conda support
State: closed | Created: 2021-01-14T15:11:25Z

Will push a new version on anaconda cloud every time a tag starting with `v` is pushed (like `v1.2.2`).

Will appear here: https://anaconda.org/huggingface/datasets

Depends on `conda-forge` for now, so the following is required for installation:

```
conda install -c huggingface -c conda-forge datasets
```
GitHub issue #1736: Adjust BrWaC dataset features name
State: closed | Created: 2021-01-13T20:39:04Z

I added this dataset some days ago, and today I used it to train some models and realized that the names of the features aren't so good.

Looking at the current features hierarchy, we have "paragraphs" with a list of "sentences" with a list of "sentences?!". But the actual hierarchy is a "text" with a list of "paragraphs" with a list of "sentences".

I confused myself trying to use the dataset with these names. So I think it's better to change it.
GitHub issue #1728: Add an entry to an arrow dataset
State: closed | Created: 2021-01-12T18:01:47Z

Is it possible to add an entry to a dataset object?

**Motivation: I want to transform the sentences in the dataset and add them to the original dataset**

For example, say we have the following code:

``` python
from datasets import load_dataset

# Load a dataset and print the first examples in the training set
squad_dataset = load_dataset('squad')
print(squad_dataset['train'][0])
```

Is it possible to add an entry to `squad_dataset`? Something like the following?

``` python
squad_dataset.append({'text': "This is a new sentence"})
```

The motivation for doing this is that I want to transform the sentences in the squad dataset and add them to the original dataset.

If the above doesn't work, is there any other way of achieving the motivation mentioned above? Perhaps by creating a new arrow dataset by using the older one and the transformer sentences?
GitHub issue #1726: Offline loading
State: closed | Created: 2021-01-12T15:21:57Z

As discussed in #824 it would be cool to make the library work in offline mode.
Currently if there's not internet connection then modules (datasets or metrics) that have already been loaded in the past can't be loaded and it raises a ConnectionError.
This is because `prepare_module` fetches online for the latest version of the module.

To make it work in offline mode one suggestion was to reload the latest local version of the module.
I implemented that and I also raise a warning saying that the module that is loaded is the latest local version.
```python
logger.warning(
    f"Using the latest cached version of the module from {cached_module_path} since it "
    f"couldn't be found locally at {input_path} or remotely ({error_type_that_prevented_reaching_out_remote_stuff})."
)
```

I added tests to make sure it works as expected and I needed to do a few changes in the code to be able to test things properly. In particular I added a parameter `hf_modules_cache` to `init_dynamic_modules` for testing purposes. It makes it possible to have temporary modules caches for testing.

I also added a `offline` context utility that allows to test part of the code by making all the requests fail as if there was no internet.

Close #824, close #761.
GitHub issue #1725: load the local dataset
State: closed | Created: 2021-01-12T12:12:55Z

your guidebook's example is like
>>>from datasets import load_dataset
>>> dataset = load_dataset('json', data_files='my_file.json')
but the first arg is path...
so how should i do if i want to load the local dataset for model training?
i will be grateful if you can help me handle this problem!
thanks a lot!
GitHub issue #1723: ADD S3 support for downloading and uploading processed datasets
State: closed | Created: 2021-01-12T07:17:34Z

# What does this PR do?

This PR adds the functionality to load and save `datasets` from and to s3. 
You can save `datasets` with either `Dataset.save_to_disk()` or `DatasetDict.save_to_disk`. 
You can load `datasets` with either `load_from_disk` or `Dataset.load_from_disk()`, `DatasetDict.load_from_disk()`. 

Loading `csv` or `json` datasets from s3 is not implemented. 

To save/load datasets to s3 you either need to provide an `aws_profile`, which is set up on your machine, per default it uses the `default` profile or you have to pass an `aws_access_key_id` and `aws_secret_access_key`. 

The implementation was done with the `fsspec` and `boto3`.


### Example `aws_profile` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")
```

</details>


### Example `aws_access_key_id` and `aws_secret_access_key` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk",
                            aws_access_key_id="fake_access_key", 
                            aws_secret_access_key="fake_secret_key"
                           )

load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk",
                            aws_access_key_id="fake_access_key", 
                            aws_secret_access_key="fake_secret_key"
                           )
```

</details>

If you want to load a dataset from a public s3 bucket you can pass `anon=True` 

### Example  `anon=True` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

load_from_disk("s3://moto-mock-s3-bucketdatasets/sdk",anon=True)
```

</details>

### Full Example

```python
import datasets

dataset = datasets.load_dataset("imdb")
print(f"DatasetDict contains {len(dataset)} datasets")
print(f"train Dataset has the size of: {len(dataset['train'])}")

dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

remote_dataset = datasets.load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")
print(f"DatasetDict contains {len(remote_dataset)} datasets")
print(f"train Dataset has the size of: {len(remote_dataset['train'])}")
```

Related to #878 


I would also adjust the documentation after the code would be reviewed, as long as I leave the PR in "draft" status. Something that we can consider is renaming the functions and changing the `_disk` maybe to `_filesystem`
GitHub issue #1724: could not run models on a offline server successfully
State: closed | Created: 2021-01-12T06:08:06Z

Hi, I really need your help about this.
I am trying to fine-tuning a RoBERTa on a remote server, which is strictly banning internet. I try to install all the packages by hand and try to run run_mlm.py on the server. It works well on colab, but when I try to run it on this offline server, it shows:
![image](https://user-images.githubusercontent.com/49967236/104276256-25a88600-546a-11eb-9776-8ec695dfa24e.png)

is there anything I can do? Is it possible to download all the things in cache and upload it to the server? Please help me out...
GitHub issue #1721: [Scientific papers] Mirror datasets zip
State: closed | Created: 2021-01-12T01:15:40Z

Datasets were uploading to https://s3.amazonaws.com/datasets.huggingface.co/scientific_papers/1.1.1/arxiv-dataset.zip and https://s3.amazonaws.com/datasets.huggingface.co/scientific_papers/1.1.1/pubmed-dataset.zip respectively to escape google drive quota and enable faster download.
GitHub issue #1720: Adding the NorNE dataset for NER
State: closed | Created: 2021-01-11T21:34:13Z

NorNE is a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names.
GitHub issue #1719: Fix column list comparison in transmit format
State: closed | Created: 2021-01-11T17:23:56Z

As noticed in #1718 the cache might not reload the cache files when new columns were added.
This is because of an issue in `transmit_format` where the column list comparison fails because the order was not deterministic. This causes the `transmit_format` to apply an unnecessary `set_format` transform with shuffled column names.

I fixed that by sorting the columns for the comparison and added a test.

To properly test that I added a third column `col_3` to the dummy_dataset used for tests.
GitHub issue #1718: Possible cache miss in datasets
State: closed | Created: 2021-01-11T15:37:31Z

Hi,

I am using the datasets package and even though I run the same data processing functions, datasets always recomputes the function instead of using cache.
I have attached an example script that for me reproduces the problem.
In the attached example the second map function always recomputes instead of loading from cache.
Is this a bug or am I doing something wrong?
Is there a way for fix this and avoid all the recomputation?

Thanks

Edit:
transformers==3.5.1
datasets==1.2.0

```
from datasets import load_dataset
from transformers import AutoTokenizer

datasets = load_dataset('wikitext', 'wikitext-103-raw-v1')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)


column_names = datasets["train"].column_names
text_column_name = "text" if "text" in column_names else column_names[0]
def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=60,
    remove_columns=[text_column_name],
    load_from_cache_file=True,
)
max_seq_length = tokenizer.model_max_length
def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {
        k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
    # customize this part to your needs.
    total_length = (total_length // max_seq_length) * max_seq_length
    # Split by chunks of max_len.
    result = {
        k: [t[i: i + max_seq_length]
            for i in range(0, total_length, max_seq_length)]
        for k, t in concatenated_examples.items()
    }
    return result

tokenized_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    num_proc=60,
    load_from_cache_file=True,
)
print(tokenized_datasets)

print('finished')
```
GitHub issue #1717: SciFact dataset - minor changes
State: closed | Created: 2021-01-11T05:26:40Z

Hi,

SciFact dataset creator here. First of all, thanks for adding the dataset to Huggingface, much appreciated!

I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?

It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?

Thanks,

Dave
GitHub issue #1713: Installation using conda
State: closed | Created: 2021-01-08T19:12:15Z

Will a conda package for installing datasets be added to the huggingface conda channel? I have installed transformers using conda and would like to use the datasets library to use some of the scripts in the transformers/examples folder but am unable to do so at the moment as datasets can only be installed using pip and using pip in a conda environment is generally a bad idea in my experience.
GitHub issue #1711: Fix windows path scheme in cached path
State: closed | Created: 2021-01-08T13:45:56Z

As noticed in #807 there's currently an issue with `cached_path` not raising `FileNotFoundError` on windows for absolute paths. This is due to the way we check for a path to be local or not. The check on the scheme using urlparse was incomplete.

I fixed this and added tests
GitHub issue #1710: IsADirectoryError when trying to download C4
State: closed | Created: 2021-01-08T07:31:30Z

**TLDR**:

I fail to download C4 and see a stacktrace originating in `IsADirectoryError` as an explanation for failure.

How can the problem be fixed? 

**VERBOSE**:

I use Python version 3.7 and have the following dependencies listed in my project:

```
datasets==1.2.0
apache-beam==2.26.0
```

When running the following code, where `/data/huggingface/unpacked/` contains a single unzipped `wet.paths` file manually downloaded as per the instructions for C4:

```
from datasets import load_dataset

load_dataset("c4", "en", data_dir="/data/huggingface/unpacked", beam_runner='DirectRunner')
```

I get the following stacktrace:

```
/Users/fredriko/venv/misc/bin/python /Users/fredriko/source/misc/main.py
Downloading and preparing dataset c4/en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /Users/fredriko/.cache/huggingface/datasets/c4/en/2.3.0/8304cf264cc42bdebcb13fca4b9cb36368a96f557d36f9dc969bebbe2568b283...
Traceback (most recent call last):
  File "/Users/fredriko/source/misc/main.py", line 3, in <module>
    load_dataset("c4", "en", data_dir="/data/huggingface/unpacked", beam_runner='DirectRunner')
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 1066, in _download_and_prepare
    pipeline=pipeline,
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 582, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/fredriko/.cache/huggingface/modules/datasets_modules/datasets/c4/8304cf264cc42bdebcb13fca4b9cb36368a96f557d36f9dc969bebbe2568b283/c4.py", line 190, in _split_generators
    file_paths = dl_manager.download_and_extract(files_to_download)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 258, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 189, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 117, in _record_sizes_checksums
    self._recorded_sizes_checksums[str(url)] = get_size_checksum_dict(path)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 80, in get_size_checksum_dict
    with open(path, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/'

Process finished with exit code 1
```
GitHub issue #1709: Databases
State: closed | Created: 2021-01-08T06:14:03Z

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1708: <html dir="ltr" lang="en" class="focus-outline-visible"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
State: closed | Created: 2021-01-07T21:45:24Z

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1707: Added generated READMEs for datasets that were missing one.
State: closed | Created: 2021-01-07T18:10:06Z

This is it: we worked on a generator with Yacine @yjernite , and we generated dataset cards for all missing ones (161), with all the information we could gather from datasets repository, and using dummy_data to generate examples when possible.

Code is available here for the moment: https://github.com/madlag/datasets_readme_generator .
We will move it to a Hugging Face repository and to https://huggingface.co/datasets/card-creator/ later.