DataCollator problem
Author: ghost
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi everybody
I found an error in the following Colab: 
https://colab.research.google.com/drive/1jwXgtOXE8v8_qkiOCbjFQRFC5semK8T7?usp=sharing

Specifically, As far I understand something changed with the implementation of the following snippet:

class T2TDataCollator(DataCollator):
    def collate_batch(self, batch: List) -> Dict[str, torch.Tensor]:
         ..........
<br>

I got the following error: **TypeError: function() argument 1 must be code, not str**

Can you suggest any workarounds?

Comments:
i have the same. It is new bug. i run this week ago and worked

try this:
```python
class T2TDataCollator:
    def __call__(self, batch):
```


@abrozso Hi and thanks for the hint, however, it doesn't seem to fix the problem.
I got the following error when the fine-tuning starts:

06/16/2020 09:03:23 - INFO - transformers.trainer -   You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.
06/16/2020 09:03:23 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
06/16/2020 09:03:23 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
06/16/2020 09:03:23 - INFO - transformers.trainer -   ***** Running training *****
06/16/2020 09:03:23 - INFO - transformers.trainer -     Num examples = 13
06/16/2020 09:03:23 - INFO - transformers.trainer -     Num Epochs = 4
06/16/2020 09:03:23 - INFO - transformers.trainer -     Instantaneous batch size per device = 8
06/16/2020 09:03:23 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 64
06/16/2020 09:03:23 - INFO - transformers.trainer -     Gradient Accumulation steps = 4
06/16/2020 09:03:23 - INFO - transformers.trainer -     Total optimization steps = 0
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/parallel_loader.py", line 141, in _loader_worker
    _, data = next(data_iter)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 352, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 392, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
TypeError: 'T2TDataCollator' object is not callable


@antoniomastro1996: perhaps you can try the xla nightly version (if you are not using that already)

@abrozso unfortunately, I'm already using the nightly version

You need to instantiate your `T2TDataCollator`: `data_collator = T2TDataCollator()` (or you could make it a simple function if you don't need any state).
Will fix the backward-compatibility this morning.

The issue is that it is not a class anymore.

Yes, that will stay. Just remove the subclass to `DataCollator` and everything should work:
```
class MyDataCollator:
    def __call__(self, features): ...
```
or (once #5060 is merged)
```
class MyDataCollator:
    def collate_batch(self, features): ...
```
but this will throw a deprecation warning.

I respectfully disagree with the decision to keep `DataCollator` as a callable. Very many existing trainers and notebooks will break as a result. I think many people would agree that it would be best to create a `DataCollatorCallable` callable or something similar as an addition, not as a replacement.

Hi all,

I'm facing issues with this part of the code (post making changes as suggested above) in T5-Base for QA.

```
import dataclasses
import logging
import os
import sys
from dataclasses import dataclass, field
from typing import Dict, List, Optional

import numpy as np
import torch

from transformers import T5ForConditionalGeneration, T5Tokenizer, EvalPrediction
from transformers import (
    HfArgumentParser,
    DataCollator,
    Trainer,
    TrainingArguments,
    set_seed,
)


logger = logging.getLogger(__name__)

# prepares lm_labels from target_ids, returns examples with keys as expected by the forward method
# this is necessacry because the trainer directly passes this dict as arguments to the model
# so make sure the keys match the parameter names of the forward method
@dataclass
class T2TDataCollator: #(DataCollator)
    def __call__(self, batch: List) -> Dict[str, torch.Tensor]: #
        """
        Take a list of samples from a Dataset and collate them into a batch.
        Returns:
            A dictionary of tensors
        """
        input_ids = torch.stack([example['input_ids'] for example in batch])
        lm_labels = torch.stack([example['target_ids'] for example in batch])
        lm_labels[lm_labels[:, :] == 0] = -100
        attention_mask = torch.stack([example['attention_mask'] for example in batch])
        decoder_attention_mask = torch.stack([example['target_attention_mask'] for example in batch])
        

        return {
            'input_ids': input_ids, 
            'attention_mask': attention_mask,
            'lm_labels': lm_labels, 
            'decoder_attention_mask': decoder_attention_mask
        }
```

**Which is fetching this error:-**

```
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/parallel_loader.py", line 133, in _loader_worker
    _, data = next(data_iter)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "<ipython-input-7-7b8c1b4d4c9a>", line 36, in __call__
    lm_labels = torch.stack([example['target_ids'] for example in batch])
  File "<ipython-input-7-7b8c1b4d4c9a>", line 36, in <listcomp>
    lm_labels = torch.stack([example['target_ids'] for example in batch])
KeyError: 'target_ids'
```

My train and validation dataset has 'target_ids' field (read from `datasets.Dataset.from_pandas()` method and mapped the `add_eos_to_examples` and `convert_to_features` successfully):

`train_dataset['target_ids']`
```
tensor([[ 1027,  9533,  3440,  ...,     0,     0,     0],
        [ 7327,  1387, 11597,  ...,     0,     0,     0],
        [  272,     5,  7130,  ...,     0,     0,     0],
        ...,
        [15810,     1,     0,  ...,     0,     0,     0],
        [ 7107,     1,     0,  ...,     0,     0,     0],
        [  454,     5,   134,  ...,     0,     0,     0]])
```

`valid_dataset['target_ids']`
```
tensor([[15810,     1,     0,  ...,     0,     0,     0],
        [ 4190,  4329,     1,  ...,     0,     0,     0],
        [ 4329,    11,  7107,  ...,     0,     0,     0],
        ...,
        [    3,     4,     1,  ...,     0,     0,     0],
        [    3,     4,     1,  ...,     0,     0,     0],
        [ 8642,  4425,     9,  ...,     0,     0,     0]])
```

I am unable to fetch this field using `class T2TDataCollator:`. Please assist, thank you!