Repository: huggingface/transformers (Python)
Date: 2020-06-16 13:28:18+00:00
Task type: Bug Fix

Issue:
DataCollator problem
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi everybody
I found an error in the following Colab: 
https://colab.research.google.com/drive/1jwXgtOXE8v8_qkiOCbjFQRFC5semK8T7?usp=sharing

Specifically, As far I understand something changed with the implementation of the following snippet:

class T2TDataCollator(DataCollator):
    def collate_batch(self, batch: List) -> Dict[str, torch.Tensor]:
         ..........
<br>

I got the following error: **TypeError: function() argument 1 must be code, not str**

Can you suggest any workarounds?


Issue discussion:
i have the same. It is new bug. i run this week ago and worked
try this:
```python
class T2TDataCollator:
    def __call__(self, batch):
```

@abrozso Hi and thanks for the hint, however, it doesn't seem to fix the problem.
I got the following error when the fine-tuning starts:

06/16/2020 09:03:23 - INFO - transformers.trainer -   You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.
06/16/2020 09:03:23 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
06/16/2020 09:03:23 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
06/16/2020 09:03:23 - INFO - transformers.trainer -   ***** Running training *****
06/16/2020 09:03:23 - INFO - transformers.trainer -     Num examples = 13
06/16/2020 09:03:23 - INFO - transformers.trainer -     Num Epochs = 4
06/16/2020 09:03:23 - INFO - transformers.trainer -     Instantaneous batch size per device = 8
06/16/2020 09:03:23 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 64
06/16/2020 09:03:23 - INFO - transformers.trainer -     Gradient Accumulation steps = 4
06/16/2020 09:03:23 - INFO - transformers.trainer -     Total optimization steps = 0
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/parallel_loader.py", line 141, in _loader_worker
    _, data = next(data_iter)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 352, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 392, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
TypeError: 'T2TDataCollator' object is not callable

@antoniomastro1996: perhaps you can try the xla nightly version (if you are not using that already)
@abrozso unfortunately, I'm already using the nightly version
You need to instantiate your `T2TDataCollator`: `data_collator = T2TDataCollator()` (or you could make it a simple function if you don't need any state).
Will fix the backward-compatibility this morning.

Solution patch:
diff --git a/src/transformers/data/data_collator.py b/src/transformers/data/data_collator.py
--- a/src/transformers/data/data_collator.py
+++ b/src/transformers/data/data_collator.py
@@ -33,31 +33,34 @@ def default_data_collator(features: List[InputDataClass]) -> Dict[str, torch.Ten
     # have the same attributes.
     # So we will look at the first element as a proxy for what attributes exist
     # on the whole batch.
+    if not isinstance(features[0], dict):
+        features = [vars(f) for f in features]
+
     first = features[0]
+    batch = {}
 
     # Special handling for labels.
     # Ensure that tensor is created with the correct type
     # (it should be automatically the case, but let's make sure of it.)
-    if hasattr(first, "label") and first.label is not None:
-        if type(first.label) is int:
-            labels = torch.tensor([f.label for f in features], dtype=torch.long)
-        else:
-            labels = torch.tensor([f.label for f in features], dtype=torch.float)
-        batch = {"labels": labels}
-    elif hasattr(first, "label_ids") and first.label_ids is not None:
-        if type(first.label_ids[0]) is int:
-            labels = torch.tensor([f.label_ids for f in features], dtype=torch.long)
+    if "label" in first:
+        dtype = torch.long if type(first["label"]) is int else torch.float
+        batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
+    elif "label_ids" in first:
+        if isinstance(first["label_ids"], torch.Tensor):
+            batch["labels"] = torch.stack([f["label_ids"] for f in features])
         else:
-            labels = torch.tensor([f.label_ids for f in features], dtype=torch.float)
-        batch = {"labels": labels}
-    else:
-        batch = {}
+            dtype = torch.long if type(first["label_ids"][0]) is int else torch.float
+            batch["labels"] = torch.tensor([f["label_ids"] for f in features], dtype=dtype)
 
-    # Handling of all other possible attributes.
+    # Handling of all other possible keys.
     # Again, we will use the first element to figure out which key/values are not None for this model.
-    for k, v in vars(first).items():
+    for k, v in first.items():
         if k not in ("label", "label_ids") and v is not None and not isinstance(v, str):
-            batch[k] = torch.tensor([getattr(f, k) for f in features], dtype=torch.long)
+            if isinstance(v, torch.Tensor):
+                batch[k] = torch.stack([f[k] for f in features])
+            else:
+                batch[k] = torch.tensor([f[k] for f in features], dtype=torch.long)
+
     return batch
 
 
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -4,6 +4,7 @@
 import random
 import re
 import shutil
+import warnings
 from contextlib import contextmanager
 from pathlib import Path
 from typing import Callable, Dict, List, Optional, Tuple
@@ -205,6 +206,15 @@ def __init__(
             # Set an xla_device flag on the model's config.
             # We'll find a more elegant and not need to do this in the future.
             self.model.config.xla_device = True
+        if not callable(self.data_collator) and callable(getattr(self.data_collator, "collate_batch", None)):
+            self.data_collator = self.data_collator.collate_batch
+            warnings.warn(
+                (
+                    "The `data_collator` should now be a simple callable (function, class with `__call__`), classes "
+                    + "with a `collate_batch` are deprecated and won't be supported in a future version."
+                ),
+                FutureWarning,
+            )
 
     def get_train_dataloader(self) -> DataLoader:
         if self.train_dataset is None:


Test patch:
diff --git a/tests/test_trainer.py b/tests/test_trainer.py
--- a/tests/test_trainer.py
+++ b/tests/test_trainer.py
@@ -24,6 +24,27 @@
 
 @require_torch
 class DataCollatorIntegrationTest(unittest.TestCase):
+    def test_default_with_dict(self):
+        features = [{"labels": i, "inputs": [0, 1, 2, 3, 4, 5]} for i in range(8)]
+        batch = default_data_collator(features)
+        self.assertTrue(batch["labels"].equal(torch.tensor(list(range(8)))))
+        self.assertEqual(batch["labels"].dtype, torch.long)
+        self.assertEqual(batch["inputs"].shape, torch.Size([8, 6]))
+
+        # With label_ids
+        features = [{"label_ids": [0, 1, 2], "inputs": [0, 1, 2, 3, 4, 5]} for i in range(8)]
+        batch = default_data_collator(features)
+        self.assertTrue(batch["labels"].equal(torch.tensor([[0, 1, 2]] * 8)))
+        self.assertEqual(batch["labels"].dtype, torch.long)
+        self.assertEqual(batch["inputs"].shape, torch.Size([8, 6]))
+
+        # Features can already be tensors
+        features = [{"labels": i, "inputs": torch.randint(10, [10])} for i in range(8)]
+        batch = default_data_collator(features)
+        self.assertTrue(batch["labels"].equal(torch.tensor(list(range(8)))))
+        self.assertEqual(batch["labels"].dtype, torch.long)
+        self.assertEqual(batch["inputs"].shape, torch.Size([8, 10]))
+
     def test_default_classification(self):
         MODEL_ID = "bert-base-cased-finetuned-mrpc"
         tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)