Repository: huggingface/transformers (Python)
Date: 2020-06-18 20:18:36+00:00
Task type: Bug Fix

Issue:
data_collator.py does not allow NoneType labels for test set predictions on Glue
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Distilbert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below) 
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)


## To reproduce

Steps to reproduce the behavior:

1. Open the  example Colab for text-classification from here https://huggingface.co/transformers/examples.html
2. Try to run the prediction function found in run_glue.py to predict on the official Glue test set. 
3. The error is as shown below. 

Earlier, this has worked with the exact same program, ever since an update recently, this error shows up. 

`TypeError                                 Traceback (most recent call last)
<ipython-input-16-9eecdd4d48b1> in <module>()
      2 output_mode = "classification"
      3 
----> 4 predictions = trainer.predict(test_dataset=test_dataset).predictions
      5 if output_mode == "classification":
      6     predictions = np.argmax(predictions, axis=1)

7 frames
/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in default_data_collator(features)
     45     if "label" in first:
     46         dtype = torch.long if type(first["label"]) is int else torch.float
---> 47         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
     48     elif "label_ids" in first:
     49         if isinstance(first["label_ids"], torch.Tensor):

TypeError: must be real number, not NoneType`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The error can be seen in the Colab notebook here https://colab.research.google.com/drive/1H_92qdsOOql2hS210qNrfMEEMRcAoHD_?usp=sharing

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Colab
- Python version: NA
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:

data_collator.py does not allow NoneType labels for test set predictions on Glue
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Distilbert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below) 
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)


## To reproduce

Steps to reproduce the behavior:

1. Open the  example Colab for text-classification from here https://huggingface.co/transformers/examples.html
2. Try to run the prediction function found in run_glue.py to predict on the official Glue test set. 
3. The error is as shown below. 

Earlier, this has worked with the exact same program, ever since an update recently, this error shows up. 

`TypeError                                 Traceback (most recent call last)
<ipython-input-16-9eecdd4d48b1> in <module>()
      2 output_mode = "classification"
      3 
----> 4 predictions = trainer.predict(test_dataset=test_dataset).predictions
      5 if output_mode == "classification":
      6     predictions = np.argmax(predictions, axis=1)

7 frames
/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in default_data_collator(features)
     45     if "label" in first:
     46         dtype = torch.long if type(first["label"]) is int else torch.float
---> 47         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
     48     elif "label_ids" in first:
     49         if isinstance(first["label_ids"], torch.Tensor):

TypeError: must be real number, not NoneType`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The error can be seen in the Colab notebook here https://colab.research.google.com/drive/1H_92qdsOOql2hS210qNrfMEEMRcAoHD_?usp=sharing

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Colab
- Python version: NA
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:



Issue discussion:

Solution patch:
diff --git a/src/transformers/data/data_collator.py b/src/transformers/data/data_collator.py
--- a/src/transformers/data/data_collator.py
+++ b/src/transformers/data/data_collator.py
@@ -42,10 +42,10 @@ def default_data_collator(features: List[InputDataClass]) -> Dict[str, torch.Ten
     # Special handling for labels.
     # Ensure that tensor is created with the correct type
     # (it should be automatically the case, but let's make sure of it.)
-    if "label" in first:
+    if "label" in first and first["label"] is not None:
         dtype = torch.long if type(first["label"]) is int else torch.float
         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
-    elif "label_ids" in first:
+    elif "label_ids" in first and first["label_ids"] is not None:
         if isinstance(first["label_ids"], torch.Tensor):
             batch["labels"] = torch.stack([f["label_ids"] for f in features])
         else:


Test patch:
diff --git a/tests/test_trainer.py b/tests/test_trainer.py
--- a/tests/test_trainer.py
+++ b/tests/test_trainer.py
@@ -25,7 +25,7 @@
 @require_torch
 class DataCollatorIntegrationTest(unittest.TestCase):
     def test_default_with_dict(self):
-        features = [{"labels": i, "inputs": [0, 1, 2, 3, 4, 5]} for i in range(8)]
+        features = [{"label": i, "inputs": [0, 1, 2, 3, 4, 5]} for i in range(8)]
         batch = default_data_collator(features)
         self.assertTrue(batch["labels"].equal(torch.tensor(list(range(8)))))
         self.assertEqual(batch["labels"].dtype, torch.long)
@@ -39,12 +39,24 @@ def test_default_with_dict(self):
         self.assertEqual(batch["inputs"].shape, torch.Size([8, 6]))
 
         # Features can already be tensors
-        features = [{"labels": i, "inputs": torch.randint(10, [10])} for i in range(8)]
+        features = [{"label": i, "inputs": torch.randint(10, [10])} for i in range(8)]
         batch = default_data_collator(features)
         self.assertTrue(batch["labels"].equal(torch.tensor(list(range(8)))))
         self.assertEqual(batch["labels"].dtype, torch.long)
         self.assertEqual(batch["inputs"].shape, torch.Size([8, 10]))
 
+    def test_default_with_no_labels(self):
+        features = [{"label": None, "inputs": [0, 1, 2, 3, 4, 5]} for i in range(8)]
+        batch = default_data_collator(features)
+        self.assertTrue("labels" not in batch)
+        self.assertEqual(batch["inputs"].shape, torch.Size([8, 6]))
+
+        # With label_ids
+        features = [{"label_ids": None, "inputs": [0, 1, 2, 3, 4, 5]} for i in range(8)]
+        batch = default_data_collator(features)
+        self.assertTrue("labels" not in batch)
+        self.assertEqual(batch["inputs"].shape, torch.Size([8, 6]))
+
     def test_default_classification(self):
         MODEL_ID = "bert-base-cased-finetuned-mrpc"
         tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)