Repository: huggingface/transformers

Issue:
Trainer fails to correctly tackle XLNetForSequenceClassification outputs
## Environment info

- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-117-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: Yes, with CUDA_VISIBLE_DEVICES=0
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger, @TevenLeScao 
 
## Information

Model I am using (Bert, XLNet ...): XLNet-base-cased

The problem arises when using:
*  the official example scripts: ```text-classification/run_glue.py```

The tasks I am working on is:
*  an official GLUE/SQUaD task: SST-2

It seems that XLNetForSequenceClassification has different result outputs compared with other models, which makes the trainer fail to correctly tackle them. 

## To reproduce

Steps to reproduce the behavior:

1.  Install ```transformers``` from master and download SST-2 data using ```download_glue_data.py```
2.  Create the following script
```bash
GLUE_DIR=~/glue
CUDA_VISIBLE_DEVICES=0
TASK_NAME=SST-2

python3 ~/applications/transformers/examples/text-classification/run_glue.py \
  --model_name_or_path ~/xlnet \
  --task_name $TASK_NAME \
  --do_eval \
  --data_dir $GLUE_DIR/$TASK_NAME \
  --max_seq_length 64 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir ~/result/$TASK_NAME/
```
3. Run this script to make predictions


## Expected behavior

Trainer should return the correct evaluation results like other models.

## Observed behavior

```bash
10/02/2020 22:33:53 - INFO - filelock -   Lock 140365777899232 acquired on /data/home/liusishun/glue/SST-2/cached_dev_XLNetTokenizer_64_sst-2.lock
10/02/2020 22:33:53 - INFO - filelock -   Lock 140365777899232 released on /data/home/liusishun/glue/SST-2/cached_dev_XLNetTokenizer_64_sst-2.lock
10/02/2020 22:33:56 - INFO - __main__ -   *** Evaluate ***
Evaluation:   0%|                                                                                                                                                                   | 0/109 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/data/home/liusishun/applications/transformers/examples/text-classification/run_glue.py", line 247, in <module>
    main()
  File "/data/home/liusishun/applications/transformers/examples/text-classification/run_glue.py", line 197, in main
    eval_result = trainer.evaluate(eval_dataset=eval_dataset)
  File "/data/home/liusishun/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1296, in evaluate
    output = self.prediction_loop(eval_dataloader, description="Evaluation")
  File "/data/home/liusishun/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1376, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only)
  File "/data/home/liusishun/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1473, in prediction_step
    logits = tuple(logit.detach() for logit in logits)
  File "/data/home/liusishun/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1473, in <genexpr>
    logits = tuple(logit.detach() for logit in logits)
AttributeError: 'tuple' object has no attribute 'detach'
```


Discussion:

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -48,6 +48,7 @@
     distributed_broadcast_scalars,
     distributed_concat,
     nested_concat,
+    nested_detach,
     nested_numpify,
     nested_xla_mesh_reduce,
     set_seed,
@@ -1466,16 +1467,18 @@ def prediction_step(
                 logits = outputs[:]
             if self.args.past_index >= 0:
                 self._past = outputs[self.args.past_index if has_labels else self.args.past_index - 1]
+                # Remove the past from the logits.
+                logits = logits[: self.args.past_index - 1] + logits[self.args.past_index :]
 
         if prediction_loss_only:
             return (loss, None, None)
 
-        logits = tuple(logit.detach() for logit in logits)
+        logits = nested_detach(logits)
         if len(logits) == 1:
             logits = logits[0]
 
         if has_labels:
-            labels = tuple(inputs.get(name).detach() for name in self.label_names)
+            labels = nested_detach(tuple(inputs.get(name) for name in self.label_names))
             if len(labels) == 1:
                 labels = labels[0]
         else:
diff --git a/src/transformers/trainer_utils.py b/src/transformers/trainer_utils.py
--- a/src/transformers/trainer_utils.py
+++ b/src/transformers/trainer_utils.py
@@ -154,6 +154,13 @@ def nested_concat(tensors, new_tensors, dim=0):
         raise ImportError("Torch must be installed to use `nested_concat`")
 
 
+def nested_deatch(tensors):
+    "Detach `tensors` (even if it's a nested list/tuple of tensors)."
+    if isinstance(tensors, (list, tuple)):
+        return type(tensors)(nested_detach(t) for t in tensors)
+    return tensors.detach()
+
+
 def nested_numpify(tensors):
     "Numpify `tensors` (even if it's a nested list/tuple of tensors)."
     if isinstance(tensors, (list, tuple)):


Test patch: