Issue 1:
Environment info

transformers version: 3.5.1

Platform: Linux version 3.10.107-1-tlinux2-0050

Python version: 3.7.6

PyTorch version (GPU?): 1.6.0+cu101

Tensorflow version (GPU?): no

Using GPU in script?: yes

Using distributed or parallel set-up in script?: yes

Who can help

@sgugger

To reproduce

CUDA_VISIBLE_DEVICES=1,2 python run_glue.py \
--model_name_or_path xlnet-base-cased \
--task_name stsb \
--do_eval \
--max_seq_length 128 \
--per_device_train_batch_size 32 \
--per_device_eval_batch_size 8 \
--max_steps 1200 \
--learning_rate 2e-5 \
--num_train_epochs 3.0 \
--output_dir ./output/tranformer/xlnet \
--cache_dir ./pretained_model/xlnet \
--overwrite_output_dir \
--overwrite_cache \
--eval_accumulation_steps 2 \
--gradient_accumulation_steps 1 \
--disable_tqdm True\
--dataloader_drop_last \
--past_index 2

error

[INFO|trainer.py:1387] 2020-11-18 15:21:21,084 >> ***** Running Evaluation *****
[INFO|trainer.py:1388] 2020-11-18 15:21:21,084 >> Num examples = 4000
[INFO|trainer.py:1389] 2020-11-18 15:21:21,085 >> Batch size = 16
./sim/lib/python3.7/site-packages/transformers/modeling_xlnet.py:297: UserWarning: Mixed memory format inputs detected while calling the operator. The operator will output contiguous tensor even if some of the inputs are in channels_last format. (Triggered internally at /pytorch/aten/src/ATen/native/TensorIterator.cpp:918.)
attn_score = (ac + bd + ef) * self.scale
./sim/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
warnings.warn('Was asked to gather along dimension 0, but all '
Traceback (most recent call last):
File "run_glue.py", line 414, in <module>
main()
File "run_glue.py", line 366, in main
eval_result = trainer.evaluate(eval_dataset=eval_dataset)
File "./sim/lib/python3.7/site-packages/transformers/trainer.py", line 1313, in evaluate
prediction_loss_only=True if self.compute_metrics is None else None,
File "./sim/lib/python3.7/site-packages/transformers/trainer.py", line 1431, in prediction_loop
preds_gatherer.add_arrays(self._gather_and_numpify(preds_host, "eval_preds"))
File "./sim/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 330, in add_arrays
slice_len = self._nested_set_tensors(self._storage, arrays)
File "./sim/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 337, in _nested_set_tensors
slice_len = self._nested_set_tensors(x, y)
File "./sim/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 337, in _nested_set_tensors
slice_len = self._nested_set_tensors(x, y)
File "./sim/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 349, in _nested_set_tensors
i * slice_len : (i + 1) * slice_len
ValueError: could not broadcast input array from shape (512,8,768) into shape (416,8,768)
```shell

Could you please help me? Thanks a lot !

Issue 2:
Environment info

transformers version: 3.3.1

Platform: Linux-4.15.0-117-generic-x86_64-with-glibc2.10

Python version: 3.8.5

PyTorch version (GPU?): 1.4.0 (True)

Tensorflow version (GPU?): 2.2.0 (False)

Using GPU in script?: Yes

Using distributed or parallel set-up in script?: No

Who can help

@sgugger

Information

Model I am using (Bert, XLNet ...): XLNet-base-cased

The problem arises when using:

the official example scripts: run_glue.py

The tasks I am working on is:

an official GLUE/SQUaD task: SST-2

To reproduce

Steps to reproduce the behavior:

Install transformers from master and download SST-2 data using download_glue_data.py

Create the following scripts

GLUE_DIR=~/glue
CUDA_VISIBLE_DEVICES=0
TASK_NAME=SST-2

python3 ~/applications/transformers/examples/text-classification/run_glue.py \
--model_name_or_path ~/xlnet \
--task_name $TASK_NAME \
--do_eval \
--data_dir $GLUE_DIR/$TASK_NAME \
--max_seq_length 64 \
--per_device_train_batch_size 32 \
--per_device_eval_batch_size 64 \
--learning_rate 2e-5 \
--num_train_epochs 3.0 \
--output_dir ~/result/$TASK_NAME/ \
--overwrite_output_dir \
--eval_steps 100

run this script

Expected behavior

Trainer should return appropriate evaluation results. Here are logs when evaluating bert-base with above-given hyperparameters.

10/05/2020 22:28:47 - INFO - filelock - Lock 140392033291808 acquired on /data/home/liusishun/glue/SST-2/cached_dev_BertTokenizer_64_sst-2.lock
10/05/2020 22:28:47 - INFO - filelock - Lock 140392033291808 released on /data/home/liusishun/glue/SST-2/cached_dev_BertTokenizer_64_sst-2.lock
10/05/2020 22:28:50 - INFO - __main__ - *** Evaluate ***
Evaluation: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:01<00:00, 7.22it/s]
{'eval_loss': 0.6916399122378148, 'eval_acc': 0.49770642201834864, 'step': 0}
/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer.py:1168: FutureWarning: This method is deprecated, use `Trainer.is_world_process_zero()` instead.
warnings.warn("This method is deprecated, use `Trainer.is_world_process_zero()` instead.", FutureWarning)
10/05/2020 22:28:52 - INFO - __main__ - ***** Eval results sst-2 *****
10/05/2020 22:28:52 - INFO - __main__ - eval_loss = 0.6916399122378148
10/05/2020 22:28:52 - INFO - __main__ - eval_acc = 0.49770642201834864

Observed behavior

10/05/2020 22:30:05 - INFO - filelock - Lock 139928226197216 acquired on /data/home/liusishun/glue/SST-2/cached_dev_XLNetTokenizer_64_sst-2.lock
10/05/2020 22:30:05 - INFO - filelock - Lock 139928226197216 released on /data/home/liusishun/glue/SST-2/cached_dev_XLNetTokenizer_64_sst-2.lock
10/05/2020 22:30:09 - INFO - __main__ - *** Evaluate ***
Evaluation: 93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 13/14 [00:02<00:00, 4.44it/s]
Traceback (most recent call last):
File "/data/home/liusishun/applications/transformers/examples/text-classification/run_glue.py", line 247, in <module>
main()
File "/data/home/liusishun/applications/transformers/examples/text-classification/run_glue.py", line 197, in main
eval_result = trainer.evaluate(eval_dataset=eval_dataset)
File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer.py", line 1297, in evaluate
output = self.prediction_loop(eval_dataloader, description="Evaluation")
File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer.py", line 1382, in prediction_loop
preds = logits if preds is None else nested_concat(preds, logits, dim=0)
File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 151, in nested_concat
return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 151, in <genexpr>
return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 151, in nested_concat
return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 151, in <genexpr>
return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 152, in nested_concat
return torch.cat((tensors, new_tensors), dim=dim)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 40 and 64 in dimension 1 at /opt/conda/conda-bld/pytorch_1579061855666/work/aten/src/THC/generic/THCTensorMath.cu:71