Issue 1:
🐛 Bug

@sshleifer

I'm using BART model (bart-large), when I try to use the BartForMaskedLM i'm getting the above error. The reason is that in the _combine_masks (line 146 in modeling_bart) is creating a tensor without the device. so by default it is on CPU. To reproduce - simply use BartForMaskedLM model with GPU.

can you help? am I missing anything?

Additional details:

transformers version: 2.5.1

Python version: 3.7.4

PyTorch version (GPU?): 1.3.0 (with GPU)

Using GPU in script?: yes

Using distributed or parallel set-up in script?: no

stuck trace:

File "/specific/netapp5_2/gamir/adi/git/BERTese/lama/training.py", line 151, in train_and_eval

outputs = model(b_in_tensor, lm_labels=b_label_tensor)

File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call

result = self.forward(*input, **kwargs)

File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 925, in forward

decoder_cached_states=decoder_cached_states,

File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 844, in forward

decoder_cached_states=decoder_cached_states,

File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 499, in forward

need_attn_weights=self.output_attentions,

File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 372, in forward

attn_mask=attention_mask,

File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 629, in forward

attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask

RuntimeError: expected device cuda:0 but got device cpu

Thanks,

Adi.

Issue 2:
🐛 Bug

Information

Model I am using (Bert, XLNet ...): BART - bart-large

Language I am using the model on (English, Chinese ...): English

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

Load model

Tokenize text

Send model and tensor to cuda

Forward the model

from transformers import BartConfig, BartTokenizer, BartForMaskedLM, BartModel

configuration = BartConfig()
tokenizer_class = BartTokenizer
model_class = BartForMaskedLM(configuration)

tokenizer = tokenizer_class.from_pretrained('bart-large')
model = model_class.from_pretrained('bart-large')

model.eval()
model.to('cuda')

tokens = tokenizer.encode("Text example to test natural language generation with bart.")
input_ids = torch.tensor([tokens])
input_ids = input_ids.to('cuda')

with torch.no_grad():
last_hidden_states = model(input_ids)[0]

print("Len tokens:", len(tokens))
print("Shape last hidden states:", last_hidden_states.shape)

This code raises the following error:

---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-24-3bba66ed6aeb> in <module>
3
4 with torch.no_grad():
----> 5 last_hidden_states = model(input_ids)[0]
6
7 print("Len tokens:", len(tokens))

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
--> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, encoder_outputs, decoder_input_ids, decoder_attention_mask, decoder_cached_states, lm_labels, **unused)
923 encoder_outputs=encoder_outputs,
924 decoder_attention_mask=decoder_attention_mask,
--> 925 decoder_cached_states=decoder_cached_states,
926 )
927 lm_logits = self.lm_head.forward(outputs[0])

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, encoder_outputs, decoder_attention_mask, decoder_cached_states)
842 attention_mask,
843 decoder_attn_mask,
--> 844 decoder_cached_states=decoder_cached_states,
845 )
846 # Attention and hidden_states will be [] or None if they aren't needed

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, encoder_hidden_states, encoder_padding_mask, combined_mask, decoder_cached_states, **unused)
497 decoder_cached_states=layer_state,
498 attention_mask=combined_mask,
--> 499 need_attn_weights=self.output_attentions,
500 )
501 if self.output_past:

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, x, encoder_hidden_states, encoder_attn_mask, decoder_cached_states, attention_mask, need_attn_weights)
370 decoder_cached_states=decoder_cached_states,
371 need_weights=need_attn_weights,
--> 372 attn_mask=attention_mask,
373 )
374 x = F.dropout(x, p=self.dropout, training=self.training)

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, query, key, value, key_padding_mask, decoder_cached_states, need_weights, static_kv, attn_mask)
627
628 if attn_mask is not None:
--> 629 attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask
630 attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
631

RuntimeError: expected device cuda:0 but got device cpu

But I have tested this code with another examples (like GPT-2) and it works.

Expected behavior

I would expect to get the tensor size, as with another models I have tested this code.

Environment info

transformers version: 2.5.1

Platform: Linux

Python version: 3.6.7

PyTorch version (GPU?): 1.4.0. GPU: Yes

Tensorflow version (GPU?): No used

Using GPU in script?: Trying to

Using distributed or parallel set-up in script?: No