TypeError: Operation 'neg_out_mps()' does not support input type 'int64' in MPS backend.
Author: serdarildercaglar
Pull request: no

### System Info

transformers version: 4.22.2
checkpoint: microsoft/deberta-v3-small
task: ner
error: `TypeError: Operation 'neg_out_mps()' does not support input type 'int64' in MPS backend.
`
![Screen Shot 2022-10-05 at 10 09 18](https://user-images.githubusercontent.com/87153193/194001325-d9eb6b50-4623-4a79-a807-e458f79cbc7e.png)






### Who can help?

@sgugger
`The following columns in the training set don't have a corresponding argument in `DebertaV2ForTokenClassification.forward` and have been ignored: ner_tags, tokens. If ner_tags, tokens are not expected by `DebertaV2ForTokenClassification.forward`,  you can safely ignore this message.
***** Running training *****
  Num examples = 43943
  Num Epochs = 5
  Instantaneous batch size per device = 16
  Total train batch size (w. parallel, distributed & accumulation) = 16
  Gradient Accumulation steps = 1
  Total optimization steps = 13735
Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [44], in <cell line: 1>()
----> 1 trainer.train()

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/trainer.py:1521, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1516     self.model_wrapped = self.model
   1518 inner_training_loop = find_executable_batch_size(
   1519     self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size
   1520 )
-> 1521 return inner_training_loop(
   1522     args=args,
   1523     resume_from_checkpoint=resume_from_checkpoint,
   1524     trial=trial,
   1525     ignore_keys_for_eval=ignore_keys_for_eval,
   1526 )

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/trainer.py:1763, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1761         tr_loss_step = self.training_step(model, inputs)
   1762 else:
-> 1763     tr_loss_step = self.training_step(model, inputs)
   1765 if (
   1766     args.logging_nan_inf_filter
   1767     and not is_torch_tpu_available()
   1768     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
   1769 ):
   1770     # if loss is nan or inf simply add the average of previous logged losses
   1771     tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/trainer.py:2499, in Trainer.training_step(self, model, inputs)
   2496     return loss_mb.reduce_mean().detach().to(self.args.device)
   2498 with self.compute_loss_context_manager():
-> 2499     loss = self.compute_loss(model, inputs)
   2501 if self.args.n_gpu > 1:
   2502     loss = loss.mean()  # mean() to average on multi-gpu parallel training

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/trainer.py:2531, in Trainer.compute_loss(self, model, inputs, return_outputs)
   2529 else:
   2530     labels = None
-> 2531 outputs = model(**inputs)
   2532 # Save past state if it exists
   2533 # TODO: this needs to be fixed and made cleaner later.
   2534 if self.args.past_index >= 0:

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/torch/nn/modules/module.py:1190, in Module._call_impl(self, *input, **kwargs)
   1186 # If we don't have any hooks, we want to skip the rest of the logic in
   1187 # this function, and just call forward.
   1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190     return forward_call(*input, **kwargs)
   1191 # Do not call functions when jit is used
   1192 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:1444, in DebertaV2ForTokenClassification.forward(self, input_ids, attention_mask, token_type_ids, position_ids, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
   1438 r"""
   1439 labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
   1440     Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
   1441 """
   1442 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-> 1444 outputs = self.deberta(
   1445     input_ids,
   1446     attention_mask=attention_mask,
   1447     token_type_ids=token_type_ids,
   1448     position_ids=position_ids,
   1449     inputs_embeds=inputs_embeds,
   1450     output_attentions=output_attentions,
   1451     output_hidden_states=output_hidden_states,
   1452     return_dict=return_dict,
   1453 )
   1455 sequence_output = outputs[0]
   1457 sequence_output = self.dropout(sequence_output)

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/torch/nn/modules/module.py:1190, in Module._call_impl(self, *input, **kwargs)
   1186 # If we don't have any hooks, we want to skip the rest of the logic in
   1187 # this function, and just call forward.
   1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190     return forward_call(*input, **kwargs)
   1191 # Do not call functions when jit is used
   1192 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:1101, in DebertaV2Model.forward(self, input_ids, attention_mask, token_type_ids, position_ids, inputs_embeds, output_attentions, output_hidden_states, return_dict)
   1091     token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=device)
   1093 embedding_output = self.embeddings(
   1094     input_ids=input_ids,
   1095     token_type_ids=token_type_ids,
   (...)
   1098     inputs_embeds=inputs_embeds,
   1099 )
-> 1101 encoder_outputs = self.encoder(
   1102     embedding_output,
   1103     attention_mask,
   1104     output_hidden_states=True,
   1105     output_attentions=output_attentions,
   1106     return_dict=return_dict,
   1107 )
   1108 encoded_layers = encoder_outputs[1]
   1110 if self.z_steps > 1:

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/torch/nn/modules/module.py:1190, in Module._call_impl(self, *input, **kwargs)
   1186 # If we don't have any hooks, we want to skip the rest of the logic in
   1187 # this function, and just call forward.
   1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190     return forward_call(*input, **kwargs)
   1191 # Do not call functions when jit is used
   1192 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:542, in DebertaV2Encoder.forward(self, hidden_states, attention_mask, output_hidden_states, output_attentions, query_states, relative_pos, return_dict)
    533     output_states = torch.utils.checkpoint.checkpoint(
    534         create_custom_forward(layer_module),
    535         next_kv,
   (...)
    539         rel_embeddings,
    540     )
    541 else:
--> 542     output_states = layer_module(
    543         next_kv,
    544         attention_mask,
    545         query_states=query_states,
    546         relative_pos=relative_pos,
    547         rel_embeddings=rel_embeddings,
    548         output_attentions=output_attentions,
    549     )
    551 if output_attentions:
    552     output_states, att_m = output_states

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/torch/nn/modules/module.py:1190, in Module._call_impl(self, *input, **kwargs)
   1186 # If we don't have any hooks, we want to skip the rest of the logic in
   1187 # this function, and just call forward.
   1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190     return forward_call(*input, **kwargs)
   1191 # Do not call functions when jit is used
   1192 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:386, in DebertaV2Layer.forward(self, hidden_states, attention_mask, query_states, relative_pos, rel_embeddings, output_attentions)
    377 def forward(
    378     self,
    379     hidden_states,
   (...)
    384     output_attentions=False,
    385 ):
--> 386     attention_output = self.attention(
    387         hidden_states,
    388         attention_mask,
    389         output_attentions=output_attentions,
    390         query_states=query_states,
    391         relative_pos=relative_pos,
    392         rel_embeddings=rel_embeddings,
    393     )
    394     if output_attentions:
    395         attention_output, att_matrix = attention_output

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/torch/nn/modules/module.py:1190, in Module._call_impl(self, *input, **kwargs)
   1186 # If we don't have any hooks, we want to skip the rest of the logic in
   1187 # this function, and just call forward.
   1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190     return forward_call(*input, **kwargs)
   1191 # Do not call functions when jit is used
   1192 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:317, in DebertaV2Attention.forward(self, hidden_states, attention_mask, output_attentions, query_states, relative_pos, rel_embeddings)
    308 def forward(
    309     self,
    310     hidden_states,
   (...)
    315     rel_embeddings=None,
    316 ):
--> 317     self_output = self.self(
    318         hidden_states,
    319         attention_mask,
    320         output_attentions,
    321         query_states=query_states,
    322         relative_pos=relative_pos,
    323         rel_embeddings=rel_embeddings,
    324     )
    325     if output_attentions:
    326         self_output, att_matrix = self_output

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/torch/nn/modules/module.py:1190, in Module._call_impl(self, *input, **kwargs)
   1186 # If we don't have any hooks, we want to skip the rest of the logic in
   1187 # this function, and just call forward.
   1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190     return forward_call(*input, **kwargs)
   1191 # Do not call functions when jit is used
   1192 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:750, in DisentangledSelfAttention.forward(self, hidden_states, attention_mask, output_attentions, query_states, relative_pos, rel_embeddings)
    748 if self.relative_attention:
    749     rel_embeddings = self.pos_dropout(rel_embeddings)
--> 750     rel_att = self.disentangled_attention_bias(
    751         query_layer, key_layer, relative_pos, rel_embeddings, scale_factor
    752     )
    754 if rel_att is not None:
    755     attention_scores = attention_scores + rel_att

File ~/opt/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:845, in DisentangledSelfAttention.disentangled_attention_bias(self, query_layer, key_layer, relative_pos, rel_embeddings, scale_factor)
    842 else:
    843     r_pos = relative_pos
--> 845 p2c_pos = torch.clamp(-r_pos + att_span, 0, att_span * 2 - 1)
    846 p2c_att = torch.bmm(key_layer, pos_query_layer.transpose(-1, -2))
    847 p2c_att = torch.gather(
    848     p2c_att,
    849     dim=-1,
    850     index=p2c_pos.squeeze(0).expand([query_layer.size(0), key_layer.size(-2), key_layer.size(-2)]),
    851 ).transpose(-1, -2)

TypeError: Operation 'neg_out_mps()' does not support i`

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`training_args = TrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=5,
    weight_decay=0.01,
    #load_best_model_at_end = True,
    #metric_for_best_model = 'eval_f1',
    overwrite_output_dir= True,
    gradient_accumulation_steps = 1,
    gradient_checkpointing = False,
    use_mps_device = True,

)
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_test,
    tokenizer=tokenizer,
    data_collator=data_collator,
    #compute_metrics= compute_metrics1,

)`


### Expected behavior

to be able to train using Apple Silicon Chip

Comments:
As far as I can tell, this an issue in PyTorch not supporting an operation on MPS, so you should probably file your issue there.

I have overcome all the difficulties I have encountered so far with your solution suggestions about transformers. :) 

Has this issue been resolved?

I'm running into the same problem. 

@forrestfaraday, can you share your solution with me?