getting error with BertForMaskedLM
Author: gems047
Pull request: no

config = BertConfig.from_pretrained("bert-base-uncased")

self.bert = BertModel.from_pretrained("bert-base-uncased")
sequence_output = self.bert(
            inputs_embeds=input_embeddings,
            position_ids=position_ids,
            token_type_ids=token_type_ids,
            attention_mask=attention_mask,
        )[0][:, max_ids:, :]

self.x1 = BertForMaskedLM(config)
x2 = self.x1(sequence_output)

While running above code , getting below error at last line (x2 = self.x1(sequence_output). Unable to relate the error with code sequence , and why is this error coming. Is there any issue with BertForMaskedLM

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "<ipython-input-9-d980e37a9621>", line 90, in forward
    x_scores = self.x_head(sequence_output).to(self.device)
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py", line 1185, in forward
    return_dict=return_dict,
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py", line 862, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py", line 198, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 160, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/gems/pytorch-env_py3.7/lib/python3.7/site-packages/torch/nn/functional.py", line 2043, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)



Comments:
Hello, do you have a reproducible code example that showcases what's in your variables such as `input_embeddings`? Thank you!

Did not mention parameters like input_embeddings, however input looks like below:



tensor([[[-0.0129, -0.0099,  0.2570,  ..., -0.1841,  0.3609,  0.2851],
         [-0.1268,  0.0133,  0.2927,  ..., -0.1167,  0.4605, -0.1288],
         [-0.9993,  0.4705,  0.5119,  ..., -0.9274,  0.5529, -0.5890],
         ...,
         [-0.4415,  0.0786,  0.3132,  ..., -0.2137, -0.0387,  0.2496],
         [ 0.2243,  0.2535,  0.2158,  ..., -0.0974, -0.1830,  0.1292],
         [ 0.0239, -0.2080,  0.4332,  ..., -0.2069,  0.0078,  0.2262]],

        [[-0.2665,  0.1647,  0.4427,  ...,  0.0847,  0.4180,  0.7866],
         [ 0.1999, -0.3408,  0.4952,  ..., -0.3468,  0.4271, -0.5220],
         [-0.6198,  0.1422,  0.5547,  ..., -0.1745, -0.0165, -0.4338],
         ...,
         [-0.0902,  0.1044,  0.2038,  ..., -0.0335,  0.4127,  0.2904],
         [-0.0747,  0.0279,  0.2409,  ..., -0.0989,  0.0915,  0.0109],
         [ 0.2910,  0.1765,  0.3457,  ...,  0.0559,  0.0067, -0.0191]],

        [[-0.0183, -0.0937,  0.6092,  ..., -0.4594,  0.2707,  0.1108],
         [ 0.5192, -0.0532,  0.4865,  ...,  0.1216,  0.0658,  0.5460],
         [-0.0984, -0.1430,  0.3035,  ..., -0.0563,  0.3445, -0.2272],
         ...,
         [ 0.1298, -0.1624,  0.1905,  ...,  0.0979, -0.0197, -0.3143],
         [-0.3790,  0.0682,  0.0601,  ...,  0.0266, -0.1095, -0.2442],
         [-0.0352, -0.0526,  0.1690,  ...,  0.0723,  0.1064, -0.2718]],

        ...,

        [[-0.3095, -0.3042,  0.2681,  ..., -0.1081, -0.0650,  0.3146],
         [ 0.3054,  0.0550,  0.1716,  ..., -0.1492, -0.0201, -0.1543],
         [-0.4458,  0.0661,  0.2862,  ..., -0.2693,  0.3367,  0.0015],
         ...,
         [-0.3311, -0.0961,  0.2018,  ...,  0.0840, -0.1578,  0.3397],
         [-0.0362,  0.0713,  0.4921,  ...,  0.0881,  0.0501, -0.1048],
         [-0.0793, -0.1054,  0.1489,  ..., -0.0762, -0.0039, -0.0471]],

        [[ 0.2076, -0.4345,  0.0533,  ..., -0.0296, -0.1365, -0.1304],
         [ 0.5159,  0.3230,  0.6001,  ..., -0.4266, -0.1751, -0.6830],
         [ 0.2633, -0.0747,  0.6887,  ..., -0.5294,  0.4353, -0.3712],
         ...,
         [ 0.3373,  0.2944,  0.3050,  ..., -0.0972, -0.1798, -0.2998],
         [-0.3282,  0.1189,  0.3962,  ..., -0.2579, -0.2661, -0.0275],
         [-0.0706,  0.0654,  0.6177,  ..., -0.1825,  0.0214, -0.1656]],

        [[ 0.0669, -0.3896, -0.0204,  ..., -0.2962, -0.3721, -0.0138],
         [ 0.4778,  0.1336,  0.5360,  ..., -0.0931, -0.3350, -0.3153],
         [ 0.3600, -0.2580,  0.1261,  ...,  0.0296, -0.0979,  0.1038],
         ...,
         [ 0.0821,  0.0034,  0.2967,  ..., -0.1719, -0.2646, -0.1868],
         [-0.0868,  0.4321,  0.0466,  ...,  0.2056, -0.4406, -0.1953],
         [-0.1131, -0.1266,  0.1438,  ..., -0.3065, -0.2185,  0.1069]]],
       device='cuda:1', grad_fn=<SliceBackward>)

Will check how to put reproducible code example

weirdly it seems that your input embeddings are treated as input IDs, which should not happen. Can you let me know the result of `transforemrs-cli env` in your environment?

I get below  result for transformers-cli env :

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.1.1
- Platform: Linux-4.15.0-107-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


Ah, does you error come from `x2 = self.x1(sequence_output)` ? What are you trying to do here, are you passing the BERT output as input IDs to a BERT Masked LM model?

Changed this line x2 = self.x1(sequence_output) to

x2 = self.x1(inputs_embeds=sequence_output) as suggested above.

It works now , Thankyou

Can I change this code 

```py
self.bert = BertModel.from_pretrained("bert-base-uncased")
sequence_output = self.bert(
    inputs_embeds=input_embeddings,
    position_ids=position_ids,
    token_type_ids=token_type_ids,
    attention_mask=attention_mask,
)[0][:, max_ids:, :]
```

as below:

```py
self.bert = BertModel.from_pretrained("bert-base-uncased")
sequence_output = self.bert(
    inputs_embeds= position_embeddings,
    #position_ids=position_ids,
    token_type_ids=token_type_ids,
    attention_mask=attention_mask,
)[0][:, max_ids:, :]
```

ie assign position_embeddings to inputs_embeds
or like this `input_embeds = input_embeddings + position_embeddings`

No, the inputs embedding are only for the input IDs. Position embeddings will be added to that variable in the embedding layer.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.