RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
Author: danyaljj
Pull request: no

Here is the complete error message: 

```
Traceback (most recent call last):
  File "app/set_expantion_eval.py", line 118, in <module>
    map_n=flags.map_n)
  File "app/set_expantion_eval.py", line 62, in Eval
    expansionWithScores = BE.set_expansion_tensorized(seeds, ["1"])
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/app/bert_expansion.py", line 109, in set_expansion_tensorized
    gold_repr_list.append(self.extract_representation(" ".join(seed), x, dim))
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/app/bert_expansion.py", line 317, in extract_representation
    output_all_encoded_layers=output_all_encoded_layers)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 626, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 193, in forward
    words_embeddings = self.word_embeddings(input_ids)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
```
Here is a summary of what I do in my code: 
```python 
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
self.model = BertModel.from_pretrained(bert_model) # loading the model  
self.model.to(self.device) # without this there is no error, but it runs in CPU (instead of GPU). 
self.model.eval() # declaring to the system that we're only doing 'forward' calculations 

# creating the input tensors here 
...
# move the tensors to the target device 
input_ids_tensor.to(self.device)
segment_ids_tensor.to(self.device)
input_mask_tensor.to(self.device)
output_all_encoded_layers.to(self.device)
encoded_layers, _ = self.model(input_ids_tensor, segment_ids_tensor, input_mask_tensor, output_all_encoded_layers=output_all_encoded_layers)
```

When I don't have `model.to(device) ` the code works fine, but I think it only uses CPU only. When I add it, it fails with the above error. 

I did a little investigation and printed the inputs to `.model(.)` to see if they are properly copied to `device`: 
```
print("\n * input_ids_tensor \n ")
print(input_ids_tensor)
print(input_ids_tensor.device)

print("\n * segment_ids_tensor \n ")
print(segment_ids_tensor)
print(segment_ids_tensor.device)

print("\n * input_mask_tensor \n ")
print(input_mask_tensor)
print(input_mask_tensor.device)

print("\n * self.device \n ")
print(self.device)
```
which outputs: 
```
 * input_ids_tensor  
tensor([[ 101, 5334, 2148, 1035, 3792, 3146,  102, 5334,  102,    0,    0],
        [ 101, 5334, 2148, 1035, 3792, 3146,  102, 2148, 1035, 3792,  102],
        [ 101, 5334, 2148, 1035, 3792, 3146,  102, 3146,  102,    0,    0]])
cpu

 * segment_ids_tensor  
tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
        [0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0]])
cpu

 * input_mask_tensor  
tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])
cpu

 * self.device 
cuda:0
```

As it can be seen, the tensors are still `cpu`, even after running `.to(device)`. 

Any thoughts where things are going wrong? 


Comments:
You failed to move the tensors to GPU. 
Replace your code with this:
```
input_ids_tensor = input_ids_tensor.to(self.device)
segment_ids_tensor = segment_ids_tensor.to(self.device)
input_mask_tensor = input_mask_tensor.to(self.device)
```

Ah I didn't realize they don't work in-place (unlike the syntax for model files `model.to(device)`). 

thank you!

Great thanks!

thanks！

use model.to(device) as well