[PyTorch] Load and run a model CPU which was traced and saved on GPU
Author: vdantu
Pull request: no

# ❓ Questions & Help

## Details
I am trying to trace/save openai_gpt on GPU and use that model on a CPU and facing issues. Is this possible to do? I have attached the link to the question posted on the discuss forum as well.

### Sample Script
```python
from transformers import OpenAIGPTTokenizer, OpenAIGPTModel
import torch

tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
model = OpenAIGPTModel.from_pretrained('openai-gpt')

inputs = torch.tensor([tokenizer.encode("Hello, my dog is cute")])
outputs = model(inputs)
print(outputs)

print("To CUDA:")
inputs = inputs.to("cuda")
model = model.to("cuda")
traced_model = torch.jit.trace(model, (inputs,))
torch.jit.save(traced_model, "openai_gpt_cuda.pt")
print(traced_model.graph)
print("\n")
print("Load model onto CPU")
loaded = torch.jit.load("openai_gpt_cuda.pt", map_location=torch.device("cpu"))
inputs = inputs.to("cpu")
print("\n")
print(loaded.graph)
outputs = loaded(inputs)
print(outputs)
```

### Error seen
```
Traceback (most recent call last):
  File "gpt.py", line 23, in <module>
    outputs = loaded(inputs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select
The above operation failed in interpreter.
Traceback (most recent call last):
Serialized   File "code/__torch__/torch/nn/modules/module/___torch_mangle_147.py", line 35
    position_ids = torch.arange(_20, dtype=4, layout=0, device=torch.device("cuda:0"), pin_memory=False)
    input0 = torch.view(torch.unsqueeze(position_ids, 0), [-1, _19])
    _21 = torch.add((_14).forward(input, ), (_13).forward(input0, ), alpha=1)
                                             ~~~~~~~~~~~~ <--- HERE
    input1 = torch.add(_21, CONSTANTS.c0, alpha=1)
    _22 = (_12).forward(input1, )
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/functional.py(1484): embedding
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/sparse.py(114): forward
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(516): _slow_forward
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(530): __call__
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_openai.py(433): forward
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(516): _slow_forward
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(530): __call__
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py(1034): trace_module
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py(882): trace
gpt.py(14): <module>
Serialized   File "code/__torch__/torch/nn/modules/module/___torch_mangle_0.py", line 8, in forward
  def forward(self: __torch__.torch.nn.modules.module.___torch_mangle_0.Module,
    input: Tensor) -> Tensor:
    position_embeds = torch.embedding(self.weight, input, -1, False, False)
                      ~~~~~~~~~~~~~~~ <--- HERE
    return position_embeds

The above operation failed in interpreter.
Traceback (most recent call last):
```
https://discuss.huggingface.co/t/pytorch-trace-on-cpu-and-use-on-gpu/181/3

Comments:
@vdantu Thanks for reporting the issue. 

The problem arises in `modeling_openai.py`when the user do not provide the `position_ids` function argument thus leading to the inner `position_ids` being created during the forward call. This is fine in classic PyTorch because `forward` is actually evaluated at each call. When it comes to tracing, this is an issue, because the device specified in the forward to actually create the tensor will be hardcoded and you can actually see it in the generated graph: 

```python
  %input.1 : Tensor = aten::view(%input_ids.1, %64)
  %140 : Device = prim::Constant[value="cuda:0"]()
  %position_ids.1 : Tensor = aten::arange(%59, %67, %45, %140, %70)
  %73 : Tensor = aten::unsqueeze(%position_ids.1, %45)
```

Above you can see `%140` is a constant which value is actually set to `"cuda:0"` and then, it is reused to create the `%position_ids.1` tensor through `aten::arange(..., %140, ...)` which of course leads to the error you're seeing.

I'll have a fix to generate the `position_ids` buffer correctly registered at the Module initialisation and not during forward, so it should be correctly handled by the `map_location` parameter while exporting.

The above PR should fix the issue, below is the output of the code you provided. If you want to give it a try, let us know if it works on your end too 👍 

```python
(pytorch) mfuntowicz@brutasse:~/Workspace/transformers$ python test.py 
ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.
Some weights of OpenAIGPTModel were not initialized from the model checkpoint at openai-gpt and are newly initialized: ['position_ids']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
cpu cpu
(tensor([[[ 7.3001e-02, -1.2431e+00,  7.9122e-01,  ...,  1.6806e+00,
          -4.3945e-01,  1.1449e+00],
         [-3.6239e-01, -8.3647e-01,  1.2019e+00,  ...,  1.5575e+00,
          -8.4237e-04,  1.0779e+00],
         [-1.0138e+00, -7.1014e-01,  6.3509e-01,  ...,  1.6684e+00,
          -4.6458e-01,  1.5093e+00],
         [-6.1989e-01, -2.9500e-01,  9.9504e-01,  ...,  2.0421e+00,
           4.2680e-01,  2.1920e+00],
         [-5.2932e-01, -1.7606e-02,  7.4836e-01,  ...,  2.2980e+00,
           3.4807e-01,  2.7045e+00],
         [-1.4679e-01, -9.8566e-02,  1.3909e+00,  ...,  1.9108e+00,
           6.0797e-01,  2.1617e+00]]], grad_fn=<ViewBackward>),)
To CUDA:
/home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:176: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w = w / math.sqrt(v.size(-1))
/home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:179: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  b = self.bias[:, :, : w.size(-2), : w.size(-1)]
graph(%self.1 : __torch__.transformers.modeling_openai.OpenAIGPTModel,
      %input_ids : Long(1:6, 6:1)):
  %4489 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4490 : __torch__.transformers.modeling_openai.___torch_mangle_139.Block = prim::GetAttr[name="11"](%4489)
  %4463 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4464 : __torch__.transformers.modeling_openai.___torch_mangle_127.Block = prim::GetAttr[name="10"](%4463)
  %4437 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4438 : __torch__.transformers.modeling_openai.___torch_mangle_115.Block = prim::GetAttr[name="9"](%4437)
  %4411 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4412 : __torch__.transformers.modeling_openai.___torch_mangle_103.Block = prim::GetAttr[name="8"](%4411)
  %4385 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4386 : __torch__.transformers.modeling_openai.___torch_mangle_91.Block = prim::GetAttr[name="7"](%4385)
  %4359 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4360 : __torch__.transformers.modeling_openai.___torch_mangle_79.Block = prim::GetAttr[name="6"](%4359)
  %4333 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4334 : __torch__.transformers.modeling_openai.___torch_mangle_67.Block = prim::GetAttr[name="5"](%4333)
  %4307 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4308 : __torch__.transformers.modeling_openai.___torch_mangle_55.Block = prim::GetAttr[name="4"](%4307)
  %4281 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4282 : __torch__.transformers.modeling_openai.___torch_mangle_43.Block = prim::GetAttr[name="3"](%4281)
  %4255 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4256 : __torch__.transformers.modeling_openai.___torch_mangle_31.Block = prim::GetAttr[name="2"](%4255)
  %4229 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4230 : __torch__.transformers.modeling_openai.___torch_mangle_19.Block = prim::GetAttr[name="1"](%4229)
  %4203 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4204 : __torch__.transformers.modeling_openai.Block = prim::GetAttr[name="0"](%4203)
  %4178 : __torch__.torch.nn.modules.dropout.Dropout = prim::GetAttr[name="drop"](%self.1)
  %4177 : __torch__.torch.nn.modules.sparse.___torch_mangle_0.Embedding = prim::GetAttr[name="positions_embed"](%self.1)
  %4175 : __torch__.torch.nn.modules.sparse.Embedding = prim::GetAttr[name="tokens_embed"](%self.1)
  %4173 : Tensor = prim::GetAttr[name="position_ids"](%self.1)
  %458 : int = prim::Constant[value=0]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:438:0
  %459 : int = aten::size(%input_ids, %458) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:438:0
  %460 : Long() = prim::NumToTensor(%459)
  %4020 : int = aten::Int(%460)
  %461 : int = prim::Constant[value=1]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:438:0
  %462 : int = aten::size(%input_ids, %461) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:438:0
  %463 : Long() = prim::NumToTensor(%462)
  %4021 : int = aten::Int(%463)
  %470 : int = aten::Int(%463)
  %464 : int = aten::Int(%463)
  %465 : int = prim::Constant[value=-1]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:439:0
  %466 : int[] = prim::ListConstruct(%465, %464)
  %input.1 : Long(1:6, 6:1) = aten::view(%input_ids, %466) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:439:0
  %468 : int = prim::Constant[value=0]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:447:0
  %469 : Long(1:512, 512:1) = aten::unsqueeze(%4173, %468) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:447:0
  %471 : int = prim::Constant[value=1]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:447:0
  %input.2 : Long(1:512) = aten::select(%469, %471, %470) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:447:0
  %4638 : Tensor = prim::CallMethod[name="forward"](%4175, %input.1)
  %4639 : Tensor = prim::CallMethod[name="forward"](%4177, %input.2)
  %481 : int = prim::Constant[value=1]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:479:0
  %482 : Float(1:4608, 6:768, 768:1) = aten::add(%4638, %4639, %481) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:479:0
  %483 : Long() = prim::Constant[value={0}]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:479:0
  %484 : int = prim::Constant[value=1]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:479:0
  %input.3 : Float(1:4608, 6:768, 768:1) = aten::add(%482, %483, %484) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:479:0
  %4640 : Tensor = prim::CallMethod[name="forward"](%4178, %input.3)
  %489 : int = prim::Constant[value=-1]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:482:0
  %490 : int = aten::size(%4640, %489) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:482:0
  %491 : Long() = prim::NumToTensor(%490)
  %4022 : int = aten::Int(%491)
  %4641 : Tensor = prim::CallMethod[name="forward"](%4204, %4640)
  %4642 : Tensor = prim::CallMethod[name="forward"](%4230, %4641)
  %4643 : Tensor = prim::CallMethod[name="forward"](%4256, %4642)
  %4644 : Tensor = prim::CallMethod[name="forward"](%4282, %4643)
  %4645 : Tensor = prim::CallMethod[name="forward"](%4308, %4644)
  %4646 : Tensor = prim::CallMethod[name="forward"](%4334, %4645)
  %4647 : Tensor = prim::CallMethod[name="forward"](%4360, %4646)
  %4648 : Tensor = prim::CallMethod[name="forward"](%4386, %4647)
  %4649 : Tensor = prim::CallMethod[name="forward"](%4412, %4648)
  %4650 : Tensor = prim::CallMethod[name="forward"](%4438, %4649)
  %4651 : Tensor = prim::CallMethod[name="forward"](%4464, %4650)
  %4652 : Tensor = prim::CallMethod[name="forward"](%4490, %4651)
  %4023 : int[] = prim::ListConstruct(%4020, %4021, %4022)
  %4024 : Float(1:4608, 6:768, 768:1) = aten::view(%4652, %4023) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:495:0
  %4025 : (Float(1:4608, 6:768, 768:1)) = prim::TupleConstruct(%4024)
  return (%4025)



Load model onto CPU


graph(%self.1 : __torch__.transformers.modeling_openai.OpenAIGPTModel,
      %input_ids.1 : Tensor):
  %78 : Tensor = prim::Constant[value={0}]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:479:0
  %47 : int = prim::Constant[value=0]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:438:0
  %53 : int = prim::Constant[value=1]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:438:0
  %61 : int = prim::Constant[value=-1]() # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:439:0
  %3 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %4 : __torch__.transformers.modeling_openai.___torch_mangle_139.Block = prim::GetAttr[name="11"](%3)
  %6 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %7 : __torch__.transformers.modeling_openai.___torch_mangle_127.Block = prim::GetAttr[name="10"](%6)
  %9 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %10 : __torch__.transformers.modeling_openai.___torch_mangle_115.Block = prim::GetAttr[name="9"](%9)
  %12 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %13 : __torch__.transformers.modeling_openai.___torch_mangle_103.Block = prim::GetAttr[name="8"](%12)
  %15 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %16 : __torch__.transformers.modeling_openai.___torch_mangle_91.Block = prim::GetAttr[name="7"](%15)
  %18 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %19 : __torch__.transformers.modeling_openai.___torch_mangle_79.Block = prim::GetAttr[name="6"](%18)
  %21 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %22 : __torch__.transformers.modeling_openai.___torch_mangle_67.Block = prim::GetAttr[name="5"](%21)
  %24 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %25 : __torch__.transformers.modeling_openai.___torch_mangle_55.Block = prim::GetAttr[name="4"](%24)
  %27 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %28 : __torch__.transformers.modeling_openai.___torch_mangle_43.Block = prim::GetAttr[name="3"](%27)
  %30 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %31 : __torch__.transformers.modeling_openai.___torch_mangle_31.Block = prim::GetAttr[name="2"](%30)
  %33 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %34 : __torch__.transformers.modeling_openai.___torch_mangle_19.Block = prim::GetAttr[name="1"](%33)
  %36 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="h"](%self.1)
  %37 : __torch__.transformers.modeling_openai.Block = prim::GetAttr[name="0"](%36)
  %39 : __torch__.torch.nn.modules.dropout.Dropout = prim::GetAttr[name="drop"](%self.1)
  %41 : __torch__.torch.nn.modules.sparse.___torch_mangle_0.Embedding = prim::GetAttr[name="positions_embed"](%self.1)
  %43 : __torch__.torch.nn.modules.sparse.Embedding = prim::GetAttr[name="tokens_embed"](%self.1)
  %45 : Tensor = prim::GetAttr[name="position_ids"](%self.1)
  %48 : int = aten::size(%input_ids.1, %47) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:438:0
  %49 : Tensor = prim::NumToTensor(%48) # :0:0
  %51 : int = aten::Int(%49)
  %54 : int = aten::size(%input_ids.1, %53) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:438:0
  %55 : Tensor = prim::NumToTensor(%54) # :0:0
  %57 : int = aten::Int(%55)
  %59 : int = aten::Int(%55)
  %63 : int = aten::Int(%55)
  %64 : int[] = prim::ListConstruct(%61, %63)
  %input.1 : Tensor = aten::view(%input_ids.1, %64) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:439:0
  %67 : Tensor = aten::unsqueeze(%45, %47) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:447:0
  %input0.1 : Tensor = aten::select(%67, %53, %59) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:447:0
  %72 : Tensor = prim::CallMethod[name="forward"](%43, %input.1) # :0:0
  %75 : Tensor = prim::CallMethod[name="forward"](%41, %input0.1) # :0:0
  %76 : Tensor = aten::add(%72, %75, %53) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:479:0
  %input1.1 : Tensor = aten::add(%76, %78, %53) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:479:0
  %82 : Tensor = prim::CallMethod[name="forward"](%39, %input1.1) # :0:0
  %84 : int = aten::size(%82, %61) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:482:0
  %85 : Tensor = prim::NumToTensor(%84) # :0:0
  %87 : int = aten::Int(%85)
  %91 : Tensor = prim::CallMethod[name="forward"](%37, %82) # :0:0
  %92 : Tensor = prim::CallMethod[name="forward"](%34, %91) # :0:0
  %97 : Tensor = prim::CallMethod[name="forward"](%31, %92) # :0:0
  %98 : Tensor = prim::CallMethod[name="forward"](%28, %97) # :0:0
  %99 : Tensor = prim::CallMethod[name="forward"](%25, %98) # :0:0
  %104 : Tensor = prim::CallMethod[name="forward"](%22, %99) # :0:0
  %105 : Tensor = prim::CallMethod[name="forward"](%19, %104) # :0:0
  %106 : Tensor = prim::CallMethod[name="forward"](%16, %105) # :0:0
  %111 : Tensor = prim::CallMethod[name="forward"](%13, %106) # :0:0
  %112 : Tensor = prim::CallMethod[name="forward"](%10, %111) # :0:0
  %113 : Tensor = prim::CallMethod[name="forward"](%7, %112) # :0:0
  %116 : Tensor = prim::CallMethod[name="forward"](%4, %113) # :0:0
  %120 : int[] = prim::ListConstruct(%51, %57, %87)
  %121 : Tensor = aten::view(%116, %120) # /home/mfuntowicz/Workspace/transformers/src/transformers/modeling_openai.py:495:0
  %123 : (Tensor) = prim::TupleConstruct(%121)
  return (%123)

(tensor([[[ 7.3003e-02, -1.2431e+00,  7.9122e-01,  ...,  1.6806e+00,
          -4.3945e-01,  1.1449e+00],
         [-3.6239e-01, -8.3647e-01,  1.2019e+00,  ...,  1.5575e+00,
          -8.4937e-04,  1.0779e+00],
         [-1.0138e+00, -7.1013e-01,  6.3510e-01,  ...,  1.6684e+00,
          -4.6459e-01,  1.5093e+00],
         [-6.1989e-01, -2.9499e-01,  9.9504e-01,  ...,  2.0421e+00,
           4.2680e-01,  2.1920e+00],
         [-5.2932e-01, -1.7599e-02,  7.4836e-01,  ...,  2.2980e+00,
           3.4806e-01,  2.7045e+00],
         [-1.4679e-01, -9.8562e-02,  1.3909e+00,  ...,  1.9108e+00,
           6.0796e-01,  2.1617e+00]]], grad_fn=<ViewBackward>),)
```

That's very interesting @mfuntowicz ! I think we will probably have multiple of such failures - I would guess for all models that use `position_ids`. Also as a rule, should one never create a tensor on the fly, but always register a buffer for that? @mfuntowicz @sshleifer 

Awesome, thanks for fixing this. I will test this fix. What release of transformers will this change be reflected in? I was testing with transformers 3.0.2. 

@patrickvonplaten : I think you are right. I remember seeing this with bert base uncased as well. It would definitely be useful to have this fix across all models. 

Looking for easier solutions than changing all the code:

1) Can we just trace the thing correctly by passing `traced_model = torch.jit.trace(model, (inputs,position_ids))`?  and then document the correct way to trace (maybe we can add `jit_inputs` or reuse `dummy_inputs`?)

2) how much faster is the model afterwards?

3) We should add a save/load test to test_modeling_common.py if we want to support. The current test_torch_script just traces and then runs forward, and can clearly pass with many unregistered buffers.

@sshleifer : Thanks for the response. In the example script I pasted above, do you see any errors in the way I am tracing and using the traced model? Please let me know if that needs to be changed.



@vdantu  
I don't know a ton about jit, but you could try:
```python
traced_model = torch.jit.trace(model, (inputs,position_ids))
```
and see if that fixes the error.

@mfuntowicz : Which version of transformers (pypi package) will these changes be available with? I am used to testing the models throug pypi package or through torch.hub. What is the recommended way to get and test these fixes?

We'll release a new version in the coming weeks, in the meantime you can install from source: `pip install git+https://github.com/huggingface/transformers`

PyTorch doesn't currently support tracing passed devices correctly: https://github.com/pytorch/pytorch/issues/31141#issuecomment-675506630

I stumbled on two problematic lines while tracing GPT-2:
1. https://github.com/huggingface/transformers/blob/1d6e71e1167dea9e026391ec5a1a2d7ec33d22af/src/transformers/modeling_gpt2.py#L554
2. https://github.com/huggingface/transformers/blob/1d6e71e1167dea9e026391ec5a1a2d7ec33d22af/src/transformers/modeling_gpt2.py#L582

Because of this, tracing the model on one device and then using it on another device doesn't work

I'm also seeing this issue when trying to trace DistilBert - https://github.com/huggingface/transformers/blob/eb3bd73ce35bfef56eeb722d697f2d39a06a8f8d/src/transformers/modeling_distilbert.py#L105

Looks to be the same issue.

Observing the same issue while trying to load traced DistilBERT on cpu.

@eugeneware @kavinsabharwal  I managed to get DistilBert work through making similar changes with the PR listed in this issue. However, there is a torch script warning saying seq_length set as constant after doing the trace.

Add this in the constructor of embedding
```
        # position_ids (1, len position emb) is contiguous in memory and exported when serialized
        self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)))
```

and change `position_ids` as following:

```
position_ids = self.position_ids[:, :seq_length]
```

To save the sake of accuracy, I finally decided to add `position_ids` as one of the inputs that passed to the model. And everything seemed working now. Just a workkaround to this problem.

This change is verified working on the 
```
model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', return_dict=False, torchscript=True)
```

@vdantu I ran some training test on the model and it seemed performing fine. The way making position_ids as input should be a safe bet to get away from the warning.

This issue has been stale for 1 month.