Can't trace any model with pytorch-transformers 1.2
Author: Bycob
Pull request: no
Labels: name: wontfix

## 🐛 Bug

I get an error whenever I try to trace any model from pytorch-transformers 1.2.0.
When I roll back to 1.1 everything is fine.

## To Reproduce

```python
from pytorch_transformers import BertModel
import torch

model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)
model.to('cuda')
model.eval()
ids = torch.LongTensor([[1, 2, 3]]).cuda()
tok = torch.zeros_like(ids)
att = torch.ones_like(ids)
torch.jit.trace(model, (ids, tok, att, ids))
```

This script produces the following error:

```
Traceback (most recent call last):
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 545, in run_mod_and_filter_tensor_outputs
    outs = wrap_retval(mod(*_clone_inputs(inputs)))
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
RuntimeError: r ASSERT FAILED at /pytorch/aten/src/ATen/core/jit_type.h:142, please report a bug to PyTorch. (expect at /pytorch/aten/src/ATen/core/jit_type.h:142)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f273aa91441 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f273aa90d7a in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
frame #2: std::shared_ptr<c10::DimensionedTensorType const> c10::Type::expect<c10::DimensionedTensorType const>() + 0x140 (0x7f27397ff810 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #3: torch::jit::fuser::compileKernel(torch::jit::fuser::KernelSpec const&, torch::jit::fuser::ArgSpec const&, std::vector<long, std::allocator<long> > const&, c10::Device) + 0xa5a (0x7f27397fbdca in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #4: torch::jit::fuser::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&, std::string*) + 0x5b0 (0x7f2739803c20 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #5: torch::jit::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x13 (0x7f2739733bc3 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #6: <unknown function> + 0xb2b066 (0x7f273973d066 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #7: <unknown function> + 0xa8ebe6 (0x7f27396a0be6 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #8: torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x22 (0x7f273969c202 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #9: <unknown function> + 0xa7685d (0x7f273968885d in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #10: <unknown function> + 0x457617 (0x7f277a2cd617 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x130d0c (0x7f2779fa6d0c in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #14: python3() [0x4fbfce]
frame #16: python3() [0x574db6]
frame #20: python3() [0x4ec2e3]
frame #22: python3() [0x4fbfce]
frame #24: python3() [0x574db6]
frame #27: python3() [0x5401ef]
frame #30: python3() [0x4ec3f7]
frame #33: python3() [0x5401ef]
frame #35: python3() [0x53fc97]
frame #37: python3() [0x53fc97]
frame #39: python3() [0x60cb42]
frame #44: __libc_start_main + 0xf0 (0x7f279df7d830 in /lib/x86_64-linux-gnu/libc.so.6)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "session.py", line 10, in <module>
    torch.jit.trace(model, (ids, tok, att, ids))
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 702, in trace
    _check_trace([example_inputs], func, executor_options, traced, check_tolerance, _force_outplace)
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/autograd/grad_mode.py", line 43, in decorate_no_grad
    return func(*args, **kwargs)
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 583, in _check_trace
    traced_outs = run_mod_and_filter_tensor_outputs(module, inputs, 'trace')
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 551, in run_mod_and_filter_tensor_outputs
    ' with test inputs.\nException:\n' + indent(str(e)))
torch.jit.TracingCheckError: Tracing failed sanity checks!
Encountered an exception while running the trace with test inputs.
Exception:
	r ASSERT FAILED at /pytorch/aten/src/ATen/core/jit_type.h:142, please report a bug to PyTorch. (expect at /pytorch/aten/src/ATen/core/jit_type.h:142)
	frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f273aa91441 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
	frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f273aa90d7a in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
	frame #2: std::shared_ptr<c10::DimensionedTensorType const> c10::Type::expect<c10::DimensionedTensorType const>() + 0x140 (0x7f27397ff810 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #3: torch::jit::fuser::compileKernel(torch::jit::fuser::KernelSpec const&, torch::jit::fuser::ArgSpec const&, std::vector<long, std::allocator<long> > const&, c10::Device) + 0xa5a (0x7f27397fbdca in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #4: torch::jit::fuser::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&, std::string*) + 0x5b0 (0x7f2739803c20 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #5: torch::jit::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x13 (0x7f2739733bc3 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #6: <unknown function> + 0xb2b066 (0x7f273973d066 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #7: <unknown function> + 0xa8ebe6 (0x7f27396a0be6 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #8: torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x22 (0x7f273969c202 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #9: <unknown function> + 0xa7685d (0x7f273968885d in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #10: <unknown function> + 0x457617 (0x7f277a2cd617 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
	frame #11: <unknown function> + 0x130d0c (0x7f2779fa6d0c in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
	<omitting python frames>
	frame #14: python3() [0x4fbfce]
	frame #16: python3() [0x574db6]
	frame #20: python3() [0x4ec2e3]
	frame #22: python3() [0x4fbfce]
	frame #24: python3() [0x574db6]
	frame #27: python3() [0x5401ef]
	frame #30: python3() [0x4ec3f7]
	frame #33: python3() [0x5401ef]
	frame #35: python3() [0x53fc97]
	frame #37: python3() [0x53fc97]
	frame #39: python3() [0x60cb42]
	frame #44: __libc_start_main + 0xf0 (0x7f279df7d830 in /lib/x86_64-linux-gnu/libc.so.6)
```

## Environment

* OS: Ubuntu 16.04.6 LTS
* Python version: Python 3.5.2
* PyTorch version:  '1.1.0'
* PyTorch Transformers version (or branch): '1.2.0'
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information:
I installed pytorch-transformers with pip:
```
pip3 install --user pytorch-transformers==1.2
```

Comments:
Hi, thank you for reporting this. I can reproduce it on my side. It seems to be a problem relative to the model being on `cuda`, as it doesn't fail if you don't put the model/ids on `cuda`.

This doesn't fail:

```py
from pytorch_transformers import BertModel
import torch

model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)
model.eval()
ids = torch.LongTensor([[1, 2, 3]])
tok = torch.zeros_like(ids)
att = torch.ones_like(ids)
torch.jit.trace(model, (ids, tok, att, ids))
```

Traced models on cpu may not be convertible to cuda due to hard coded tensor creation in torchscript. I tried a while ago and it wasn't working, and I found an issue (#1010) referencing a similar problem.

Yes, with the merge of #1195, the jit tracing issue of #1010 should now be fixed on master.

You test installing from source and see if it solves your issue. 

I ran BERT on GPU after tracing it on CPU, and it works fine! Any information about when this fix will be available from the official distribution?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, while the model outputs has multi outputs like ``sequence_output`` and ``pooled_output``，how to get one of them in C++?