Can't export TransfoXLModel model
Author: virajkarandikar
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->
I am trying to export TransfoXLModel and use it for inference from C++ API.
I tried torch.jit.trace(), torch.jit.script() and torch.onnx.export(). But none of these work.

Model I am using - TransfoXLModel:

Language I am using the model on - English

The problem arise when using:
```
model = TransfoXLModel.from_pretrained("transfo-xl-wt103", torchscript=True)
model.eval()
tokenizer = TransfoXLTokenizer.from_pretrained("transfo-xl-wt103", torchscript=True)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
torch.jit.script(model, (input_ids))
```

The tasks I am working on is:
Running inference using C++ API

## To Reproduce
Executing above python code throws error.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py:1200: UserWarning: `optimize` is deprecated and has no effect. Use `with torch.jit.optimized_execution() instead
  warnings.warn("`optimize` is deprecated and has no effect. Use `with torch.jit.optimized_execution() instead")
Traceback (most recent call last):
  File "test_bert_jit.py", line 28, in <module>
    torch.jit.script(model, (input_ids))
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1203, in script
    return torch.jit.torch.jit._recursive.recursive_script(obj)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/_recursive.py", line 172, in recursive_script
    stubs = list(map(make_stub, filtered_methods))
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/_recursive.py", line 169, in make_stub
    return torch.jit.script_method(func, _jit_internal.createResolutionCallbackFromClosure(func))
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1280, in script_method
    ast = get_jit_def(fn, self_name="ScriptModule")
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 169, in get_jit_def
    return build_def(ctx, py_ast.body[0], type_line, self_name)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 209, in build_def
    build_stmts(ctx, body))
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 127, in build_stmts
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 127, in <listcomp>
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 185, in __call__
    return method(ctx, node)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 352, in build_If
    build_stmts(ctx, stmt.body),
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 127, in build_stmts
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 127, in <listcomp>
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 185, in __call__
    return method(ctx, node)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 283, in build_Assign
    rhs = build_expr(ctx, stmt.value)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 185, in __call__
    return method(ctx, node)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 442, in build_Call
    args = [build_expr(ctx, py_arg) for py_arg in expr.args]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 442, in <listcomp>
    args = [build_expr(ctx, py_arg) for py_arg in expr.args]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 184, in __call__
    raise UnsupportedNodeError(ctx, node)
torch.jit.frontend.UnsupportedNodeError: GeneratorExp aren't supported:
at /home/user/transformers/transformers/modeling_transfo_xl.py:767:24
        core_out = self.drop(core_out)

        new_mems = self._update_mems(hids, mems, mlen, qlen)

        # We transpose back here to shape [bsz, len, hidden_dim]
        outputs = [core_out.transpose(0, 1).contiguous(), new_mems]
        if self.output_hidden_states:
            # Add last layer and transpose to library standard shape [bsz, len, hidden_dim]
            hids.append(core_out)
            hids = list(t.transpose(0, 1).contiguous() for t in hids)
                        ~ <--- HERE
            outputs.append(hids)
        if self.output_attentions:
            # Transpose to library standard shape [bsz, n_heads, query_seq_len, key_seq_len]
            attentions = list(t.permute(2, 3, 0, 1).contiguous() for t in attentions)
            outputs.append(attentions)

        return outputs  # last hidden state, new_mems, (all hidden states), (all attentions)
```

## Expected behavior

torch.jit.script() succeeds without any error

## Environment

* OS: Ubunut 18.04
* Python version: 3.7.4
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): master @ ae1d03fc51bb22ed59517ee6f92c560417fdb049
* Using GPU ? Yes
* Distributed of parallel setup ? No.
* Any other relevant information:
Using torch.onnx.export() throws below error:
```
/home/user/transformers/transformers/modeling_transfo_xl.py:452: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator mul_. This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  embed.mul_(self.emb_scale)
/home/user/transformers/transformers/modeling_transfo_xl.py:725: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if mask_len > 0:
/home/user/transformers/transformers/modeling_transfo_xl.py:729: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  dec_attn_mask = (torch.triu(all_ones, 1+mlen)
/home/user/transformers/transformers/modeling_transfo_xl.py:730: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  + torch.tril(all_ones, -mask_shift_len))[:, :, None] # -1
/home/user/transformers/transformers/modeling_transfo_xl.py:290: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w_head_q = w_head_q[-qlen:]
/home/user/transformers/transformers/modeling_transfo_xl.py:321: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_mask is not None and torch.sum(attn_mask).item():
/home/user/transformers/transformers/modeling_transfo_xl.py:684: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  end_idx = mlen + max(0, qlen - 0 - self.ext_len)
/home/user/transformers/transformers/modeling_transfo_xl.py:685: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  beg_idx = max(0, end_idx - self.mem_len)
/home/user/transformers/transformers/modeling_transfo_xl.py:689: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  new_mems.append(cat[beg_idx:end_idx].detach())
/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py:617: UserWarning: ONNX export failed on ATen operator triu because torch.onnx.symbolic_opset10.triu does not exist
  .format(op_name, opset_version, op_name))
Traceback (most recent call last):
  File "test_bert_jit.py", line 37, in <module>
    output_names = ['output']) # the model's output names
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/__init__.py", line 143, in export
    strip_doc_string, dynamic_axes, keep_initializers_as_inputs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 66, in export
    dynamic_axes=dynamic_axes, keep_initializers_as_inputs=keep_initializers_as_inputs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 382, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 262, in _model_to_graph
    fixed_batch_size=fixed_batch_size)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 132, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/__init__.py", line 174, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 618, in _run_symbolic_function
    op_fn = sym_registry.get_registered_op(op_name, '', opset_version)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/symbolic_registry.py", line 91, in get_registered_op
    return _registry[(domain, version)][opname]
KeyError: 'triu'
```

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Yes, this is a known issue, `TransformerXL` is not traceable.

Fixing this is not on our short-term roadmap (cc @LysandreJik) but feel free to investigate and propose a solution in a PR if you want.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any update? Does it work successfully?

Same problem here.

`hids = [t.transpose(0, 1).contiguous() for t in hids]` maybe work.