TF Longformer has some graph compilation/execution issue
Author: jplu
Pull request: no
Labels: name: wontfix

TF longformer has the following issues to make it 100% graph compilation/execution compliant. I succeed to fix most of the issues but two still remains:

1.  The first issue starts at line [1762](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_tf_longformer.py#L1762). The test to know if the inputs needs to be padded prevent the graph to be compiled because `input_ids`, `position_ids` and `input_embeds` can be `None` at the end of the main branch.

As a solution I propose to export the padding process (from line 1769 to 1786) outside the `if` as if `padding_len == 0` the calls to `tf.pad(...)` and `tf.concat(...)` will have no effect on the different inputs.

2. The second issue is at line [1527](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_tf_longformer.py#L1527). Here `all_global_attentions` can be either a tuple or `None` in a same execution because `is_global_attn` is not defined globally but during the execution.

I don't know how to solve this one.

As a first test you can run:
```
from transformers import TFLongformerModel
model = TFLongformerModel.from_pretrained("lysandre/tiny-longformer-random", output_attentions=True, output_hidden_states=True)
model.save("path")
```

Ping @patrickvonplaten the Longformer expert :)

Comments:
Can we define `is_global_attn` in the config?

Or can we assume that `is_global_attn == output_attentions`  ? The main issue here is that we cannot build a returned value that depends of a tensor that is created during the execution (same issue we had with `output_attentions` and `output_hidden_states` before we decide to take the config values in graph mode)

I think Longformer has an inherent design problem with TF serving. The variable `is_global_attn` is decided by the user at execution time and depends on the **values** (not just the shape) of `global_attention_mask`. `is_global_attn` is not a boolean to indicate whether the user wants to output the `attentions`, but whether the model will make use of `global_attention`.

If TF serving only works when `is_global_attn` has to be known before execution time, then I guess the best option is to add a `config.use_global_attn_tf` that would default to `False`. Could we then add an assert statement that `is_global_attn == config.use_global_attn_tf` with a nice error message saying the in TF serving `config.use_global_attn_tf` has to be set according to the use case? 

For some more information on the logic, see https://huggingface.co/transformers/model_doc/longformer.html#longformer-self-attention


Regarding the 1. case: `input_ids`, `position_ids` and `input_embeds` can only be `None` if they have been `None` before entering the function. I don't fully understand your proposed solution, but I think this would be easy to discuss in a PR.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.