New TF model design
Author: jplu
Pull request: yes

# What does this PR do?

This PR aims to improve the TensorFlow code base of transformers. For sake of simplicity the changes are made only on the BERT model, but the new features can be applied to all the others. The PR brings among some bug fix the following main new features;

1. It is now possible to train an LM model from scratch (see this [notebook ](https://colab.research.google.com/drive/1As9iz2_2eQp1Ct8trxRG3ScNfxeaVAH2?usp=sharing)as example).
2. The default outputs of the models are now dictionaries. One can only return tuples in eager execution otherwise a warning message will be displayed saying that only dictionaries are allowed in graph mode. This update fix two issues, the first one in graph mode where a layer cannot return different size of outputs, and a second one where an output cannot have a `None` value.
3. Better inputs handle. Now the inputs of each model and the main layer are parsed with a single generic function bringing a more robust parsing and a better error handling in case of wrong input. This feature fix an issue when the input was a list of symbolic inputs (i.e. `tf.keras.layers.Input`).
4. TensorFlow models looks now much more similar to what looks PyTorch models making easier for users to switch from a PyTorch model to its TensorFlow implementation and vice versa.
5. Old and new model implementations can coexist in the library making this new implementation 100% backward compatible. Including the tests.

Comments:
Thanks for your comments! I will try to answer as clearly as possible.

### From embedding building to `tf.keras.layers.Embedding`
> Would we now be relying on the LM head weights being in the checkpoint?

Exactly, also another difference is to have the `word_embeddings` weights initialized at the model instanciation instead of model building (after a first call). The weights are anyway saved in the checkpoints and shared with the layers that needs to have access to it. You can also see that with the `resize_token_embeddings` methods. The only test with a decoder we have for BERT is `create_and_check_bert_lm_head` and is passing. Do you see any other test we can apply to check if the sharing part works as expected?

### Dense -> DenseEinsum
> Does the change from tf.keras.layers.Dense to tf.keras.layers.experimental.EinsumDense imply a breaking change or some magic that we must do to ensure that the weights get correctly loaded?

No breaking change at all. Old model format can be loaded in the new one and vice versa. And the magic is as simple as a single `reshape` call because both must have compliant shapes:

```
if K.int_shape(symbolic_weight) != saved_weight_value.shape:
    try:
        array = np.reshape(saved_weight_value, K.int_shape(symbolic_weight))
    except AssertionError as e:
        e.args += (K.int_shape(symbolic_weight), saved_weight_value.shape)
        raise e
    else:
        array = saved_weight_value
```

And this is already integrated in the current release https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_utils.py#L290

> Is that change the main one that unlocks better performance?

Exactly! Only that change to unlock serving performance.

### Tuples -> tf.TensorArray
> Why is that change necessary? Does it give better performance? Is it necessary for graph mode?

As you said it is necessary to be able to use the `output_hidden_states` and `output_attentions` boolean parameters in graph mode. Otherwise we have to disable them 100% of the time in graph mode and it is pitty to remove that feature. I don't think it gives better performance but thanks to this we can have a variable output size. Second good reason to adopt this feature is for serving, if a SavedModel has been created with `output_attentions=True` the model will give you 12 outputs (one for each element of the tuple) instead of just one as it is the case with `tf.TensorArray`. At the end it is not really a breaking change as a tensor can be used as a tuple, you can test this by yourself simply by doing:

```
import tensorflow as tf
t = tf.constant([[1,2],[3,4]])
a, b = t
```

### Joining the `nsp` + `mlm` in one `cls`
> This is cool as it's more similar to what we do in PyTorch! However, you're now handling the names of the weights directly in the `modeling_tf_utils.py` file. I cannot imagine that scales as more models are converted to this implementation?

Exactly, everything will rely on the `load_tf_weights()` function in `modeling_tf_utils.py`. I didn't have major issues to handle that case, it was quite simple, at least in BERT. Let's see how it goes for the others.

Ok @sgugger I should have addressed all your comments.

Ok @patrickvonplaten @sgugger @LysandreJik I should have done the changes you proposed. May I proceed the same way for the other models? Or you want me to do other updates?

I gave up to filter out the empty tensors from the output, it was too complex to implement. So now we will always have 3 outputs corresponding to `attentions`, `hidden_states` and `logits`. But if `output_attentions` or `output_hidden_states` equals `False` they will be empty tensors (first dim equals to 0).

Ok, now BERT looks exactly like I expected. Properly optimized + cleaner code base + full compliance with AutoGraph. Next step is to apply the same changes to the other models.

> flip the return_dict=True switch (I don't think this should be done here - as it affects all PT models as well)

This flip is part of the improvements, if this flip is not here I can remove basically almost half of the improvements because the model will not be able to run properly in graph mode.

> discuss whether we want to do the parameter renaming

Ok. But for now as far as I have seen, it concerns only BERT, but I still need to update other models I did not updated all of them, just few for now (wait the big next push).

I agree that this PR concerns already huge changes with only updating BERT, and indeed to do one PR per model would be easier to handle. Nevertheless, I already started to update the other models, so I will revert locally and create one branch for each on my fork.

@patrickvonplaten I removed the naming updates in order to better discuss this in a later PR.

I still have work to do on the model part, mainly still two things:

- Being able to properly read the model summary
- Being able to properly build a graph visualization

Saying this because usual subclass models cannot be parsed by Keras internals ([see this issue](https://github.com/tensorflow/tensorflow/issues/31647) to have a better explanation of the problem)

Thanks @sgugger for your comments.

> On the comments related to this PR specifically now, the big problem is that we cannot change the global switch of return_dict now. This will break changes in TF, PyTorch and Flax fopr all users of the library. Depending on the time needed to finish the TF push, there might be intermediate releases before v4 where we can't afford to have that breaking change. There is a crude way to have that switch set to True for the TFBert models just now I suggested in the comments and there is probably some clever way with a private flag that could allow us to add the TF models one by one with a new default without breaking anything. I can look more into it if you want.

I don't expect this PR to be released before the v4.0 (to appear in an intermediate release). All would like that all the TF improvements comes directly in once in the next major release. Then I'm acting like if it was the next major release, thus the `return_dict=True` by default in the config. But for sure I can open a PR for each model, I agree that it will be easier to handle. Also nothing prevent to make this change in a specific PR and I will rebase this one on it, this will be the same.

> Another comment is that we should add some more tests of forward/backward compatibility with the new loading function, just to be absolutely sure we don't break anything.

Any idea of what are the other tests I can add, for this? For me being able to load "old" and "new" model in same time is enough. I will be happy to have your opinion on how to improve this :)

@sgugger @patrickvonplaten @LysandreJik I have revert the conflicting changes in order to move the discussion into another PR (the general `return_dict` update and the layers naming).

@sgugger I have put a warning as you suggested. But please let me know if you have a better one, I'm not really happy of the one I put.

In general, do you have any other comments on this? So I can move to the other models in order to apply the same updates if we all agree of these new changes.

Once you're happy with the state of that PR, could you close this PR and open a new one explaining the exact changes? This is a very big thread that takes a while to load, and it's hard to understand the state this current PR is in. Thanks!

You are totally right and this was my plan. Since last week I'm shrinking this PR to get only the optimization part and put the other improvements into separate PRs. Once done, I will reopen a clean one 👍 

Fantastic! Thanks @jplu!