Support keras JSON/HDF5 serialization of main layers
Author: gthb
Pull request: yes

Fixes #3101

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3103?src=pr&el=h1) Report
> Merging [#3103](https://codecov.io/gh/huggingface/transformers/pull/3103?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a088d75e510d5641808ccd72f5dca4df36d95b8e?src=pr&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3103/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3103?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3103      +/-   ##
==========================================
+ Coverage   77.82%   77.88%   +0.05%     
==========================================
  Files          98       98              
  Lines       16422    16461      +39     
==========================================
+ Hits        12781    12821      +40     
+ Misses       3641     3640       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3103?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hbGJlcnQucHk=) | `85.53% <100%> (+0.08%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `96.16% <100%> (+0.02%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `95.84% <100%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_tf\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG5ldC5weQ==) | `89.03% <100%> (+0.04%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.2% <100%> (+0.63%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `96.07% <100%> (+0.01%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90cmFuc2ZvX3hsLnB5) | `91.18% <100%> (+0.04%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9jdHJsLnB5) | `99.57% <100%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3103/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `94.61% <0%> (+0.15%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3103?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3103?src=pr&el=footer). Last update [a088d75...4c91a3a](https://codecov.io/gh/huggingface/transformers/pull/3103?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Crud, as the test shows, of course load doesn't work in this first-stab implementation, because the layer classes get instantiated with a `dict` instead of a `PretrainedConfig` instance. So needs more work.

With this change, seven of the 11 `TF*MainLayer` classes pass the test (saving, loading and producing same output for same input after the Keras save/load roundtrip). So I'm just not marking the other four `@keras_serializable` as I haven't gotten it working for those. Specifically:

* TFT5MainLayer does not accept the same model inputs directly, as produced by `self.model_tester.prepare_config_and_inputs_for_common()`, so calling it fails
* `TFXLMMainLayer`, `TFOpenAIGPTMainLayer`, and `TFDistilBertMainLayer` all fail the test (if I add the `@keras_serializable` attribute to them) in the same way: by outputting a tensor of shape `(7, 32)` after the save/load round-trip, which is just the first row of the `(13, 7, 32)` tensor that's output before. I haven't figured out the cause of this.

Ok this looks good to me. 

Do you want to make `make style` and `make quality` to pass the code quality checks (see our [contributor guide](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) if needed), and I guess we can merge it.

Fixed quality thing, added @functools.wraps (really shouldn't wrap without that as it causes confusing function metadata), added a docstring to the `keras_serializable` decorator, and changed to use a name specific to this library rather than the general name `config`, to be clearer where `transformers` is used along with other things in a Keras model. @thomwolf OK with all that?

Ok this is good to me.
Thanks a lot for the awesome work on this.

Merging @LysandreJik @julien-c 

This landed in 2.6.0 but is missing in the release notes there https://github.com/huggingface/transformers/releases/tag/v2.6.0

Indeed, that is my fault, I'm sorry for missing it.

I've added it to the v2.7.0 release notes of this morning: https://github.com/huggingface/transformers/releases/tag/v2.7.0

Thanks again for your contribution @gthb !