enable pickling for TF Bert models
Author: btel
Pull request: yes
Labels: name: wontfix

this implements `__getstate__` for BERT models to enable pickling (without this PR the pickle attempt due to `weakref` errors). It also adds a unit test.

Comments:
Hi! Why would you prefer using pickle rather than `save_pretrained`/`from_pretrained` or `torch.save`/`torch.save`?

HI @LysandreJik, pickle is mainly useful for parallel processing frameworks like `joblib` or `dask`. The use case is to parallelize some (embarassingly parallel) computation on multiple CPUs/GPUs. Usually, they use pickled objects to send to workers.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4911?src=pr&el=h1) Report
> Merging [#4911](https://codecov.io/gh/huggingface/transformers/pull/4911?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ac99217e92c43066af7ec96554054d75532565d7&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4911/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4911?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4911   +/-   ##
=======================================
  Coverage   76.99%   77.00%           
=======================================
  Files         128      128           
  Lines       21602    21607    +5     
=======================================
+ Hits        16633    16639    +6     
+ Misses       4969     4968    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4911?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4911/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.51% <100.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4911/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.49% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4911/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.49% <0.00%> (+0.40%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4911?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4911?src=pr&el=footer). Last update [ac99217...dd7120b](https://codecov.io/gh/huggingface/transformers/pull/4911?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This is cool! Could we also add it to the PyTorch models?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.