Fix tokenizer saving/loading with custom token objects
Author: mozharovsky
Pull request: yes
Labels: name: wontfix

## Summary

This PR fixes issue #5571. 

Pre-trained tokenizers might wrap tokens around custom types (e.g. `AddedToken` from **🤗/tokenizers**) which makes (de)serialization without additional meta information difficult. This PR uses `jsonpickle` library which particularly solves  objects (de)serialization problem.

A little drawback of this approach is that type information is a subject of change and such changes will break back-compatibility once happened. But this seems just like an agreement for making careful back-compatible changes.

Comments:
Investigating the issues 🤓 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6183?src=pr&el=h1) Report
> Merging [#6183](https://codecov.io/gh/huggingface/transformers/pull/6183?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b25cec13c57656941aac3b920eeb488c1915df18&el=desc) will **increase** coverage by `0.57%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6183/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6183?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6183      +/-   ##
==========================================
+ Coverage   79.08%   79.66%   +0.57%     
==========================================
  Files         149      147       -2     
  Lines       27685    26603    -1082     
==========================================
- Hits        21894    21192     -702     
+ Misses       5791     5411     -380     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6183?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_io.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX2lvLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `93.71% <100.00%> (-0.14%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.22% <0.00%> (-63.51%)` | :arrow_down: |
| [src/transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxuZXQucHk=) | `66.66% <0.00%> (-23.43%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `71.65% <0.00%> (-23.08%)` | :arrow_down: |
| [src/transformers/trainer\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3V0aWxzLnB5) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [src/transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fb3BlbmFpLnB5) | `71.21% <0.00%> (-12.88%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hbGJlcnQucHk=) | `76.47% <0.00%> (-11.88%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `80.05% <0.00%> (-2.14%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.21% <0.00%> (-1.51%)` | :arrow_down: |
| ... and [54 more](https://codecov.io/gh/huggingface/transformers/pull/6183/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6183?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6183?src=pr&el=footer). Last update [cdf1f7e...d159584](https://codecov.io/gh/huggingface/transformers/pull/6183?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks much better now 🎉

@n1t0 @thomwolf could you please take a glance? :)

I did a bit of refactoring to have load/dump methods reusable across the library. 


These changes introduce a new scope for the common File IO methods (like json pickle load/dump). Although there is an already existing `file_utils` scope, I believe it's not supposed to contain the common IO operations as the docs say (`Utilities for working with the local dataset cache.`).

@LysandreJik, could you please review the changes? :)

It looks like #6026 fixed the issue. But nevertheless this PR is still relevant since it guarantees tokenizers saving/loading with objects of arbitrary types. 

Hi, thanks a lot for your contribution! Unfortunately we're very strict about adding new dependencies, and this doesn't really change the existing behavior. I'm not sure I see the pros of introducing this new scope vs the cons of integrating a new dependencies + updating existing code.

Hi, thanks for replying! I respect your policy regarding the third-party dependencies, this makes sense. Anyway, the current approaches compromise tokenizers (de)serialization with introducing new objects to include (besides `AddedToken` instances). This might be a subject of future improvements once new types are introduced into the tokenizer configs, but I think it makes sense to consider generalizing the behavior.

If it makes sense, I could wrap saving/loading configs into the new scope without a third-party library, though it's far from generalizing, at least adding new cases to handle will be easier. What do you think? :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.