Sort unique_no_split_tokens to make it deterministic
Author: lhoestq
Pull request: yes

The `unique_no_split_tokens` attribute of tokenizers is not deterministic, and it makes the hashing in the `nlp` lib return different hashes for the same tokenizer over different sessions.

To fix that I changed its type to a `set` instead of a `list`.

Fix #6460 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6461?src=pr&el=h1) Report
> Merging [#6461](https://codecov.io/gh/huggingface/transformers/pull/6461?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9d94aecd516c7540a94b9d781ef28d7375a796bc&el=desc) will **decrease** coverage by `0.47%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6461/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6461?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6461      +/-   ##
==========================================
- Coverage   80.09%   79.62%   -0.48%     
==========================================
  Files         153      153              
  Lines       28005    28005              
==========================================
- Hits        22430    22298     -132     
- Misses       5575     5707     +132     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6461?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.40% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/6461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `25.13% <0.00%> (-70.95%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.97% <0.00%> (-0.33%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.44% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `94.72% <0.00%> (+22.87%)` | :arrow_up: |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `87.50% <0.00%> (+58.65%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6461?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6461?src=pr&el=footer). Last update [9d94aec...dfb7549](https://codecov.io/gh/huggingface/transformers/pull/6461?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I've actually switched in the last version of transformers from a `set` to a `list` for the same reason (not deterministic for `nlp`). Are you sure this really solve the problem @lhoestq ?
Also regarding backward compatibility, I'm fine with changing this from a list to a set @sgugger 

Maybe we should rather have a sorted list?

`sorted` should solves the issue.

I just tested and a `set` doesn't solve it actually. I'll change to `sorted`, thanks @thomwolf 

This is such an important use-case (and potential source of regression) for us that we may want to add a test on that in `nlp` or `transformers` in a not too far future.

Yes definitely. Not sure how to test consistency across sessions in the CI though.
I guess we could have tests with hardcoded hashes for some tokenizers but I'm not sure that's ideal.

Or maybe there's a way to do two CI jobs in a row: one to generate the hashes in a first session, and one to verify that they're the same in another session.