Dataset and DataCollator for BERT Next Sentence Prediction (NSP) task.
Author: mojave-pku
Pull request: yes

I am working with some pre-training tasks recently, and found that transformers have not yet supported NSP tasks. 
So I write the code of **Dataset and DataCollator for BERT NSP task**, now I try to contribute.

This is my first contribution to an open source project. If there is something wrong, please point it out, thank you.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6572?src=pr&el=h1) Report
> Merging [#6572](https://codecov.io/gh/huggingface/transformers/pull/6572?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cfa26d2b412ac3494eef06506004ca857c115ad9?el=desc) will **increase** coverage by `1.01%`.
> The diff coverage is `13.15%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6572/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6572?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6572      +/-   ##
==========================================
+ Coverage   79.24%   80.26%   +1.01%     
==========================================
  Files         156      156              
  Lines       28130    28243     +113     
==========================================
+ Hits        22292    22668     +376     
+ Misses       5838     5575     -263     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6572?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.28% <ø> (ø)` | |
| [...rc/transformers/data/datasets/language\_modeling.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL2xhbmd1YWdlX21vZGVsaW5nLnB5) | `56.32% <10.52%> (-35.52%)` | :arrow_down: |
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `62.80% <13.33%> (-28.11%)` | :arrow_down: |
| [src/transformers/data/datasets/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `28.84% <0.00%> (-58.66%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `71.84% <0.00%> (-23.17%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `79.69% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (ø)` | |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.59% <0.00%> (+0.68%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.29% <0.00%> (+0.97%)` | :arrow_up: |
| ... and [7 more](https://codecov.io/gh/huggingface/transformers/pull/6572/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6572?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6572?src=pr&el=footer). Last update [cfa26d2...e924e84](https://codecov.io/gh/huggingface/transformers/pull/6572?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for reply and carefully review!
I understand what you and @JetRunner mean now. 
The purpose of submitting this script is to provide a tool that can test `DataCollator` and `TextDataset` (I don't know if I am doing this right). 
I agree with you and think it is not elegant to add a `run_bert_nsp.py` script alone. I will remove this script and support NSP together with MLM objective later.

Mmmm, your rebase makes the PR touch way too many files now and quite unreadable. You should close and reopen it (once ready) so we can see the diff a little bit better.

ok i will reopen the pr. thanks~

I’m very sorry, I didn’t know that pr should be closed before rebase, which resulted in PR touch way too many files.
I didn't notice you have already implement the `BertForPreTraining` in the modeling_bert file before, so I modified `BertForNextSencencePrediction` to support both mlm and nsp objectives. 
These changes have now been revert.

@sgugger  `DataCollatorForNextSencencePrediction` and `TextDatasetForNextSencencePrediction` now support mlm and nsp together. 

You need to reopen a new PR, this one will forever have all those files diff with all the old commits you rebased, so we can't review it.

got it.