[WIP] BertAbs summarization
Author: rlouf
Pull request: yes

This PR builds on the encoder-decoder mechanism to do abstractive summarizaton. Contributions:

- A BeamSearch class that takes any `PreTranedEncoderDecoder` as an input;
- A script `run_summarization.py` that allows to pre-train the model and generate summaries.

Note that to save the checkpoints I had to add a parameter in the `save_pretrained_model` method, but I am not sure this is the best solution.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1773?src=pr&el=h1) Report
> Merging [#1773](https://codecov.io/gh/huggingface/transformers/pull/1773?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0cb163865a4c761c226b151283309eedb2b1ca4d?src=pr&el=desc) will **increase** coverage by `1.06%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1773/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1773?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1773      +/-   ##
==========================================
+ Coverage   82.67%   83.74%   +1.06%     
==========================================
  Files         111      108       -3     
  Lines       16162    15749     -413     
==========================================
- Hits        13362    13189     -173     
+ Misses       2800     2560     -240
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1773?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `87.7% <100%> (+0.02%)` | :arrow_up: |
| [transformers/hf\_api.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2hmX2FwaS5weQ==) | `96.87% <0%> (-0.63%)` | :arrow_down: |
| [transformers/tests/modeling\_bert\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2JlcnRfdGVzdC5weQ==) | `96.5% <0%> (-0.58%)` | :arrow_down: |
| [transformers/tests/modeling\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2NvbW1vbl90ZXN0LnB5) | `74.23% <0%> (-0.31%)` | :arrow_down: |
| [transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9vcGVuYWkucHk=) | `81.81% <0%> (-0.3%)` | :arrow_down: |
| [transformers/tests/modeling\_openai\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX29wZW5haV90ZXN0LnB5) | `93.2% <0%> (-0.2%)` | :arrow_down: |
| [transformers/tests/modeling\_xlnet\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3hsbmV0X3Rlc3QucHk=) | `94.68% <0%> (-0.2%)` | :arrow_down: |
| [transformers/tests/modeling\_albert\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2FsYmVydF90ZXN0LnB5) | `95.08% <0%> (-0.16%)` | :arrow_down: |
| [transformers/tests/modeling\_gpt2\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2dwdDJfdGVzdC5weQ==) | `94.01% <0%> (-0.15%)` | :arrow_down: |
| [transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl94bG0ucHk=) | `83.2% <0%> (-0.14%)` | :arrow_down: |
| ... and [40 more](https://codecov.io/gh/huggingface/transformers/pull/1773/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1773?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1773?src=pr&el=footer). Last update [0cb1638...6fb9900](https://codecov.io/gh/huggingface/transformers/pull/1773?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@rlouf Thanks for the the PR!
I'm trying to run run_summarization.py with the following command, but got an error. Any idea why?
I have unzipped the cnn and dailymail folders in my data directory. 

```
python examples/run_summarization.py \
        --data_dir /data/home/hlu/notebooks/summarization/data \
        --output_dir ./summarization_output \
        --do_train True \
        --model_name_or_path bert-base-cased \
        --num_train_epochs 1 \
```

W1108 21:29:27.643532 139837444753152 run_summarization.py:488] Process rank: 0, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
I1108 21:29:27.643716 139837444753152 run_summarization.py:491] Training/evaluation parameters Namespace(data_dir='/data/home/hlu/notebooks/summarization/data', device=device(type='cuda'), do_evaluate=False, do_overwrite_output_dir=False, do_train=True, gradient_accumulation_steps=1, max_grad_norm=1.0, max_steps=-1, model_name_or_path='bert-base-cased', model_type='bert', n_gpu=1, num_train_epochs=1, output_dir='./summarization_output', per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, seed=42, to_cpu=False)
I1108 21:30:40.805522 139837444753152 run_summarization.py:198] ***** Running training *****
I1108 21:30:40.805693 139837444753152 run_summarization.py:199]   Num examples = 312085
I1108 21:30:40.806245 139837444753152 run_summarization.py:200]   Num Epochs = 1
I1108 21:30:40.806311 139837444753152 run_summarization.py:201]   Instantaneous batch size per GPU = 4
I1108 21:30:40.806385 139837444753152 run_summarization.py:204]   Total train batch size (w. parallel, distributed & accumulation) = 4
I1108 21:30:40.806446 139837444753152 run_summarization.py:207]   Gradient Accumulation steps = 1
I1108 21:30:40.806501 139837444753152 run_summarization.py:208]   Total optimization steps = 78022
Traceback (most recent call last):
  File "examples/run_summarization.py", line 545, in <module>
    main()
  File "examples/run_summarization.py", line 497, in main
    global_step, tr_loss = train(args, model, tokenizer)
  File "examples/run_summarization.py", line 236, in train
    decoder_lm_labels=lm_labels,
  File "/data/anaconda/envs/nlp_gpu/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/home/hlu/notebooks/huggingface/transformers/transformers/modeling_encoder_decoder.py", line 240, in forward
    decoder_outputs = self.decoder(decoder_input_ids, **kwargs_decoder)
  File "/data/anaconda/envs/nlp_gpu/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/home/hlu/notebooks/huggingface/transformers/transformers/modeling_bert.py", line 865, in forward
    encoder_attention_mask=encoder_attention_mask)
  File "/data/anaconda/envs/nlp_gpu/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/home/hlu/notebooks/huggingface/transformers/transformers/modeling_bert.py", line 677, in forward
    extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
RuntimeError: expected device cuda:0 and dtype Long but got device cuda:0 and dtype Bool


Hi! Thank you for raising the issue. I think it comes from the following lines in `modeling_bert.py`:

```python
causal_mask = seq_ids[None, None, :].repeat(batch_size, seq_length, 1) <= seq_ids[None, :, None]
extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
```

`causal_mask` as defined is a boolean tensor, while the `attention_mask` being passed is of type `long`. I am very confused because I never got the error when I tried locally. I will fix the issue asap. In the meantime, would you mind executing the following code in your environment and pasting the result here?

```python
import platform; print("Platform", platform.platform())
import sys; print("Python", sys.version)
import torch; print("PyTorch", torch.__version__)
```

@rlouf Thanks! Please see the environment info below.
>>> import platform; print("Platform", platform.platform())
Platform Linux-4.15.0-1061-azure-x86_64-with-debian-stretch-sid
>>> import sys; print("Python", sys.version)
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) 
[GCC 7.3.0]
>>> import torch; print("Pytorch", torch.__version__)
Pytorch 1.2.0

Thank you! I’m working with PyTorch 1.3.0, and they introduced type promotion in this release (https://github.com/pytorch/pytorch/releases) which surely explains why it worked for me but not for you. I will work on a fix this week, in the meantime you can simply convert causal mask manually to long, or upgrade to pytorch 1.3 if that is possible.

Edit: I pushed the changes. Let me know if there still is an issue.

Here is the latest on summarization:

- I added utilities to load the CNN/DailyMail dataset, to separate stories from summaries. The utilities are tested;
- I wrapped the BertAbs model along with its configuration to be able to use it with the library's usual API `BertAbs.from_pretrained`. I uploaded the finetuned weights and configuration on S3 and tested that model loads properly;
- I also wrapped the sequence generation so we can generate summaries from any text using Beam Search;
- I removed my previous attempt at Beam Search and saved it for later;
- I added a small fix in `modeling_bert.py` for users that use PyTorch v < 1.30. We convert the causal mask explicitly from `bool` to `long` since type promotion was only introduce in PyTorch 1.3
- I added ROUGE evaluation and am able to reproduce the authors' results on CNN/DailyMail

- [ ] revert changes in `modeling_encoderdecoder.py`



I squashed my commits and rebased on the repo's master branch. The docs are updated. Good to go on my side.