Sampling sequence generator for transformers
Author: rlouf
Pull request: yes

[@thomwolf version]: In this PR I introduce a flexible API to generate sequences for the models of the library provided with LM heads.
The API is designed to have a very flexible and quite exhaustive API:
- with/without a prompt
- with/without beam search
- with/without greedy decoding/sampling
- with any (and combination) of top-k/top-p/penalized repetitions

Only single-stack architectures are currently supported. 

Here is an example generating 4 continuations of a prompt with using a beam search of 3 for each continuation and token sampling at each step with top-p filtering and repetition penalization until we reach a <unk> token or a max length of 50.

```python
import torch
from transformers import OpenAIGPTLMHeadModel, OpenAIGPTTokenizer

tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')

input_ids = tokenizer.encode("This is the", return_tensors='pt')

output = model.generate(input_ids,
                            max_length=50,
                            do_sample=True,
                            num_beams=3,
                            temperature=0.7,
                            top_k=0,
                            top_p=0.8,
                            repetition_penalty=1.4,
                            bos_token_id=None,
                            pad_token_id=None,
                            eos_token_ids=tok.unk_token_id,
                            batch_size=None,
                            length_penalty=None,
                            num_return_sequences=4)[0]

for j in range(len(output)):
    print(tok.decode(output[j].tolist()))
```

To-add features:
- Add support for encoder-decoder architectures
- Add support for past and cached states

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1840?src=pr&el=h1) Report
> Merging [#1840](https://codecov.io/gh/huggingface/transformers/pull/1840?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8618bf15d6edc8774cedc0aae021d259d89c91fc?src=pr&el=desc) will **increase** coverage by `0.32%`.
> The diff coverage is `13.93%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1840/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1840?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1840      +/-   ##
==========================================
+ Coverage   78.91%   79.23%   +0.32%     
==========================================
  Files         131      131              
  Lines       19450    19680     +230     
==========================================
+ Hits        15348    15593     +245     
+ Misses       4102     4087      -15
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1840?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `92.1% <ø> (ø)` | :arrow_up: |
| [transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fdXRpbHMucHk=) | `93.75% <100%> (+0.89%)` | :arrow_up: |
| [transformers/configuration\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25feGxtLnB5) | `96.42% <100%> (+0.13%)` | :arrow_up: |
| [transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbS5weQ==) | `86.49% <11.11%> (-1.81%)` | :arrow_down: |
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `73.72% <12.5%> (+1.51%)` | :arrow_up: |
| [transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2VuY29kZXJfZGVjb2Rlci5weQ==) | `27.9% <16.66%> (+1.98%)` | :arrow_up: |
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `64.29% <5.78%> (-28.74%)` | :arrow_down: |
| [transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RyYW5zZm9feGwucHk=) | `75.89% <80%> (-0.01%)` | :arrow_down: |
| [transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3BpcGVsaW5lcy5weQ==) | `67.94% <0%> (+0.58%)` | :arrow_up: |
| ... and [6 more](https://codecov.io/gh/huggingface/transformers/pull/1840/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1840?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1840?src=pr&el=footer). Last update [8618bf1...f86ed23](https://codecov.io/gh/huggingface/transformers/pull/1840?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Notes:
 
1. XLM's CLM model gives good results for only one of the languages.
2. Although using XLM's MLM pretrained weights + a mask token at the end of sequence could have worked, it doesn't give sequences that make sense. I think we should instead display a warning when the user attempts to use a mlm.

Ok this looks good! It's quite a change to what we were doing previously, which was hosting all the generation logic inside the script. It may have become a bit too cluttered as models with different behaviors and attributes were added, so I believe this is a welcome change.

This also means that we're adding a level of abstraction for the user; it does not showcase the way the models should be used for generations with different inputs anymore. This is fine with me but I think we need to add some additional snippets showcasing how the models should be used when doing generation, perhaps in https://huggingface.co/transformers/quickstart.html?

Do you mean exposing the logic inside `SamplerXLM` and `SamplerXLNet`?

I think moving some decoding in the core library is a good idea.

Though I have doubts on a few aspects of this PR:
- the functional approach currently proposed for implementing this feature is very different from the way the rest of the library is designed, and
- having to maintain another location (`generation/sampler`) with model-specific peculiarities adds burden when implementing a new model and is not really the way we are trying to take right now (simplifying the addition of new models).

I'll try to think a little bit about how we could move this PR closer to the current and future directions of the library and come back with proposals.

> * the functional approach currently proposed for implementing this feature is very different from the way the rest of the library is designed, and

I agree it looks different from most of the code in the library, but I am no sure what you mean by functional. The code could not get a lot more OO than this. Plus I think it is very legible as is.

> * having to maintain another location (`generation/sampler`) with model-specific peculiarities adds burden when implementing a new model and is not really the way we are trying to take right now (simplifying the addition of new models).

Let's consider for a second what happens when we add model X to the library and X could be used to generate sequences.

1. X does not require specific pre-treatment or input, computed masks etc at each forward pass in which case we have nothing to do and it is supported out of the box (`SamplerSingleStack`).
2. X does require some specific preprocessing before each forward pass (hooks on the forward pass if you wish), you subclass `SamplerSingleStack` and reference it in the factory.
3. You're done

Taking a step back I don't think it is a great idea to add model-specific logic in `generate` either (this would belong in a more battery-included library). What we could do, however, is move `SamplerSingleStack` subclasses (`SamplerFroXLM` and `SamplerForXLNet`) in the example script. I am fine with keeping the `SamplerSingleStack` and the future `SamplerEncoderDecoder` there as they are very general.

And yes, please give me suggestions to future-proof the PR as I currently have little visibility on this.

After some thoughts, here is a proposal for re-writing the user experience.

Context:
- I'm ok with adding abstractions as long as they are used internally. For user-facing abstraction like here, we need a really strong reason to increase the learning curve for the user and I don't see one here.
- the second important element is to reduce the burden for us (and increasingly the community) of updating many things at many places when adding a new model (currently already too high).

Proposal:
- let's have the decoding logic be accessed by the user by calling a (new) `decode` method of `PretrainedModel` instead of a new class (so no new abstraction).
- a Decoder can be added to the model at instantiation (in the `PretrainedModel` class `__init__`), probably only if `get_output_embeddings` is not None (i.e. with have a model with a LM head of some kind), and selected with a configuration parameter (to add to `PretrainedConfig`).
- a specific `_prepare_input_for_decoding` method (can find a better name) can be in charge of preparing the inputs for decoding and overridden in the specific model classes with model-specific preprocessing if needed (ex in XLM and XLNet). This way all the specificities of the model architectures are in the model class. This method should also take care of PyTorch/TF 2.0 specificities (currently the decoding classes are only PyTorch specific). I think the samplers can even be in `model_utils.py` if they are PyTorch specific or in another file if they are framework agnostic.

Tell me if it's not clear, I can detail further.

Question:
- if we have a decoder with trained parameters in the future, we'll want to have the `decode` method behind the `forward` method to catch all PyTorch hooks (like in [XLM](https://github.com/facebookresearch/XLM/blob/master/src/model/transformer.py#L320-L330) for instance). Probably for now we can just not worry about that.

Clear for now, but the devil is in the details so I'll probably ask for clarification later.

I followed your idea to move the logic closer to the models, and simplified the implementation a little bit. As a result, the `sampler.py` only contains the `Sampler` and `SamplerSingleStack` class (`SamplerEncoderDecoder` coming, slightly different from the single stack case as we only want to pass through the encoder once). Here is a summary of changes:

1. Removed model-specific classes in the `generate` module. It is much nicer.
2. Added a `decode` and a `_prepare_input_for_decoding` method to `PretrainedModel`. By default, the first raises a `NotImplementedError`, the second one returns a dictionary `{"input_ids": input_ids}`
3. Overrided `decode` in `GPT2LMHeadModel,`OpenAIGPTLMHeadModel`,` XLNetLMHeadModel`, `TransfoXLLMHeadModel`, `CTRLLMHeadModel`, `XLMWithLMHeadModel`; If the user tries to initiate a sampler with another model, they will be greated with an explanatory error message.

As a result the design feels nicer and easier to maintain & to use.

I needed to add Encoder-Decoder support as a backup to the beam search in `example-summarization`, so I went on with it.

Adding encoder-decoder support with the previous changes was a breeze: I added a `decode` and an `encoder` method to the `PreTrainedEncoderDecoder` class and added a very simple `SamplerEncoderDecoder` class. I also moved the preparation of the encoder/decoder kwargs into a new method (which can thus be overriden).

Routing the factory function is based on the presence/absence of the `decode` and `encode` methods in the respective classes.

Improvements for another PR:
- Support batch size > 1
- Support num_samples > 1

@thomwolf I believe I implemented a simpler version of your recommendations. Let me know if there is something that I did not understand correctly.

Hi @thomwolf could you tell me if the changes (to the single-stack case) I just made are in the spirit of your proposal? I decided to remove the `SamplerSingleStack` class that doesn't really have a purpose  now and `SamplerEncoderDecoder` will have the same fate. All the decoding logic is in the `decode` function in `PreTrainedModel`.

With this implementation I don't think we do need a `Sampler` class anymore. We can just have a collection of functions (its current methods) and pass the configuration as a dictionary. It is simpler, and makes `decode` truly indempotent. What do you think?

It am still afraid the `decode` function is going to become a big plate of spaghetti when we add new methods, but maybe there is a way to prevent that from happening?

(there may be a few errors here and there, I just quickly drafted the new implementation).

@thomwolf I followed your API recommendations. As a result:
- The `Sampler` class has been moved to `modeling_utils.py`. `SingleStackSampler` an `EncoderDecoderSampler` are not needed anymore so I removed them. I thus also removed the `generate` folder;
- `PreTrainedModel`  and `PreTrainedEncoderDecoder` now have a `decode` method that raises an error if the model (decoder) does not have a `get_output_embeddings` method and raises an warning if the model is not on the same device as the prompt;
- I added a `get_output_embeddings` method to `TransfoXLLMHeadModel` **please check as I did not take the time to dive in the intricacies of adaptive softmax**
- I initialize `Sampler` each time `decode` is called; it is simpler than initializing it at the same time as the model and incurs little to no overhead;
- I add a `_prepare_inputs_for_decoding` method to `PreTrainedModel` that is overridden for every model that needs it;
- The examples in `run_generation.py` have been updated to reflect the API change. I ran the script for every model;
- I updated all tests;
- I documented the `decode` methods;
- `device` is only needed to check that the decoder and prompt are on the same device. It is not needed in the `Sampler` class at all, so I removed it there.
- I squashed my commits and rebased on master.