Converting GPT2 logits to token ids directly
Author: aclifton314
Pull request: no

## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 2.11.0
Python: 3.7.6

## Background Info
Here is the constructor and forward method for the model I am trying to build. Ultimately it will finetune GPT2 with a loss from a ngrams model I made. The loss isn't implemented yet because I want to test the GPT2 generation part first.

```python
class GPT2FinetunedWithNgrams(GPT2LMHeadModel):
    def __init__(self, config):
        super().__init__(config)
        self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

def forward(
        self,
        input_ids=None,
        past=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        use_cache=True,
):
    temperature = 0.85
    tmp_input_ids = input_ids
    max_gen_length = 30
    counter = 0
    orig_input_str = self.tokenizer.decode(input_ids[0], skip_special_tokens=True)
    strs_to_join = orig_input_str.split()
    while counter < max_gen_length:
        transformer_outputs = self.transformer(
            tmp_input_ids,
            past=past,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
        )

        hidden_states = transformer_outputs[0]
        lm_logits = self.lm_head(hidden_states) / (temperature)
        last_token = lm_logits[:, -1]
        last_token_softmax = torch.softmax(last_token, dim=-1).squeeze()

        next_token = torch.argmax(last_token_softmax).tolist()
        next_gen_token_str = self.tokenizer.decode(next_token, clean_up_tokenization_spaces=True).strip()
        strs_to_join.append(next_gen_token_str)

        new_str_input = ' '.join(strs_to_join)
        tmp_input_ids = self.tokenizer.encode(new_str_input, return_tensors='pt')
        counter += 1
    return new_str_input
```
Right now the code will take the `lm_logits`, calculate the softmax, and then get the next token predicted by GPT2. I then add that next token to the original input sequence and feed that combination back into GPT2, until the `max_gen_length` is reached. Finally it returns the original input sequence with the generated sequence appended to it.

## Question 1
Is there a way to directly go from logits to token ids that I am missing in HF Transformers? Or better yet, is there a better, more efficient way of doing this?

## Question 2
Given that I am using things like a `max_length` and `temperature`, I feel like the `generate()` method would be useful here. However, I am not exactly sure how to implement that. Since ultimately I want my model to finetune GPT2 based off a ngrams loss, I don't know if calling `generate()` will call that method from the GPT2 model that is involved in the finetuning or some fixed GPT2 model coming elsewhere. Part of this is my nonfamiliarity with how pytorch trains (which I'm working on understanding better). 

Said differently, if I can call `generate()` within the above forward method (I imagine something like `super().generate()`) will that generate a sequence using the GPT2 model whose weights are currently being modified based off my dataset and finetuning, or will it generate a sequence from some static version of GPT2 that is not having its weights modified?

I hope those questions aren't too convoluted. I can elaborate more if needed. Thanks in advance for your help!


Comments:
Hi! This is a very interesting question, I think you would have additional answers if you asked it over on the forums: https://discuss.huggingface.co