Calculate position ids on the fly in `forward`
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

I noticed that for some models the position ids are calculated correctly on the fly only in `prepare_ids_for_generation`. However, if calling the `forward` of the model with left padding, position ids do not take into account any attention mask. So they return incorrect logits, since these models use absolute position embeddings.

Given that models like GptBigCode already have correct way to get position ids in `forward`, I decided to add it for other GPT models with absolute position embeddings.

The script I used is:
```python
model = AutoModelForCausalLM.from_pretrained("gpt2-medium")
tokenizer = AutoTokenizer.from_pretrained("gpt2-medium")
tokenizer.pad_token_id = tokenizer.eos_token_id
tokenizer.padding_side = "left"
inputs_padded = tokenizer("Alice and Bob", padding="max_length", max_length=14, return_tensors="pt")
inputs = tokenizer("Alice and Bob", return_tensors="pt")

#make sure that `generate` is working for padded and unpadded inputs
gen_out_padded = tokenizer.batch_decode(model.generate(**inputs_padded, max_new_tokens=10), skip_special_tokens=True)
gen_out = tokenizer.batch_decode(model.generate(**inputs, max_new_tokens=10), skip_special_tokens=True)
#assert(gen_out_padded == gen_out), f"{gen_out_padded} is not equal to {gen_out}"

# if generate is working, then the logits for next token should be nearly-equal
outputs_padded = model(**inputs_padded)
outputs = model(**inputs)
print(torch.max(torch.abs(outputs_padded.logits[:, -1] - outputs.logits[:, -1]))) 
#>>> tensor(172.8631, grad_fn=<MaxBackward1>)
print(outputs_padded.logits[:, -1].argmax(), outputs.logits[:, -1].argmax()) 
#>>> tensor(290) tensor(338)


# Right padding (same result with or w/o fixes in this PR)
tokenizer.padding_side = "right"
inputs_padded = tokenizer("Alice and Bob", padding="max_length", max_length=14, return_tensors="pt")
inputs = tokenizer("Alice and Bob", return_tensors="pt")

#`generate` does not work for right padding apparently
# but the logits for next token are nearly-equal
outputs_padded = model(**inputs_padded)
outputs = model(**inputs)
print(torch.max(torch.abs(outputs_padded.logits[:, 2] - outputs.logits[:, 2]))) 
#>>> tensor(8.3923e-05, grad_fn=<MaxBackward1>)
print(outputs_padded.logits[:, 2].argmax(), outputs.logits[:, 2].argmax()) 
#>>> tensor(338) tensor(338)

```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29583). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gante Thanks for outlining all the pros and cons! 

We already create the attention_mask is quite a few cases when it's not passed in explicitly. I think it's fine to create the correct position_ids and attention_mask if both are `None` and we can correctly infer what they should be. My only question would be if we can reliably do this inference e.g. do we always know which side we should be padding from? 

If I've understood correctly, there's also a possible case when the attention_mask and/or position_ids passed in are wrong. In this case, I think we should raise a warning and we shouldn't overwrite what the user has passed in. 

Let me know if this seems reasonable - or if there's something I might be overlooking 

>  My only question would be if we can reliably do this inference e.g. do we always know which side we should be padding from?

@amyeroberts This is not a problem: if we have `input_ids` containing pad tokens and we know the id of the pad token, we automatically know the padding side and the correct value for the other tensors. If there are no pad tokens in `input_ids`, then we don't need to know the padding side, building the missing tensors is even simpler, and there are no warnings to be raised :)

____________________________

The suggestion [build unset tensors when we have all needed information, warn if they seem incorrect] seems reasonable. Going to re-review the PR and think how to tackle this problem in general :)

Hey @zucchini-nlp 👋 

Revisiting this PR -- how about we do the following:
1. Abstract the logic you suggest [which infers position_ids from the attention mask when it exists, an upgrade on the existing logic on most models] into a function in  `PreTrainedModel` in `modeling_utils.py`. Something that can be called like `position_ids = self.infer_position_ids(input_ids, attention_mask, past_kv_length)`
2. Replace all cases of `if position_ids is None: <compute position_ids>` in `forward`
3. Replace the same logic in `prepare_inputs_for_generation`
4. Add tests to this function

This way, there is a single source of truth for this argument inference logic, as opposed to being replicated (and incomplete!) in many places. Moreover, it would be well tested. WDYT?

(then we can open a similar PR for `attention_mask` and other common inputs that can be derived)

@gante Yes, that sounds great. I am totally pro of having `prepare_inputs_for_generation` more or less same across models by calling the general "prepare_if_none", and updating only model specific kwargs within the method.

I can start working on it next week. I guess that this rework will also require to fix the incorrect generation cases when position ids are passed to `generate`. This way we can be done with one model kwargs and move on to the next one.