[GPT2] Propose fix for #21080
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Propose fix for #21080.
Modify the default generation of the positional ids for the `gpt-2` as well as the `decidion_transformer` model. 
In the issue @LysandreJik proposed to update the doc, but when I checked it seemed like this would only affect 2 models, and is backward compatible: 
- potential impact is only for people that where using batched padded input when generating with a single sentence. This means that their output will now be corrected
- default behavior is kept if not attention mask is given. 

If this is not acceptable, I am also glad to add a warning when creating the positional ids.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'd like to summon the generation expert @gante to ask what he thinks of the PR

The summon has been heard 📣 

### TL;DR

I approve the change 👍 But we need to have a second look at the position embeddings, I suspect there are several bugs in the codebase (see why below)

### Context
This PR actually took me on a long trip, whose findings I summarize below:
1. I saw that @ArthurZucker wrote `this would only affect 2 models`, and my first thought was `what about GPT-J`?
2. Then I saw #21869 (remove `position_ids` input from GPT-J). From the TF XLA `.generate` transition, I remember that getting the `position_ids` to work with GPT-J was a nice piece of work, and I remember that it made a difference in the outputs. See the example below

<details>
  <summary>GPT-J + positions_ids</summary>
  
  ```py
  from transformers import TFAutoModelForCausalLM, AutoTokenizer
  import tensorflow as tf
  
  tf.keras.backend.set_floatx('float16')
  
  tok = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B", padding_side="left")
  model = TFAutoModelForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", revision="float16", from_pt=True)
  tok.pad_token = tok.eos_token
  model.generation_config.pad_token_id = model.generation_config.eos_token_id
  
  inputs = tok(["and the prime minister"], return_tensors="tf", padding=True)
  out_1 = model(**inputs)
  out_2 = model(**inputs)
  
  position_ids = tf.math.cumsum(inputs.attention_mask, axis=-1, exclusive=True)
  out_3 = model(**inputs, position_ids=position_ids + 10)
  
  print(tf.reduce_max(tf.abs(out_1.logits[:, -1, :] - out_2.logits[:, -1, :]))) # tf.Tensor(0.0, shape=(), dtype=float16)
  print(tf.reduce_max(tf.abs(out_1.logits[:, -1, :] - out_3.logits[:, -1, :]))) # tf.Tensor(0.01563, shape=(), dtype=float16)
  ```
</details>


3. Despite the above, and looking at this PR, I think what @ArthurZucker wrote here is the way to go. In `prepare_inputs_for_generation`, we were computing `position_ids` from the `attention_mask` anyways. If we pass the logic from there to the forward pass, we cut a source of bugs (users trying to generate without `.generate()`) 👍 
4. GPT-J needs to be fixed (see TF/FLAX) :p To be clear, the issue is long-standing and not a result of #21869 !
5. We should double-check at least the main models. Some models, like `OPT`, are okay, as they compute the position embedding directly from the attention mask. Others, like `Codegen`, suffer from the same problem as `GPT-J`.
 


@gante I was coincidentally just having this problem with codegen and so have opened #22069 following your hints above.

Thanks for the in depth review @gante ! 

Will re-open this with a fix for the cross PT-TF tests. The TF code has to be modified as otherwise the default positional ids are wrong. 

Sorry for missing that the tf version also needed an update on this one! 

Thank you @ArthurZucker . No worry - we didn't detect this because the PT/TF cross tests in the corresponding TF model test files are not fetched by the test fetcher script. The current version will only detect the tests in the (modified + involved indirectly) PyTorch modeling/test files.