🚨🚨🚨 [`Pix2Struct`] Attempts to fix training issues 🚨🚨🚨
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR attempts to partially fix: https://github.com/huggingface/transformers/issues/22903 for better user experience when training `Pix2Struct`.

As stated in the aformentioned issue, some users are having hard times to train pix2struct, for many reasons, some of them being:
- Force adding the special tokens when encoding text --> otherwise the model will keep repeating the generated text
- Remove label smoothing to comply with other model architectures design
- Also remove label masking to be consistent with other models. As referred in https://github.com/huggingface/transformers/issues/22903#issuecomment-1518275840 I agree it should be users responsibility to add that masking

With these fixes, the following script:
```python
import requests
from PIL import Image
from transformers import Pix2StructForConditionalGeneration, AutoProcessor
from torch.optim import AdamW
import torch

torch.manual_seed(42)

model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-base", torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("google/pix2struct-base")

dummy_target = "The model should overfit this sentence"
image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)

encoded_image = processor(images=image, return_tensors="pt")
encoded_text = processor(text=dummy_target, return_tensors='pt', max_length=20)
optimizer = AdamW(model.parameters(), lr=1e-4)

model.train()

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model.to(device)
flattened_patches=encoded_image.flattened_patches.to(device).to(torch.bfloat16)
attention_mask=encoded_image.attention_mask.to(device)
labels=encoded_text.input_ids.to(device)

for i in range(1000):
    outputs = model(
        flattened_patches=flattened_patches,
        attention_mask=attention_mask,
        labels=labels
                   )
    loss = outputs.loss
    
    loss.backward()

    optimizer.step()
    optimizer.zero_grad()
    if i % 50 == 0:
        model.eval()
        prediction = model.generate(
            flattened_patches=flattened_patches,
            attention_mask=attention_mask)
        print(f'step: {i} train_loss: {loss.item()} prediction: {processor.batch_decode(prediction)}')
        model.train()
```
Goes from outputting:
```bash
step: 0 train_loss: 8.259493827819824 prediction: ['<pad> <img_src=cropped-img-20180924']
step: 50 train_loss: 1.9695181846618652 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 100 train_loss: 2.071323871612549 prediction: ['<pad> <The model should overfit this sentence should overfit this sentence should overfit this sentence should']
step: 150 train_loss: 2.0366554260253906 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 200 train_loss: 1.8225889205932617 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 250 train_loss: 1.6568734645843506 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 300 train_loss: 1.6770282983779907 prediction: ['<pad> The model should overfit this sentence sentence should overfit this sentence sentence should overfit this sentence']
step: 350 train_loss: 1.688515067100525 prediction: ['<pad> The model should overfit this sentence sentence overfit this sentence sentence overfit this sentence sentence over']
step: 400 train_loss: 1.6118296384811401 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 450 train_loss: 1.6204414367675781 prediction: ['<pad> The model should overfit this sentence sentence should overfit this sentence should overfit this sentence should']
step: 500 train_loss: 1.59645676612854 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 550 train_loss: 1.5818239450454712 prediction: ['<pad> The model should overfit this sentence sentence sentence sentence sentence sentence sentence sentence sentence sentence sentence sentence sentence']
step: 600 train_loss: 1.5775129795074463 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 650 train_loss: 1.561257243156433 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 700 train_loss: 1.5319150686264038 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 750 train_loss: 1.646193504333496 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 800 train_loss: 1.533736228942871 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 850 train_loss: 1.6203268766403198 prediction: ['<pad> The model should overfit this sentence should overfit this sentence should overfit this sentence should over']
step: 900 train_loss: 1.5132172107696533 prediction: ['<pad> The model should overfit this sentence sentence should overfit this sentence sentence should overfit this sentence']
step: 950 train_loss: 1.491452693939209 prediction: ['<pad> The model should overfit this sentence The model should overfit this sentence The model should overfit']
```
To:
```bash
step: 0 train_loss: 9.75 prediction: ['<pad> <<img_src=1> <img_src=2> <img_src=']
step: 50 train_loss: 0.125 prediction: ['<pad> <<img_src=1> <img_src=1> <img_src=']
step: 100 train_loss: 0.0089111328125 prediction: ['<pad> The model should overfit this sentence
']
...
```

cc @sgugger @amyeroberts @NielsRogge 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

For me it's the same thing as what we discussed with Luke yesterday. It's important to have a consistent API so 100% for:
- leaving label_smoothing out of the loss computation by default (users can compute the loss themselves by not passing the logits or using the Trainer with label_smoothing)
- using -100 as ignore index and not the pad token (this is something I should have caught in the review, and we have already gone to a lot of trouble to harmonize all models to this)

Thanks for the review! 
Let me know if I should also make the changes for BLIP as well as you suggested @amyeroberts 

@younesbelkada Yes please! 

Hmm actually this broke some slow tests, I will need to dig more into that combined with https://github.com/huggingface/transformers/issues/22903#issuecomment-1525771904 , will let you know


False alarm, I will just properly document how to use pix2struct for conditional text generation!

Hi,

I'm having problems when trying to fine-tune the pix2struct model.
I am basing myself on the notebook from https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Pix2Struct and that is why I have generated an issue in that repository where I explain my problem in detail: https://github.com/NielsRogge/Transformers-Tutorials/issues/293

Mainly it is that after several trainings and inferences with the generated models I see that the result of the inference is always the same, regardless of the input image.

Do you know what could be happening or how to fix it?