GPT Neo/J Padding Side Results in Different Generation Outputs
Author: anas-awadalla
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: None

### Who can help
@patil-suraj @patrickvonplaten 

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART, BART, Marian, Pegasus: @patil-suraj
- Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- Longformer, BigBird: @ydshieh
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patil-suraj, @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:
- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information
I find that for GPT Neo and GPT J the padding side has a great effect on the quality of the generation. More specifically it seems that while padding on the left results in decent generation output, padding on the right causes the model to generate empty spaces or repeating characters (ex. "AAAAAAA"). I am wondering why padding has this effect on generation.

Model I am using (Bert, XLNet ...): GPT Neo 125M (although I also observed the same issue for larger variants of Neo and GPT-J)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Run this [Colab](https://colab.research.google.com/drive/1LdJ4kt_egJprxmpWZJSwkbvx71kN5RFv?usp=sharing) 

or

1. Load any of the GPT Neo or GPT J models
2. Compare the results of padding inputs on the left vs right on generation outcome

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I expect that it shouldn't matter whether we pad the input from the left or right but it seems like that has a effect on generation results. 
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Hi @anas-awadalla !

This is because these are auto-regressive models, which generate next tokens based on previous tokens. So if the inputs are padded to the right the model will receive padded tokens as context, which is not useful during generation. 

Because of this for generation with any auto-regressive models, the input is padded to the left.

Thanks for your reply! Did you mean the input should be padded to the left? In both cases the input is passed in as part of the context so can you clarify why one padding strategy is better? Also in the generate function we pass input ids and an attention mask, shouldn’t the attention mask indicate what parts of the input is padding tokens and the model should not consider those as part of the context?

> Thanks for your reply! Did you mean the input should be padded to the left? In both cases the input is passed in as part of the context so can you clarify why one padding strategy is better? Also in the generate function we pass input ids and an attention mask, shouldn’t the attention mask indicate what parts of the input is padding tokens and the model should not consider those as part of the context?

Bumping this :)

I meet the same problem... Even padding to the left still results in worse performance than not padding.

Could you post a code snippet so we could take a look ?

@patil-suraj Could you clarify your response above? I closed the issue previously because padding on the left works well for me but would still like to understand why that's the case.