Fail to reproduce tf2 bert-large f1-score on SQuADv1.1
Author: zhuango
Pull request: no
Labels: name: bug

### System Info

transformers=4.20.1, python=3.7, tensorflow-gpu=2.9.1.

### Who can help?

@Rocketknight1 @sgugger @patil-suraj 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Download bert-large tf2 pretrained models, .h5 file from [huggingface](https://huggingface.co/bert-large-uncased/tree/main).
2. Run [pytorch question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) for reference. My script is:
```
python run_qa.py \
  --model_name_or_path $BERT_DIR \
  --dataset_name $SQUAD_DIR \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 128 \
  --doc_stride 48 \
  --output_dir $OUTPUT \
  --save_steps 10000 \
  --overwrite_cache \

```
I got an f1-score **90.3953%**. Note that the pretrained model are loaded from tf2 .h5 checkpoints.

3. Run [tf2 question-answering example](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) with the same setting of pytorch example. My script is:
```
python run_qa.py \
  --model_name_or_path $BERT_DIR \
  --dataset_name $SQUAD_DIR \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 128 \
  --doc_stride 48 \
  --output_dir $OUTPUT \
  --save_steps 10000 \
  --overwrite_cache \
```
I only got an f1-score **88.5672%**, which is much lower than expected and pytorch results (90.3953%).


### Expected behavior

TF2 question-answering example should achieve similar f1-score with results of corresponding pytroch example.
Or you guys could provide script examples that achieves the target F1-score. Thanks.

Comments:
Hi @zhuango, it's possible there's a bug here, but there is also always some variation between runs. Can you run the training a couple of times for both PyTorch and TensorFlow and report the range of F1-scores you get for both? 

@Rocketknight1 Sure, I willl try more runs with exactly the same hyper-parameters. But the hyper-parameter setting I showed above is the one that can achieve the highest f1-score. I have run the tf2 bert-large fine-tuning on SQuADv1.1 with different hyper-parameter setting combinations, like total batchsize=12, 24/warmup_rate=0, 0.1/learning rate=3e-5, 1e-5/sequence length=384, 128/doc_stride=128, 48/Adam, AdamW optimizers. All the results are lower than 87% except the one I showed above. 
So I doubt that maybe I miss something in the hyper-parameter setting if there is no bug in the example. And Is it pissible that you share the setting that you guys used to reproduced the F1-score?
Note that I modified the example code to make sure the model's inputs are complete, Please have a check on [#18223 ](https://github.com/huggingface/transformers/issues/18223) for details.

Hi @zhuango - I just saw your other issue, I don't know how I missed it the first time. You're completely right - that example is hardcoding the input columns to the model, which is totally wrong and will degrade performance for models with `token_type_ids` inputs. I'll work on a PR and let you know when it's ready.

@Rocketknight1 Thanks, I tried more runs with the same setting (and fixed #18223 ) and still got 88% like F1-socre. 

But, I tried a linear decay learning rate scheduler plus 0.1 * total training steps warmup and AdamW optimizers (from tensorflow_addons), and I finally got **90.247%** F1-score. Here is my optimizers:
```
            # Calculate the total trainnig steps.
            total_steps = int(training_args.num_train_epochs * ( len(processed_datasets["train"]) / (training_args._n_gpu * training_args.per_device_train_batch_size) ))

            # Linear decay learning rate.
            linear_decay = tf.keras.optimizers.schedules.PolynomialDecay(
                training_args.learning_rate,
                total_steps,
                end_learning_rate=0.0,
                power=1.0,
                cycle=False,
                name=None
            )

            # 0.1 * total_steps warmup.
            warmup_schedule = tfm.optimization.lr_schedule.LinearWarmup(
                warmup_learning_rate = 0,
                after_warmup_lr_sched = linear_decay,
                warmup_steps = total_steps * 0.1,
            )

            # AdamW optimizers with no weight decay.
            optimizer = tfa.optimizers.AdamW(
                weight_decay=0.0,
                learning_rate=warmup_schedule,
                beta_1=training_args.adam_beta1,
                beta_2=training_args.adam_beta2,
                epsilon=1e-06,
                amsgrad=False,
                name='AdamW',
                clipnorm=training_args.max_grad_norm,
            )
```
Maybe you would like to try the above setting with the tf example code.

@zhuango Thank you for that! It's reminding me that all of our examples need to be overhauled, honestly. I'm going to make a PR to modernize all of them - I'll ping you when it's ready if you want to review it.


@Rocketknight1 Sure, Thank you.

Hi @zhuango, I've [made a PR](https://github.com/huggingface/transformers/pull/18451) to rewrite all of our examples. If you get a chance, please confirm that the SQUaD performance using the new examples matches the PyTorch performance!