IBert Problems of hugging face pretrained
Author: CSuperlei
Pull request: no

### Pretrained the IBert
I want to test IBERT's, and I have done exactly what is said in https://huggingface.co/kssteven/ibert-roberta-base. For the quantization part, when I set quant_mode to true and run the evaluation again, I get a much low accuracy model.  What am I doing wrong?

Comments:
cc @kssteven418 

{                                  
  "_name_or_path": "kssteven/ibert-roberta-base",       
  "architectures": [               
    "IBertForSequenceClassification"                    
  ],                               
  "attention_probs_dropout_prob": 0.1,                  
  "bos_token_id": 0,               
  "eos_token_id": 2,               
  "finetuning_task": "mrpc",       
  "force_dequant": "none",         
  "hidden_act": "gelu",            
  "hidden_dropout_prob": 0.1,      
  "hidden_size": 768,              
  "initializer_range": 0.02,       
  "intermediate_size": 3072,       
  "layer_norm_eps": 1e-05,         
  "max_position_embeddings": 514,  
  "model_type": "ibert",           
  "num_attention_heads": 12,       
  "num_hidden_layers": 12,         
  "pad_token_id": 1,               
  "position_embedding_type": "absolute",                
  "quant_mode": true,             
  "tokenizer_class": "RobertaTokenizer",                
  "transformers_version": "4.4.0.dev0",                 
  "type_vocab_size": 1,            
  "vocab_size": 50265              
}                   

==================================
I set teh quant_mode:true and processed the "Integer-only finetuning".
The result shows:
SequenceClassifierOutput(loss=None, logits=tensor([[ 0.0006, -0.0006]], grad_fn=<AddmmBackward0>), hidden_states=None, attentions=None)
tensor([[0.5003, 0.4997]], grad_fn=<SoftmaxBackward0>)

The model has bad performance
==================================



@kssteven418

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I met the same issue on squad dataset.

## Poor accuracy after finetuning

I follow the exact instruction in the [model card ](https://huggingface.co/kssteven/ibert-roberta-base)
`python examples/text-classification/run_glue.py \
         --model_name_or_path kssteven/ibert-roberta-base \
         --task_name MRPC \
         --do_eval \
         --do_train \
         --evaluation_strategy epoch \
         --max_seq_length 128 \
         --per_device_train_batch_size 32 \
         --save_steps 115 \
         --learning_rate 2e-5 \
         --num_train_epochs 10 \
         --output_dir $OUTPUT_DIR`
For finetuning on MRPC and then

`python examples/pytorch/text-classification/run_glue.py \
    --model_name_or_path ../output_dir/checkpoint-575/  \
    --task_name MRPC  \
    --do_eval        \
    --do_train        \
    --evaluation_strategy epoch    \
    --max_seq_length 128    \
    --per_device_train_batch_size 8        \
    --save_steps 115      \
    --learning_rate 1e-6      \
    --num_train_epochs 10      \
    --output_dir $OUTPUT_DIR`

for quantization-aware finetuning. Note that I lowered batch size since after enabling quantization my GPU memory could not handle the batch size of 32...

Result is horrible accuracy:
 | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1  | Combined Score |
 |:-------------:|:-----:|:----:|:---------------:|:--------:|:---:|:--------------:|
 | No log        | 1.0   | 230  | 0.6933          | 0.3162   | 0.0 | 0.1581         |