BERT2BERT_finetuned / README.md
JulienRPA's picture
update model card README.md
1937f42
|
raw
history blame
2.01 kB
metadata
tags:
  - generated_from_trainer
metrics:
  - bleu
model-index:
  - name: BERT2BERT_finetuned
    results: []

BERT2BERT_finetuned

This model is a fine-tuned version of JulienRPA/BERT2BERT_pretrained_LC-QuAD_2.0 on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.4253
  • Bleu: 95.481
  • Em: 0.6019
  • Rm: 0.6547
  • Gen Len: 80.4676

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 64
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 2000
  • num_epochs: 300.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Bleu Em Rm Gen Len
2.9129 50.0 1000 0.3861 92.7571 0.2302 0.2422 79.2806
0.052 100.0 2000 0.3532 95.3257 0.5156 0.5635 80.6859
0.0155 150.0 3000 0.4280 94.8017 0.5659 0.6355 79.3165
0.0086 200.0 4000 0.4077 95.5885 0.5803 0.6283 80.6978
0.0051 250.0 5000 0.4169 95.5923 0.6019 0.6523 80.6523
0.0038 300.0 6000 0.4253 95.481 0.6019 0.6547 80.4676

Framework versions

  • Transformers 4.30.0.dev0
  • Pytorch 2.0.1+cu118
  • Datasets 2.12.0
  • Tokenizers 0.13.3