Edit model card

bigbird-pegasus-large-arxiv-finetuned-pubmed

This model is a fine-tuned version of google/bigbird-pegasus-large-arxiv on the pub_med_summarization_dataset dataset. It achieves the following results on the evaluation set:

  • Loss: 1.6049
  • Rouge1: 45.4807
  • Rouge2: 20.0199
  • Rougel: 28.3621
  • Rougelsum: 41.4618
  • Gen Len: 219.144

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 5
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Rouge1 Rouge2 Rougel Rougelsum Gen Len
2.594 1.0 500 1.9879 33.6364 13.5074 21.4286 29.7158 189.014
1.9146 2.0 1000 1.6494 44.0056 19.0069 27.5142 40.0492 210.528
1.7378 3.0 1500 1.6213 44.7071 19.3559 27.6806 40.6124 213.596
1.692 4.0 2000 1.6081 45.1505 19.7355 28.06 41.0108 213.674
1.6656 5.0 2500 1.6049 45.4807 20.0199 28.3621 41.4618 219.144

Framework versions

  • Transformers 4.17.0
  • Pytorch 1.9.1
  • Datasets 1.18.4
  • Tokenizers 0.11.6
Downloads last month
19

Evaluation results