results

This model is a fine-tuned version of LA1512/PubMed-fine-tune on the pubmed-summarization dataset. It achieves the following results on the evaluation set:

• Loss: 3.6196
• Rouge1: 40.7402
• Rouge2: 16.1978
• Rougel: 24.4278
• Rougelsum: 36.5282
• Gen Len: 179.6185

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 3
• label_smoothing_factor: 0.1

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
3.6132	1.0	2500	3.6766	40.5092	15.7678	24.1228	36.3318	183.7205
3.5939	2.0	5000	3.6276	40.7583	16.1779	24.4375	36.5537	181.4365
3.5419	3.0	7500	3.6196	40.7402	16.1978	24.4278	36.5282	179.6185

Framework versions

• Transformers 4.39.3
• Pytorch 2.1.2
• Datasets 2.18.0
• Tokenizers 0.15.2