t5-small-finetuned-pubmed

This model is a fine-tuned version of t5-small on a truncated PubMed Summarization dataset. It achieves the following results on the evaluation set:

• Loss: 2.7252
• Rouge1: 19.4457
• Rouge2: 3.125
• Rougel: 18.3168
• Rougelsum: 18.5625

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5.6e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 8

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum
3.2735	1.0	13	2.9820	18.745	3.7918	15.7876	15.8512
3.0428	2.0	26	2.8828	17.953	2.5	15.49	15.468
2.6259	3.0	39	2.8283	21.5532	5.9278	19.7523	19.9232
3.0795	4.0	52	2.7910	20.9244	5.9278	19.8685	20.0181
2.8276	5.0	65	2.7613	20.6403	3.125	18.0574	18.2227
2.64	6.0	78	2.7404	19.4457	3.125	18.3168	18.5625
2.5525	7.0	91	2.7286	19.4457	3.125	18.3168	18.5625
2.4951	8.0	104	2.7252	19.4457	3.125	18.3168	18.5625

Framework versions

• Transformers 4.25.1
• Pytorch 1.13.0
• Datasets 2.8.0
• Tokenizers 0.13.2