pegasus-samsum-nlp-with-transformers-ch06

This model is a fine-tuned version of google/pegasus-cnn_dailymail on the SAMSum dataset.

It achieves the following results on the evaluation set:

• Loss: 1.4839

It achieves the following ROUGE scores on the test set:

• rouge1: 0.555556
• rouge2: 0.230769
• rougeL: 0.518519
• rougeLsum: 0.518519

Quick human evaluation of summarization quality: the results are generally good, after visual inspection of the summaries generated on test set conversations. However it seems some entities/attributions are incorrect (saw an example where model confuses peoples' roles in multi-person chat)

Model description

PEGASUS doc can be found here: https://huggingface.co/docs/transformers/model_doc/pegasus

Intended uses & limitations

This model was trained while studying the NLP With Transformers book; it is not intended to be used for any real applications.

Training and evaluation data

The finetuning data is the SAMSum dataset only.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.6592	0.54	500	1.4839

Framework versions

• Transformers 4.39.3
• Pytorch 2.1.2
• Datasets 2.18.0
• Tokenizers 0.15.2