README.md

metadata

tags:
  - generated_from_trainer
metrics:
  - rouge
model-index:
  - name: ViT-Bert_Mimic
    results: []

ViT-Bert_Mimic

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1692
• Rouge1: 20.3071
• Rouge2: 17.0769
• Rougel: 20.3425
• Rougelsum: 20.2544
• Gen Len: 17.62

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 15

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
No log	1.0	250	0.1710	0.0	0.0	0.0	0.0	20.0
0.3208	2.0	500	0.1424	0.0	0.0	0.0	0.0	20.0
0.3208	3.0	750	0.1396	28.6163	23.3316	28.7485	28.5768	15.76
0.087	4.0	1000	0.1349	15.1174	12.3333	15.1103	15.0938	18.16
0.087	5.0	1250	0.1373	1.5	0.6667	1.5	1.5	19.8
0.0621	6.0	1500	0.1418	29.7634	24.3915	29.9052	29.7154	16.0
0.0621	7.0	1750	0.1453	1.0	1.0	1.0	1.0	19.92
0.0415	8.0	2000	0.1504	6.75	5.6667	6.75	6.75	19.22
0.0415	9.0	2250	0.1547	18.9444	16.7436	19.1202	18.9032	18.06
0.0254	10.0	2500	0.1591	19.2342	15.6667	19.2782	19.1123	17.44
0.0254	11.0	2750	0.1601	19.9154	17.3654	20.011	19.8499	17.91
0.0147	12.0	3000	0.1647	16.9972	13.0769	16.9972	16.9154	17.96
0.0147	13.0	3250	0.1676	22.2859	16.9093	22.4548	22.1609	16.66
0.0094	14.0	3500	0.1684	20.2714	16.7308	20.4544	20.2507	17.47
0.0094	15.0	3750	0.1692	20.3071	17.0769	20.3425	20.2544	17.62

Framework versions

• Transformers 4.37.1
• Pytorch 1.13.1+cu117
• Datasets 2.15.0
• Tokenizers 0.15.1