---
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: ViT-Bert_Mimic
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# ViT-Bert_Mimic

This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1305
- Rouge1: 34.725
- Rouge2: 21.4916
- Rougel: 33.3614
- Rougelsum: 34.1142
- Gen Len: 20.706

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
|:-------------:|:-----:|:------:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| 0.0684        | 1.0   | 7500   | 0.0752          | 34.4312 | 25.586  | 34.2067 | 34.2816   | 14.065  |
| 0.0626        | 2.0   | 15000  | 0.0694          | 38.0498 | 26.9882 | 37.2064 | 37.6682   | 19.492  |
| 0.0599        | 3.0   | 22500  | 0.0676          | 37.9403 | 26.7796 | 37.0514 | 37.571    | 21.805  |
| 0.054         | 4.0   | 30000  | 0.0661          | 38.1215 | 26.8065 | 37.3608 | 37.7763   | 18.883  |
| 0.0484        | 5.0   | 37500  | 0.0658          | 39.0689 | 27.489  | 38.0601 | 38.8175   | 20.556  |
| 0.043         | 6.0   | 45000  | 0.0679          | 38.5537 | 26.6503 | 37.4722 | 38.1314   | 20.994  |
| 0.0378        | 7.0   | 52500  | 0.0701          | 37.8821 | 26.1994 | 36.7872 | 37.4123   | 19.978  |
| 0.0324        | 8.0   | 60000  | 0.0741          | 38.5791 | 26.2187 | 37.3411 | 38.0767   | 21.761  |
| 0.0269        | 9.0   | 67500  | 0.0787          | 36.2698 | 24.3513 | 35.1553 | 35.7864   | 20.512  |
| 0.0199        | 10.0  | 75000  | 0.0848          | 34.8266 | 22.0111 | 33.591  | 34.3348   | 19.67   |
| 0.0158        | 11.0  | 82500  | 0.0921          | 34.5083 | 21.5876 | 33.273  | 34.0396   | 20.663  |
| 0.0114        | 12.0  | 90000  | 0.0990          | 33.6601 | 20.3509 | 32.3799 | 33.1785   | 21.574  |
| 0.0078        | 13.0  | 97500  | 0.1057          | 33.5222 | 20.262  | 32.3084 | 33.0449   | 20.7    |
| 0.0057        | 14.0  | 105000 | 0.1122          | 32.9482 | 19.0875 | 31.6809 | 32.4176   | 21.562  |
| 0.0037        | 15.0  | 112500 | 0.1172          | 33.2572 | 19.0712 | 31.8675 | 32.7193   | 21.432  |
| 0.0027        | 16.0  | 120000 | 0.1215          | 34.0583 | 20.5815 | 32.5961 | 33.4699   | 21.379  |
| 0.0019        | 17.0  | 127500 | 0.1257          | 34.3046 | 21.1929 | 33.0026 | 33.6992   | 20.687  |
| 0.0013        | 18.0  | 135000 | 0.1280          | 34.9621 | 21.8578 | 33.6017 | 34.3908   | 21.249  |
| 0.001         | 19.0  | 142500 | 0.1298          | 35.1328 | 21.8242 | 33.7634 | 34.5288   | 20.567  |
| 0.0007        | 20.0  | 150000 | 0.1305          | 34.725  | 21.4916 | 33.3614 | 34.1142   | 20.706  |


### Framework versions

- Transformers 4.37.1
- Pytorch 1.13.1+cu117
- Datasets 2.15.0
- Tokenizers 0.15.1