---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- wmt19
metrics:
- bleu
model-index:
- name: wmt19-ende-t5-small
  results:
  - task:
      name: Sequence-to-sequence Language Modeling
      type: text2text-generation
    dataset:
      name: wmt19
      type: wmt19
      config: de-en
      split: validation
      args: de-en
    metrics:
    - name: Bleu
      type: bleu
      value: 16.085214160195623
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wmt19-ende-t5-small

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the wmt19 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.5150
- Bleu: 16.0852
- Brevity Penalty: 0.5512

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 256
- eval_batch_size: 512
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 512
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- training_steps: 10000

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Bleu    | Brevity Penalty |
|:-------------:|:-----:|:-----:|:---------------:|:-------:|:---------------:|
| 2.7369        | 0.01  | 100   | 2.0018          | 9.0851  | 0.5107          |
| 3.3896        | 0.02  | 200   | 1.9108          | 9.9970  | 0.5127          |
| 3.0442        | 0.03  | 300   | 1.8627          | 10.7670 | 0.5245          |
| 2.5136        | 0.04  | 400   | 1.8244          | 10.9280 | 0.5132          |
| 2.4092        | 0.05  | 500   | 1.7951          | 11.4717 | 0.5260          |
| 3.2441        | 0.06  | 600   | 1.7736          | 11.7350 | 0.5197          |
| 2.6997        | 0.07  | 700   | 1.7563          | 12.0741 | 0.5260          |
| 2.5072        | 0.08  | 800   | 1.7416          | 12.3735 | 0.5283          |
| 2.3788        | 0.09  | 900   | 1.7267          | 12.4288 | 0.5285          |
| 2.3533        | 0.1   | 1000  | 1.7247          | 12.4395 | 0.5249          |
| 2.2911        | 0.11  | 1100  | 1.7078          | 12.3887 | 0.5201          |
| 2.3949        | 0.12  | 1200  | 1.6997          | 12.8109 | 0.5288          |
| 2.2343        | 0.13  | 1300  | 1.6930          | 12.8213 | 0.5283          |
| 2.2525        | 0.14  | 1400  | 1.6851          | 13.1221 | 0.5285          |
| 2.2604        | 0.15  | 1500  | 1.6795          | 13.0896 | 0.5261          |
| 2.3146        | 0.16  | 1600  | 1.6723          | 13.1741 | 0.5291          |
| 2.5767        | 0.17  | 1700  | 1.6596          | 13.4224 | 0.5248          |
| 2.698         | 0.18  | 1800  | 1.6576          | 13.6733 | 0.5334          |
| 2.6416        | 0.19  | 1900  | 1.6514          | 13.7184 | 0.5350          |
| 3.0841        | 0.2   | 2000  | 1.6448          | 13.9079 | 0.5357          |
| 2.5039        | 0.21  | 2100  | 1.6375          | 13.9860 | 0.5361          |
| 2.5829        | 0.22  | 2200  | 1.6366          | 13.9246 | 0.5328          |
| 2.5332        | 0.23  | 2300  | 1.6348          | 13.4895 | 0.5209          |
| 2.5832        | 0.24  | 2400  | 1.6240          | 14.0445 | 0.5349          |
| 2.8577        | 0.25  | 2500  | 1.6182          | 14.1085 | 0.5344          |
| 2.9157        | 0.26  | 2600  | 1.6285          | 13.7982 | 0.5365          |
| 2.6758        | 0.27  | 2700  | 1.6249          | 13.8638 | 0.5392          |
| 2.0391        | 0.28  | 2800  | 1.6205          | 13.9645 | 0.5396          |
| 2.8146        | 0.29  | 2900  | 1.6210          | 14.2823 | 0.5409          |
| 2.6602        | 0.3   | 3000  | 1.6219          | 13.9663 | 0.5391          |
| 1.7745        | 0.31  | 3100  | 1.6088          | 14.4206 | 0.5413          |
| 2.3483        | 0.32  | 3200  | 1.6050          | 14.6208 | 0.5471          |
| 1.9911        | 0.33  | 3300  | 1.6004          | 14.5458 | 0.5396          |
| 1.8973        | 0.34  | 3400  | 1.5985          | 14.5387 | 0.5400          |
| 2.6956        | 0.35  | 3500  | 1.6005          | 14.7482 | 0.5458          |
| 2.322         | 0.36  | 3600  | 1.5949          | 14.7322 | 0.5448          |
| 1.5147        | 0.37  | 3700  | 1.5966          | 14.8456 | 0.5431          |
| 2.0606        | 0.38  | 3800  | 1.5899          | 14.6267 | 0.5333          |
| 3.0341        | 0.39  | 3900  | 1.5842          | 14.7705 | 0.5414          |
| 1.5069        | 0.4   | 4000  | 1.5911          | 14.6861 | 0.5372          |
| 2.339         | 0.41  | 4100  | 1.5949          | 14.6970 | 0.5481          |
| 2.5221        | 0.42  | 4200  | 1.5870          | 14.6996 | 0.5403          |
| 1.6398        | 0.43  | 4300  | 1.5790          | 14.8826 | 0.5431          |
| 2.2758        | 0.44  | 4400  | 1.5818          | 14.5580 | 0.5375          |
| 2.2622        | 0.45  | 4500  | 1.5821          | 15.0062 | 0.5428          |
| 1.3329        | 0.46  | 4600  | 1.5792          | 14.7609 | 0.5377          |
| 1.7537        | 0.47  | 4700  | 1.5744          | 15.1037 | 0.5425          |
| 2.5379        | 0.48  | 4800  | 1.5756          | 15.2684 | 0.5479          |
| 2.1236        | 0.49  | 4900  | 1.5822          | 14.8229 | 0.5478          |
| 2.9621        | 0.5   | 5000  | 1.5747          | 14.9948 | 0.5443          |
| 1.9832        | 0.51  | 5100  | 1.5838          | 14.8682 | 0.5468          |
| 1.4962        | 0.52  | 5200  | 1.5836          | 14.8094 | 0.5397          |
| 2.4318        | 0.53  | 5300  | 1.5826          | 14.8213 | 0.5422          |
| 1.9338        | 0.54  | 5400  | 1.5869          | 14.5571 | 0.5402          |
| 1.404         | 0.55  | 5500  | 1.5891          | 14.5103 | 0.5414          |
| 2.2803        | 0.56  | 5600  | 1.5864          | 14.6338 | 0.5417          |
| 2.3725        | 0.57  | 5700  | 1.5893          | 14.3405 | 0.5385          |
| 1.1436        | 0.58  | 5800  | 1.5703          | 15.3309 | 0.5457          |
| 2.1695        | 0.59  | 5900  | 1.5690          | 15.3571 | 0.5438          |
| 1.7295        | 0.6   | 6000  | 1.5653          | 15.3547 | 0.5421          |
| 1.3033        | 0.61  | 6100  | 1.5649          | 15.3084 | 0.5442          |
| 2.396         | 0.62  | 6200  | 1.5592          | 15.5594 | 0.5440          |
| 2.133         | 0.63  | 6300  | 1.5634          | 15.3689 | 0.5420          |
| 1.1775        | 0.64  | 6400  | 1.5639          | 15.4869 | 0.5389          |
| 2.0793        | 0.65  | 6500  | 1.5541          | 15.6320 | 0.5453          |
| 1.7569        | 0.66  | 6600  | 1.5588          | 15.7405 | 0.5429          |
| 1.1035        | 0.67  | 6700  | 1.5520          | 15.7011 | 0.5450          |
| 1.5799        | 0.68  | 6800  | 1.5517          | 15.9203 | 0.5490          |
| 1.7737        | 0.69  | 6900  | 1.5473          | 15.8992 | 0.5480          |
| 1.3071        | 0.7   | 7000  | 1.5491          | 15.7140 | 0.5446          |
| 2.2214        | 0.71  | 7100  | 1.5460          | 15.9360 | 0.5479          |
| 1.7848        | 0.72  | 7200  | 1.5431          | 15.9338 | 0.5490          |
| 1.1231        | 0.73  | 7300  | 1.5398          | 15.8774 | 0.5444          |
| 1.7741        | 0.74  | 7400  | 1.5399          | 15.9724 | 0.5451          |
| 1.7098        | 0.75  | 7500  | 1.5361          | 15.9098 | 0.5447          |
| 1.0787        | 0.76  | 7600  | 1.5393          | 15.9781 | 0.5457          |
| 1.9856        | 0.77  | 7700  | 1.5348          | 15.9521 | 0.5462          |
| 2.1294        | 0.78  | 7800  | 1.5345          | 16.0042 | 0.5463          |
| 1.1938        | 0.79  | 7900  | 1.5314          | 16.0554 | 0.5495          |
| 1.9579        | 0.8   | 8000  | 1.5307          | 15.9349 | 0.5482          |
| 1.844         | 0.81  | 8100  | 1.5285          | 15.8589 | 0.5448          |
| 1.1464        | 0.82  | 8200  | 1.5413          | 15.9210 | 0.5435          |
| 2.2903        | 0.83  | 8300  | 1.5230          | 16.0164 | 0.5405          |
| 2.1489        | 0.84  | 8400  | 1.5263          | 15.9423 | 0.5443          |
| 1.8138        | 0.85  | 8500  | 1.5350          | 15.8267 | 0.5464          |
| 2.4025        | 0.86  | 8600  | 1.5275          | 15.8493 | 0.5430          |
| 1.6758        | 0.87  | 8700  | 1.5206          | 15.9246 | 0.5464          |
| 1.3671        | 0.88  | 8800  | 1.5235          | 15.9662 | 0.5460          |
| 2.3341        | 0.89  | 8900  | 1.5221          | 16.0465 | 0.5456          |
| 1.8405        | 0.9   | 9000  | 1.5201          | 16.0834 | 0.5454          |
| 1.4133        | 0.91  | 9100  | 1.5250          | 15.8619 | 0.5442          |
| 2.4374        | 0.92  | 9200  | 1.5261          | 15.8174 | 0.5429          |
| 1.3627        | 0.93  | 9300  | 1.5257          | 15.7541 | 0.5450          |
| 1.5003        | 0.94  | 9400  | 1.5249          | 15.9109 | 0.5463          |
| 2.2002        | 0.95  | 9500  | 1.5252          | 15.8338 | 0.5434          |
| 2.3461        | 0.96  | 9600  | 1.5262          | 15.9195 | 0.5469          |
| 1.2607        | 0.97  | 9700  | 1.5197          | 15.8370 | 0.5459          |
| 2.3737        | 0.98  | 9800  | 1.5178          | 16.0579 | 0.5475          |
| 1.3968        | 0.99  | 9900  | 1.5132          | 16.1729 | 0.5522          |
| 1.1816        | 1.0   | 10000 | 1.5150          | 16.0852 | 0.5512          |


### Framework versions

- Transformers 4.30.2
- Pytorch 2.0.1+cu118
- Datasets 2.12.0
- Tokenizers 0.13.3