indo-t5-base

This model is a fine-tuned version of google/mt5-base on the lazarus-project/alkitab-sabda-mt dataset. It achieves the following results on the evaluation set:

• Loss: 1.5657
• Bleu: 18.7296
• Gen Len: 49.6657

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0008
• train_batch_size: 32
• eval_batch_size: 16
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 16
• total_train_batch_size: 4096
• total_eval_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• training_steps: 5000

Training results

Training Loss	Epoch	Step	Validation Loss	Bleu	Gen Len
1.5774	7.64	1000	1.5461	3.3312	18.8762
1.137	15.28	2000	1.4426	3.8148	18.8755
0.9109	22.92	3000	1.4754	3.9571	18.8752
0.7807	30.56	4000	1.5373	3.9767	18.8761
0.7288	38.2	5000	1.5657	3.9838	18.8778

Framework versions

• Transformers 4.27.1
• Pytorch 2.0.0+cu117
• Datasets 2.9.0
• Tokenizers 0.13.2