--- language: - de tags: - question-generation - german - text2text-generation - generated_from_trainer datasets: - lmqg/qg_dequad metrics: - bleu4 - f1 - rouge - exact_match model-index: - name: german-jeopardy-longt5-base-128 results: - task: name: Sequence-to-sequence Language Modeling type: text2text-generation dataset: name: lmqg/qg_dequad type: default args: default metrics: - name: BLEU-4 type: bleu4 value: 10.73 - name: F1 type: f1 value: 34.55 - name: ROUGE-1 type: rouge1 value: 35.34 - name: ROUGE-2 type: rouge2 value: 16.82 - name: ROUGE-L type: rougel value: 34.13 - name: ROUGE-Lsum type: rougelsum value: 34.14 - name: Exact Match type: exact_match value: 1.41 --- # german-jeopardy-longt5-base-128 This model is a fine-tuned version of [google/long-t5-tglobal-base](https://huggingface.co/google/long-t5-tglobal-base) on the [lmqg/qg_dequad](https://huggingface.co/datasets/lmqg/qg_dequad) dataset. It achieves the following results on the evaluation set: - Loss: 1.8010 - Brevity Penalty: 0.8577 - System Length: 18026 - Reference Length: 20793 - ROUGE-1: 35.34 - ROUGE-2: 16.82 - ROUGE-L: 34.13 - ROUGE-Lsum: 34.14 - Exact Match: 1.41 - BLEU: 10.73 - F1: 34.55 ## Model description See [google/long-t5-tglobal-base](https://huggingface.co/google/long-t5-tglobal-base) for more information about the model architecture. The model was trained on a single NVIDIA RTX 3090 GPU with 24GB of VRAM. ## Intended uses & limitations This model can be used for question generation on German text. ## Training and evaluation data See [lmqg/qg_dequad](https://huggingface.co/datasets/lmqg/qg_dequad). ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0001 - train_batch_size: 8 - eval_batch_size: 4 - seed: 7 - gradient_accumulation_steps: 16 - total_train_batch_size: 128 - optimizer: Adafactor - lr_scheduler_type: constant - num_epochs: 20 ### Training results | Training Loss | Epoch | Step | Validation Loss | Counts 1 | Counts 2 | Counts 3 | Counts 4 | Totals 1 | Totals 2 | Totals 3 | Totals 4 | Precisions 1 | Precisions 2 | Precisions 3 | Precisions 4 | Brevity Penalty | System Length | Reference Length | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-Lsum | Exact Match | BLEU | Mean Generated Length | F1 | |:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:------------:|:------------:|:------------:|:------------:|:---------------:|:-------------:|:----------------:|:-------:|:-------:|:-------:|:----------:|:-----------:|:-------:|:---------------------:|:------:| | 3.458 | 0.99 | 72 | 2.3696 | 5618 | 1383 | 463 | 116 | 15080 | 12876 | 10672 | 8468 | 37.2546 | 10.7409 | 4.3385 | 1.3699 | 0.6642 | 15080 | 21250 | 0.2266 | 0.0841 | 0.2197 | 0.2196 | 0.0005 | 4.6384 | 11.3013 | 0.2226 | | 2.7548 | 1.99 | 145 | 2.1310 | 6361 | 1807 | 700 | 254 | 16130 | 13926 | 11722 | 9518 | 39.4358 | 12.9757 | 5.9717 | 2.6686 | 0.728 | 16130 | 21250 | 0.2706 | 0.1122 | 0.2596 | 0.2596 | 0.0036 | 6.9183 | 12.206 | 0.2635 | | 2.5084 | 2.99 | 218 | 2.0244 | 6758 | 2001 | 780 | 285 | 16871 | 14667 | 12463 | 10259 | 40.0569 | 13.6429 | 6.2585 | 2.778 | 0.7714 | 16871 | 21250 | 0.2888 | 0.1258 | 0.2766 | 0.2767 | 0.0045 | 7.616 | 12.8825 | 0.2832 | | 2.3562 | 4.0 | 291 | 1.9501 | 7011 | 2193 | 908 | 360 | 16796 | 14592 | 12388 | 10184 | 41.7421 | 15.0288 | 7.3297 | 3.535 | 0.7671 | 16796 | 21250 | 0.303 | 0.1375 | 0.2892 | 0.2894 | 0.0077 | 8.6611 | 12.9142 | 0.2978 | | 2.2383 | 5.0 | 364 | 1.8874 | 7245 | 2386 | 1015 | 435 | 16708 | 14504 | 12300 | 10096 | 43.3625 | 16.4506 | 8.252 | 4.3086 | 0.762 | 16708 | 21250 | 0.3198 | 0.1498 | 0.3077 | 0.3079 | 0.0113 | 9.6159 | 12.8417 | 0.3155 | | 2.1576 | 5.99 | 436 | 1.8593 | 7378 | 2382 | 997 | 429 | 17014 | 14810 | 12606 | 10402 | 43.3643 | 16.0837 | 7.9089 | 4.1242 | 0.7796 | 17014 | 21250 | 0.326 | 0.1497 | 0.3132 | 0.3132 | 0.0109 | 9.5745 | 13.2187 | 0.3215 | | 2.0356 | 6.99 | 509 | 1.8133 | 7570 | 2520 | 1097 | 482 | 16999 | 14795 | 12591 | 10387 | 44.532 | 17.0328 | 8.7126 | 4.6404 | 0.7787 | 16999 | 21250 | 0.3384 | 0.158 | 0.3258 | 0.3257 | 0.0123 | 10.3053 | 13.0368 | 0.3339 | | 1.9575 | 7.99 | 582 | 1.7856 | 7764 | 2637 | 1175 | 545 | 17379 | 15175 | 12971 | 10767 | 44.6746 | 17.3773 | 9.0587 | 5.0618 | 0.8003 | 17379 | 21250 | 0.345 | 0.1625 | 0.3322 | 0.3324 | 0.0136 | 10.993 | 13.4719 | 0.3407 | | 1.8889 | 9.0 | 655 | 1.7666 | 7766 | 2644 | 1184 | 532 | 17102 | 14898 | 12694 | 10490 | 45.4099 | 17.7473 | 9.3272 | 5.0715 | 0.7846 | 17102 | 21250 | 0.3487 | 0.1636 | 0.3348 | 0.335 | 0.0123 | 10.9637 | 13.2164 | 0.3438 | | 1.8201 | 10.0 | 728 | 1.7415 | 7737 | 2680 | 1238 | 587 | 17156 | 14952 | 12748 | 10544 | 45.0979 | 17.924 | 9.7113 | 5.5671 | 0.7877 | 17156 | 21250 | 0.3453 | 0.1666 | 0.3332 | 0.3333 | 0.0163 | 11.3891 | 13.1388 | 0.3406 | | 1.7882 | 10.99 | 800 | 1.7331 | 7859 | 2722 | 1241 | 572 | 17364 | 15160 | 12956 | 10752 | 45.2603 | 17.9551 | 9.5786 | 5.3199 | 0.7995 | 17364 | 21250 | 0.3524 | 0.1673 | 0.3387 | 0.3385 | 0.0145 | 11.4047 | 13.4052 | 0.3473 | | 1.7095 | 11.99 | 873 | 1.7194 | 7968 | 2783 | 1292 | 625 | 17467 | 15263 | 13059 | 10855 | 45.6175 | 18.2336 | 9.8936 | 5.7577 | 0.8053 | 17467 | 21250 | 0.3547 | 0.1708 | 0.3418 | 0.3414 | 0.0154 | 11.8807 | 13.4437 | 0.3495 | | 1.6619 | 12.99 | 946 | 1.7032 | 8011 | 2796 | 1286 | 604 | 17433 | 15229 | 13025 | 10821 | 45.9531 | 18.3597 | 9.8733 | 5.5817 | 0.8034 | 17433 | 21250 | 0.3584 | 0.1736 | 0.3454 | 0.3454 | 0.0154 | 11.7968 | 13.4964 | 0.3526 | | 1.6103 | 13.99 | 1019 | 1.7028 | 8154 | 2891 | 1347 | 636 | 17665 | 15461 | 13257 | 11053 | 46.1591 | 18.6987 | 10.1607 | 5.7541 | 0.8163 | 17665 | 21250 | 0.3659 | 0.1795 | 0.3509 | 0.3508 | 0.015 | 12.235 | 13.7223 | 0.3602 | | 1.565 | 15.0 | 1092 | 1.6955 | 8135 | 2897 | 1362 | 665 | 17530 | 15326 | 13122 | 10918 | 46.4062 | 18.9025 | 10.3795 | 6.0909 | 0.8088 | 17530 | 21250 | 0.3668 | 0.1808 | 0.3518 | 0.3516 | 0.02 | 12.4116 | 13.6107 | 0.3603 | | 1.522 | 16.0 | 1165 | 1.6793 | 8271 | 2982 | 1414 | 697 | 17946 | 15742 | 13538 | 11334 | 46.0883 | 18.943 | 10.4447 | 6.1496 | 0.8318 | 17946 | 21250 | 0.3695 | 0.1828 | 0.354 | 0.354 | 0.0191 | 12.8008 | 13.9192 | 0.3632 | | 1.5022 | 16.99 | 1237 | 1.6849 | 8244 | 2967 | 1392 | 680 | 17510 | 15306 | 13102 | 10898 | 47.0817 | 19.3846 | 10.6243 | 6.2397 | 0.8077 | 17510 | 21250 | 0.3728 | 0.184 | 0.3569 | 0.3569 | 0.0191 | 12.6672 | 13.6243 | 0.366 | | 1.4359 | 17.99 | 1310 | 1.6862 | 8328 | 3050 | 1448 | 717 | 17873 | 15669 | 13465 | 11261 | 46.5954 | 19.4652 | 10.7538 | 6.3671 | 0.8278 | 17873 | 21250 | 0.3742 | 0.1866 | 0.3582 | 0.3583 | 0.0181 | 13.0683 | 13.7255 | 0.3671 | | 1.3994 | 18.99 | 1383 | 1.6775 | 8272 | 2998 | 1417 | 704 | 17645 | 15441 | 13237 | 11033 | 46.8801 | 19.4158 | 10.7048 | 6.3809 | 0.8152 | 17645 | 21250 | 0.3739 | 0.1866 | 0.3583 | 0.3581 | 0.0213 | 12.8728 | 13.6956 | 0.3673 | | 1.3609 | 19.78 | 1440 | 1.6884 | 8347 | 3062 | 1465 | 723 | 17823 | 15619 | 13415 | 11211 | 46.8327 | 19.6043 | 10.9206 | 6.449 | 0.8251 | 17823 | 21250 | 0.3761 | 0.1886 | 0.3601 | 0.3596 | 0.0204 | 13.1569 | 13.7328 | 0.3692 | ### Framework versions - Transformers 4.32.1 - Pytorch 2.1.0 - Datasets 2.12.0 - Tokenizers 0.13.3