--- language: - de tags: - question-generation - german - text2text-generation - generated_from_trainer datasets: - lmqg/qg_dequad metrics: - bleu4 - f1 - rouge - exact_match model-index: - name: german-jeopardy-longt5-large results: - task: name: Sequence-to-sequence Language Modeling type: text2text-generation dataset: name: lmqg/qg_dequad type: default args: default metrics: - name: BLEU-4 type: bleu4 value: 9.50 - name: F1 type: f1 value: 32.03 - name: ROUGE-1 type: rouge1 value: 32.79 - name: ROUGE-2 type: rouge2 value: 14.95 - name: ROUGE-L type: rougel value: 31.56 - name: ROUGE-Lsum type: rougelsum value: 31.57 - name: Exact Match type: exact_match value: 1.36 --- # german-jeopardy-longt5-large-1k-64-constant This model is a fine-tuned version of [google/long-t5-tglobal-large](https://huggingface.co/google/long-t5-tglobal-large) on the [lmqg/qg_dequad](https://huggingface.co/datasets/lmqg/qg_dequad) dataset. It achieves the following results on the evaluation set: - Loss: 2.5907 - Brevity Penalty: 0.9367 - System Length: 19517 - Reference Length: 20793 - ROUGE-1: 32.79 - ROUGE-2: 14.95 - ROUGE-L: 31.56 - ROUGE-Lsum: 31.57 - Exact Match: 1.36 - BLEU: 9.50 - F1: 32.03 ## Model description See [google/long-t5-tglobal-large](https://huggingface.co/google/long-t5-tglobal-large) for more information about the model architecture. The model was trained on a single NVIDIA RTX 3090 GPU with 24GB of VRAM. ## Intended uses & limitations This model can be used for question generation on German text. ## Training and evaluation data See [lmqg/qg_dequad](https://huggingface.co/datasets/lmqg/qg_dequad). ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0001 - train_batch_size: 2 - eval_batch_size: 2 - seed: 7 - gradient_accumulation_steps: 32 - total_train_batch_size: 64 - optimizer: Adafactor - lr_scheduler_type: constant - num_epochs: 20 ### Training results | Training Loss | Epoch | Step | Validation Loss | Counts 1 | Counts 2 | Counts 3 | Counts 4 | Totals 1 | Totals 2 | Totals 3 | Totals 4 | Precisions 1 | Precisions 2 | Precisions 3 | Precisions 4 | Brevity Penalty | System Length | Reference Length | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-Lsum | Exact Match | BLEU | Mean Generated Length | F1 | |:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:------------:|:------------:|:------------:|:------------:|:---------------:|:-------------:|:----------------:|:-------:|:-------:|:-------:|:----------:|:-----------:|:-------:|:---------------------:|:------:| | 6.5987 | 1.0 | 145 | 5.0696 | 3804 | 134 | 2 | 0 | 22913 | 20709 | 18505 | 16301 | 16.6019 | 0.6471 | 0.0108 | 0.0031 | 1.0 | 22913 | 21250 | 0.0783 | 0.007 | 0.0769 | 0.0768 | 0.0 | 0.1374 | 16.2899 | 0.0814 | | 4.7443 | 2.0 | 291 | 4.2270 | 4022 | 188 | 20 | 0 | 17366 | 15162 | 12958 | 10754 | 23.1602 | 1.2399 | 0.1543 | 0.0046 | 0.7996 | 17366 | 21250 | 0.1028 | 0.012 | 0.0991 | 0.099 | 0.0 | 0.303 | 12.9038 | 0.1073 | | 4.1412 | 3.0 | 436 | 3.7838 | 3723 | 187 | 26 | 2 | 16515 | 14311 | 12107 | 9903 | 22.5431 | 1.3067 | 0.2148 | 0.0202 | 0.7507 | 16515 | 21250 | 0.0899 | 0.0124 | 0.0886 | 0.0884 | 0.0 | 0.4488 | 12.4769 | 0.0938 | | 3.6791 | 4.0 | 582 | 3.4246 | 4576 | 549 | 134 | 26 | 21871 | 19667 | 17463 | 15259 | 20.9227 | 2.7915 | 0.7673 | 0.1704 | 1.0 | 21871 | 21250 | 0.1259 | 0.0296 | 0.1204 | 0.1201 | 0.0 | 1.6623 | 14.5676 | 0.1323 | | 3.3523 | 5.0 | 727 | 3.1723 | 4900 | 796 | 210 | 41 | 19389 | 17185 | 14981 | 12777 | 25.2721 | 4.6319 | 1.4018 | 0.3209 | 0.9085 | 19389 | 21250 | 0.1542 | 0.0449 | 0.1486 | 0.1484 | 0.0005 | 2.4472 | 14.3943 | 0.1585 | | 3.0161 | 6.0 | 873 | 2.9268 | 5633 | 1182 | 390 | 111 | 19045 | 16841 | 14637 | 12433 | 29.5773 | 7.0186 | 2.6645 | 0.8928 | 0.8907 | 19045 | 21250 | 0.204 | 0.069 | 0.196 | 0.1961 | 0.0045 | 4.1987 | 14.5789 | 0.2074 | | 2.7639 | 7.0 | 1018 | 2.7601 | 6100 | 1461 | 499 | 165 | 17924 | 15720 | 13516 | 11312 | 34.0326 | 9.2939 | 3.6919 | 1.4586 | 0.8306 | 17924 | 21250 | 0.2409 | 0.0885 | 0.2332 | 0.2331 | 0.0073 | 5.3362 | 13.8553 | 0.2431 | | 2.5036 | 8.0 | 1164 | 2.5729 | 6765 | 1845 | 701 | 273 | 20179 | 17975 | 15771 | 13567 | 33.525 | 10.2643 | 4.4449 | 2.0122 | 0.9483 | 20179 | 21250 | 0.2682 | 0.1079 | 0.2589 | 0.259 | 0.0059 | 7.0633 | 15.7232 | 0.2689 | | 2.307 | 8.99 | 1309 | 2.4637 | 7018 | 2047 | 826 | 348 | 19054 | 16850 | 14646 | 12442 | 36.8322 | 12.1484 | 5.6398 | 2.797 | 0.8911 | 19054 | 21250 | 0.2907 | 0.1218 | 0.2799 | 0.2798 | 0.0095 | 8.1681 | 14.8076 | 0.2907 | | 2.1012 | 10.0 | 1455 | 2.3614 | 7147 | 2127 | 883 | 389 | 18473 | 16269 | 14065 | 11861 | 38.6889 | 13.0739 | 6.278 | 3.2797 | 0.8604 | 18473 | 21250 | 0.3003 | 0.1275 | 0.289 | 0.2888 | 0.0118 | 8.6921 | 14.2736 | 0.3008 | | 1.9538 | 10.99 | 1600 | 2.2980 | 7481 | 2339 | 997 | 459 | 18524 | 16320 | 14116 | 11912 | 40.3854 | 14.3321 | 7.0629 | 3.8533 | 0.8632 | 18524 | 21250 | 0.3192 | 0.1423 | 0.3064 | 0.3068 | 0.0127 | 9.67 | 14.3757 | 0.3167 | | 1.7909 | 12.0 | 1746 | 2.2389 | 7675 | 2546 | 1144 | 546 | 18849 | 16645 | 14441 | 12237 | 40.7183 | 15.2959 | 7.9219 | 4.4619 | 0.8804 | 18849 | 21250 | 0.3299 | 0.1528 | 0.3174 | 0.3175 | 0.015 | 10.724 | 14.583 | 0.3279 | | 1.6691 | 12.99 | 1891 | 2.1813 | 7858 | 2635 | 1179 | 576 | 18643 | 16439 | 14235 | 12031 | 42.1499 | 16.029 | 8.2824 | 4.7876 | 0.8695 | 18643 | 21250 | 0.344 | 0.1626 | 0.33 | 0.33 | 0.0163 | 11.1241 | 14.3848 | 0.3395 | | 1.5361 | 14.0 | 2037 | 2.1546 | 8016 | 2729 | 1249 | 606 | 18754 | 16550 | 14346 | 12142 | 42.7429 | 16.4894 | 8.7063 | 4.9909 | 0.8754 | 18754 | 21250 | 0.3494 | 0.1664 | 0.3349 | 0.3351 | 0.0163 | 11.5803 | 14.564 | 0.3462 | | 1.4365 | 14.99 | 2182 | 2.1358 | 8112 | 2839 | 1316 | 647 | 18390 | 16186 | 13982 | 11778 | 44.1109 | 17.5398 | 9.4121 | 5.4933 | 0.856 | 18390 | 21250 | 0.3581 | 0.1761 | 0.3448 | 0.3448 | 0.02 | 12.1055 | 14.1656 | 0.3538 | | 1.3263 | 16.0 | 2328 | 2.1190 | 8381 | 2990 | 1430 | 731 | 18892 | 16688 | 14484 | 12280 | 44.3627 | 17.9171 | 9.873 | 5.9528 | 0.8827 | 18892 | 21250 | 0.3681 | 0.1831 | 0.3532 | 0.3534 | 0.0209 | 12.9765 | 14.5445 | 0.363 | | 1.2329 | 17.0 | 2474 | 2.1202 | 8449 | 3101 | 1520 | 786 | 18612 | 16408 | 14204 | 12000 | 45.3954 | 18.8993 | 10.7012 | 6.55 | 0.8678 | 18612 | 21250 | 0.3743 | 0.1901 | 0.3603 | 0.3603 | 0.0227 | 13.5903 | 14.1779 | 0.3692 | | 1.1557 | 18.0 | 2619 | 2.1282 | 8406 | 3154 | 1558 | 804 | 17958 | 15754 | 13550 | 11346 | 46.8092 | 20.0203 | 11.4982 | 7.0862 | 0.8325 | 17958 | 21250 | 0.3761 | 0.194 | 0.3633 | 0.3636 | 0.0277 | 13.8388 | 13.677 | 0.371 | | 1.0658 | 19.0 | 2765 | 2.1232 | 8614 | 3241 | 1610 | 839 | 18955 | 16751 | 14547 | 12343 | 45.4445 | 19.3481 | 11.0676 | 6.7974 | 0.886 | 18955 | 21250 | 0.3803 | 0.196 | 0.3654 | 0.3656 | 0.0272 | 14.2084 | 14.3816 | 0.3749 | | 0.9944 | 19.93 | 2900 | 2.1203 | 8658 | 3273 | 1625 | 859 | 18853 | 16649 | 14445 | 12241 | 45.9237 | 19.6588 | 11.2496 | 7.0174 | 0.8806 | 18853 | 21250 | 0.3833 | 0.1977 | 0.369 | 0.3691 | 0.0268 | 14.3883 | 14.2881 | 0.3775 | ### Framework versions - Transformers 4.32.1 - Pytorch 2.1.0 - Datasets 2.12.0 - Tokenizers 0.13.3