--- language: - de tags: - question-generation - german - text2text-generation - generated_from_trainer datasets: - lmqg/qg_dequad metrics: - bleu4 - f1 - rouge - exact_match model-index: - name: german-jeopardy-mt5-large results: - task: name: Sequence-to-sequence Language Modeling type: text2text-generation dataset: name: lmqg/qg_dequad type: default args: default metrics: - name: BLEU-4 type: bleu4 value: 15.09 - name: F1 type: f1 value: 40.69 - name: ROUGE-1 type: rouge1 value: 41.68 - name: ROUGE-2 type: rouge2 value: 22.07 - name: ROUGE-L type: rougel value: 40.20 - name: ROUGE-Lsum type: rougelsum value: 40.19 - name: Exact Match type: exact_match value: 2.77 --- # german-jeopardy-mt5-large-1k-64-constant This model is a fine-tuned version of [google/mt5-large](https://huggingface.co/google/mt5-large) on the [lmqg/qg_dequad](https://huggingface.co/datasets/lmqg/qg_dequad) dataset. It achieves the following results on the evaluation set: - Loss: 1.8162 - Brevity Penalty: 0.9152 - System Length: 19102 - Reference Length: 20793 - ROUGE-1: 41.68 - ROUGE-2: 22.07 - ROUGE-L: 40.20 - ROUGE-Lsum: 40.19 - Exact Match: 2.77 - BLEU: 15.09 - F1: 40.69 ## Model description See [google/mt5-large](https://huggingface.co/google/mt5-large) for the model architecture. The model was trained on a single NVIDIA RTX 3090 GPU with 24GB of VRAM. ## Intended uses & limitations This model can be used for question generation on German text. ## Training and evaluation data See [lmqg/qg_dequad](https://huggingface.co/datasets/lmqg/qg_dequad). ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0001 - train_batch_size: 1 - eval_batch_size: 1 - seed: 7 - gradient_accumulation_steps: 64 - total_train_batch_size: 64 - optimizer: Adafactor - lr_scheduler_type: constant - num_epochs: 20 ### Training results | Training Loss | Epoch | Step | BLEU | Brevity Penalty | Counts 1 | Counts 2 | Counts 3 | Counts 4 | Exact Match | F1 | Mean Generated Length | Validation Loss | Precisions 1 | Precisions 2 | Precisions 3 | Precisions 4 | Reference Length | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-Lsum | System Length | Totals 1 | Totals 2 | Totals 3 | Totals 4 | |:-------------:|:-----:|:----:|:-------:|:---------------:|:--------:|:--------:|:--------:|:--------:|:-----------:|:------:|:---------------------:|:---------------:|:------------:|:------------:|:------------:|:------------:|:----------------:|:-------:|:-------:|:-------:|:----------:|:-------------:|:--------:|:--------:|:--------:|:--------:| | 2.732 | 1.0 | 145 | 12.4473 | 0.7805 | 7779 | 2893 | 1393 | 685 | 0.0168 | 0.3393 | 12.2523 | 1.2989 | 45.6809 | 19.5143 | 11.0372 | 6.5758 | 21250 | 0.3487 | 0.1796 | 0.3329 | 0.3327 | 17029 | 17029 | 14825 | 12621 | 10417 | | 1.5514 | 2.0 | 291 | 14.7663 | 0.7871 | 8297 | 3336 | 1711 | 899 | 0.025 | 0.3743 | 12.441 | 1.2100 | 48.3931 | 22.3278 | 13.4333 | 8.5351 | 21250 | 0.3839 | 0.2089 | 0.3688 | 0.369 | 17145 | 17145 | 14941 | 12737 | 10533 | | 1.3546 | 3.0 | 435 | 1.1428 | 8930 | 3713 | 1905 | 1022 | 17018 | 14814 | 12610 | 10406 | 52.4739 | 25.0641 | 15.1071 | 9.8213 | 0.7798 | 17018 | 21250 | 0.4225 | 0.2345 | 0.4075 | 0.4074 | 0.034 | 16.3903 | 12.6021 | 0.4155 | | 1.1969 | 4.0 | 581 | 1.1113 | 9456 | 3994 | 2096 | 1157 | 18171 | 15967 | 13763 | 11559 | 52.039 | 25.0141 | 15.2292 | 10.0095 | 0.8441 | 18171 | 21250 | 0.4409 | 0.246 | 0.4251 | 0.4251 | 0.0386 | 17.8161 | 13.4061 | 0.4334 | | 1.0876 | 5.0 | 726 | 1.1032 | 9606 | 4162 | 2233 | 1243 | 18179 | 15975 | 13771 | 11567 | 52.8412 | 26.0532 | 16.2152 | 10.7461 | 0.8446 | 18179 | 21250 | 0.4504 | 0.2571 | 0.4356 | 0.4357 | 0.0377 | 18.6911 | 13.5599 | 0.443 | | 0.9881 | 6.0 | 872 | 1.1119 | 9608 | 4167 | 2235 | 1246 | 18245 | 16041 | 13837 | 11633 | 52.661 | 25.9772 | 16.1523 | 10.7109 | 0.8481 | 18245 | 21250 | 0.4505 | 0.2567 | 0.4348 | 0.4349 | 0.044 | 18.7071 | 13.6978 | 0.4429 | | 0.9142 | 7.0 | 1017 | 1.1106 | 9757 | 4285 | 2311 | 1310 | 18291 | 16087 | 13883 | 11679 | 53.3432 | 26.6364 | 16.6463 | 11.2167 | 0.8506 | 18291 | 21250 | 0.4587 | 0.2641 | 0.4427 | 0.443 | 0.0495 | 19.3053 | 13.5826 | 0.451 | | 0.8323 | 8.0 | 1163 | 1.1327 | 9757 | 4300 | 2341 | 1317 | 18293 | 16089 | 13885 | 11681 | 53.3373 | 26.7263 | 16.8599 | 11.2747 | 0.8507 | 18293 | 21250 | 0.4587 | 0.2662 | 0.4429 | 0.4426 | 0.0472 | 19.4102 | 13.6239 | 0.4513 | | 0.7742 | 9.0 | 1308 | 1.1574 | 9757 | 4273 | 2324 | 1320 | 18273 | 16069 | 13865 | 11661 | 53.3957 | 26.5916 | 16.7616 | 11.3198 | 0.8497 | 18273 | 21250 | 0.4585 | 0.2653 | 0.4431 | 0.443 | 0.049 | 19.3574 | 13.5944 | 0.451 | | 0.7101 | 10.0 | 1454 | 1.1674 | 9861 | 4403 | 2438 | 1416 | 18641 | 16437 | 14233 | 12029 | 52.8995 | 26.7871 | 17.1292 | 11.7716 | 0.8694 | 18641 | 21250 | 0.4594 | 0.2689 | 0.444 | 0.4435 | 0.0531 | 20.1003 | 13.9133 | 0.4525 | | 0.6642 | 10.99 | 1599 | 1.1889 | 9868 | 4380 | 2358 | 1337 | 18386 | 16182 | 13978 | 11774 | 53.6713 | 27.0671 | 16.8694 | 11.3555 | 0.8558 | 18386 | 21250 | 0.4622 | 0.2694 | 0.4469 | 0.4466 | 0.0476 | 19.655 | 13.9142 | 0.4551 | | 0.6067 | 12.0 | 1745 | 1.2207 | 9872 | 4384 | 2408 | 1395 | 18894 | 16690 | 14486 | 12282 | 52.2494 | 26.2672 | 16.6229 | 11.3581 | 0.8828 | 18894 | 21250 | 0.4569 | 0.2667 | 0.441 | 0.4408 | 0.0472 | 19.9169 | 14.2482 | 0.4489 | | 0.5684 | 12.99 | 1890 | 1.2587 | 9870 | 4356 | 2360 | 1329 | 18901 | 16697 | 14493 | 12289 | 52.2195 | 26.0885 | 16.2837 | 10.8145 | 0.8831 | 18901 | 21250 | 0.4581 | 0.2651 | 0.4414 | 0.4409 | 0.0485 | 19.5451 | 14.2432 | 0.4506 | | 0.5288 | 14.0 | 2036 | 1.2804 | 9815 | 4360 | 2389 | 1335 | 18367 | 16163 | 13959 | 11755 | 53.4382 | 26.9752 | 17.1144 | 11.3569 | 0.8547 | 18367 | 21250 | 0.4592 | 0.2671 | 0.4443 | 0.4436 | 0.0454 | 19.6648 | 13.7432 | 0.4504 | | 0.4902 | 14.99 | 2181 | 1.3211 | 9886 | 4407 | 2398 | 1359 | 18777 | 16573 | 14369 | 12165 | 52.6495 | 26.5914 | 16.6887 | 11.1714 | 0.8766 | 18777 | 21250 | 0.4582 | 0.2674 | 0.4426 | 0.4421 | 0.0495 | 19.8138 | 14.1225 | 0.451 | | 0.4498 | 16.0 | 2327 | 1.3621 | 10008 | 4477 | 2456 | 1381 | 19399 | 17195 | 14991 | 12787 | 51.5903 | 26.0366 | 16.3832 | 10.8 | 0.909 | 19399 | 21250 | 0.4569 | 0.2679 | 0.4415 | 0.4412 | 0.0476 | 20.0703 | 14.3725 | 0.4491 | | 0.4216 | 16.99 | 2472 | 1.3967 | 10016 | 4483 | 2455 | 1385 | 19125 | 16921 | 14717 | 12513 | 52.3712 | 26.4937 | 16.6814 | 11.0685 | 0.8948 | 19125 | 21250 | 0.4615 | 0.2705 | 0.4457 | 0.4451 | 0.0481 | 20.1319 | 14.3008 | 0.4531 | | 0.3829 | 18.0 | 2618 | 1.4460 | 9976 | 4407 | 2412 | 1374 | 19464 | 17260 | 15056 | 12852 | 51.2536 | 25.533 | 16.0202 | 10.6909 | 0.9123 | 19464 | 21250 | 0.4556 | 0.2627 | 0.4387 | 0.4385 | 0.0476 | 19.8508 | 14.7046 | 0.4479 | | 0.3551 | 19.0 | 2764 | 1.4725 | 10010 | 4451 | 2438 | 1385 | 19131 | 16927 | 14723 | 12519 | 52.3235 | 26.2953 | 16.5591 | 11.0632 | 0.8952 | 19131 | 21250 | 0.4606 | 0.2672 | 0.4438 | 0.4434 | 0.0463 | 20.0572 | 14.3807 | 0.4523 | | 0.3301 | 19.93 | 2900 | 1.5030 | 9858 | 4378 | 2406 | 1368 | 18872 | 16668 | 14464 | 12260 | 52.2361 | 26.2659 | 16.6344 | 11.1582 | 0.8816 | 18872 | 21250 | 0.4569 | 0.2644 | 0.4412 | 0.4405 | 0.0495 | 19.8047 | 14.2795 | 0.4483 | ### Framework versions - Transformers 4.32.1 - Pytorch 2.1.0 - Datasets 2.12.0 - Tokenizers 0.13.3