--- tags: - generated_from_trainer metrics: - accuracy model-index: - name: nl_electra results: [] --- # nl_electra This model is a pretrained version of [ELECTRA](https://huggingface.co/docs/transformers/model_doc/electra) on the Dutch subset of the [CC100](https://huggingface.co/datasets/cc100) dataset. It achieves the following results on the evaluation set: - Loss: 2.4650 - Accuracy: 0.5392 ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-05 - train_batch_size: 16 - eval_batch_size: 16 - seed: 703 - gradient_accumulation_steps: 32 - total_train_batch_size: 512 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 8000 - num_epochs: 400.0 ### Training results | Training Loss | Epoch | Step | Validation Loss | Accuracy | |:-------------:|:------:|:------:|:---------------:|:--------:| | No log | 0.67 | 500 | 9.9977 | 0.0486 | | No log | 1.35 | 1000 | 9.5620 | 0.0543 | | No log | 2.02 | 1500 | 8.9306 | 0.0741 | | No log | 2.69 | 2000 | 8.2617 | 0.0826 | | No log | 3.36 | 2500 | 7.6880 | 0.0792 | | No log | 4.04 | 3000 | 7.3316 | 0.0757 | | No log | 4.71 | 3500 | 7.1944 | 0.0747 | | No log | 5.38 | 4000 | 7.1349 | 0.0802 | | No log | 6.06 | 4500 | 7.0752 | 0.0887 | | 8.201 | 6.73 | 5000 | 7.0046 | 0.1021 | | 8.201 | 7.4 | 5500 | 6.9295 | 0.1090 | | 8.201 | 8.08 | 6000 | 6.8483 | 0.1132 | | 8.201 | 8.75 | 6500 | 6.7750 | 0.1171 | | 8.201 | 9.42 | 7000 | 6.7116 | 0.1187 | | 8.201 | 10.09 | 7500 | 6.6560 | 0.1218 | | 8.201 | 10.77 | 8000 | 6.6178 | 0.1239 | | 8.201 | 11.44 | 8500 | 6.5824 | 0.1255 | | 8.201 | 12.11 | 9000 | 6.5521 | 0.1273 | | 8.201 | 12.79 | 9500 | 6.5203 | 0.1292 | | 6.7257 | 13.46 | 10000 | 6.5027 | 0.1303 | | 6.7257 | 14.13 | 10500 | 6.4809 | 0.1314 | | 6.7257 | 14.8 | 11000 | 6.4631 | 0.1322 | | 6.7257 | 15.48 | 11500 | 6.4483 | 0.1329 | | 6.7257 | 16.15 | 12000 | 6.4320 | 0.1338 | | 6.7257 | 16.82 | 12500 | 6.4169 | 0.1348 | | 6.7257 | 17.5 | 13000 | 6.4067 | 0.1359 | | 6.7257 | 18.17 | 13500 | 6.3994 | 0.1359 | | 6.7257 | 18.84 | 14000 | 6.3823 | 0.1368 | | 6.7257 | 19.52 | 14500 | 6.3759 | 0.1373 | | 6.4502 | 20.19 | 15000 | 6.3629 | 0.1374 | | 6.4502 | 20.86 | 15500 | 6.3638 | 0.1373 | | 6.4502 | 21.53 | 16000 | 6.3505 | 0.1382 | | 6.4502 | 22.21 | 16500 | 6.3416 | 0.1387 | | 6.4502 | 22.88 | 17000 | 6.3420 | 0.1383 | | 6.4502 | 23.55 | 17500 | 6.3330 | 0.1389 | | 6.4502 | 24.23 | 18000 | 6.3289 | 0.1388 | | 6.4502 | 24.9 | 18500 | 6.3184 | 0.1389 | | 6.4502 | 25.57 | 19000 | 6.3099 | 0.1396 | | 6.4502 | 26.24 | 19500 | 6.2789 | 0.1405 | | 6.3474 | 26.92 | 20000 | 6.2398 | 0.1404 | | 6.3474 | 27.59 | 20500 | 6.2012 | 0.1412 | | 6.3474 | 28.26 | 21000 | 6.1803 | 0.1414 | | 6.3474 | 28.94 | 21500 | 6.1579 | 0.1414 | | 6.3474 | 29.61 | 22000 | 6.1403 | 0.1431 | | 6.3474 | 30.28 | 22500 | 6.1316 | 0.1423 | | 6.3474 | 30.96 | 23000 | 6.1102 | 0.1435 | | 6.3474 | 31.63 | 23500 | 6.0998 | 0.1439 | | 6.3474 | 32.3 | 24000 | 6.0867 | 0.1446 | | 6.3474 | 32.97 | 24500 | 6.0700 | 0.1451 | | 6.1758 | 33.65 | 25000 | 6.0554 | 0.1457 | | 6.1758 | 34.32 | 25500 | 6.0487 | 0.1457 | | 6.1758 | 34.99 | 26000 | 6.0328 | 0.1469 | | 6.1758 | 35.67 | 26500 | 6.0265 | 0.1469 | | 6.1758 | 36.34 | 27000 | 5.9992 | 0.1486 | | 6.1758 | 37.01 | 27500 | 5.9934 | 0.1485 | | 6.1758 | 37.68 | 28000 | 5.9702 | 0.1501 | | 6.1758 | 38.36 | 28500 | 5.9661 | 0.1503 | | 6.1758 | 39.03 | 29000 | 5.9558 | 0.1512 | | 6.1758 | 39.7 | 29500 | 5.9321 | 0.1528 | | 6.052 | 40.38 | 30000 | 5.9147 | 0.1532 | | 6.052 | 41.05 | 30500 | 5.8975 | 0.1545 | | 6.052 | 41.72 | 31000 | 5.8784 | 0.1566 | | 6.052 | 42.4 | 31500 | 5.8584 | 0.1586 | | 6.052 | 43.07 | 32000 | 5.8325 | 0.1603 | | 6.052 | 43.74 | 32500 | 5.7583 | 0.1664 | | 6.052 | 44.41 | 33000 | 5.6158 | 0.1787 | | 6.052 | 45.09 | 33500 | 5.4580 | 0.1917 | | 6.052 | 45.76 | 34000 | 5.3396 | 0.2010 | | 6.052 | 46.43 | 34500 | 5.2568 | 0.2082 | | 5.7995 | 47.11 | 35000 | 5.1775 | 0.2146 | | 5.7995 | 47.78 | 35500 | 5.1076 | 0.2204 | | 5.7995 | 48.45 | 36000 | 5.0457 | 0.2258 | | 5.7995 | 49.13 | 36500 | 4.9932 | 0.2313 | | 5.7995 | 49.8 | 37000 | 4.9164 | 0.2384 | | 5.7995 | 50.47 | 37500 | 4.7844 | 0.2521 | | 5.7995 | 51.14 | 38000 | 4.6598 | 0.2642 | | 5.7995 | 51.82 | 38500 | 4.5472 | 0.2757 | | 5.7995 | 52.49 | 39000 | 4.4374 | 0.2871 | | 5.7995 | 53.16 | 39500 | 4.3399 | 0.2982 | | 5.0341 | 53.84 | 40000 | 4.2549 | 0.3083 | | 5.0341 | 54.51 | 40500 | 4.1795 | 0.3170 | | 5.0341 | 55.18 | 41000 | 4.1017 | 0.3274 | | 5.0341 | 55.85 | 41500 | 4.0308 | 0.3375 | | 5.0341 | 56.53 | 42000 | 3.9673 | 0.3462 | | 5.0341 | 57.2 | 42500 | 3.9130 | 0.3538 | | 5.0341 | 57.87 | 43000 | 3.8672 | 0.3599 | | 5.0341 | 58.55 | 43500 | 3.8249 | 0.3656 | | 5.0341 | 59.22 | 44000 | 3.7748 | 0.3728 | | 5.0341 | 59.89 | 44500 | 3.7459 | 0.3768 | | 4.2119 | 60.57 | 45000 | 3.7089 | 0.3808 | | 4.2119 | 61.24 | 45500 | 3.6732 | 0.3857 | | 4.2119 | 61.91 | 46000 | 3.6546 | 0.3881 | | 4.2119 | 62.58 | 46500 | 3.6205 | 0.3921 | | 4.2119 | 63.26 | 47000 | 3.5908 | 0.3960 | | 4.2119 | 63.93 | 47500 | 3.5627 | 0.3991 | | 4.2119 | 64.6 | 48000 | 3.5466 | 0.4019 | | 4.2119 | 65.28 | 48500 | 3.5262 | 0.4039 | | 4.2119 | 65.95 | 49000 | 3.4987 | 0.4074 | | 4.2119 | 66.62 | 49500 | 3.4817 | 0.4093 | | 3.8182 | 67.29 | 50000 | 3.4608 | 0.4119 | | 3.8182 | 67.97 | 50500 | 3.4467 | 0.4142 | | 3.8182 | 68.64 | 51000 | 3.4280 | 0.4163 | | 3.8182 | 69.31 | 51500 | 3.4165 | 0.4175 | | 3.8182 | 69.99 | 52000 | 3.3970 | 0.4199 | | 3.8182 | 70.66 | 52500 | 3.3738 | 0.4227 | | 3.8182 | 71.33 | 53000 | 3.3640 | 0.4242 | | 3.8182 | 72.01 | 53500 | 3.3583 | 0.4252 | | 3.8182 | 72.68 | 54000 | 3.3319 | 0.4279 | | 3.8182 | 73.35 | 54500 | 3.3153 | 0.4303 | | 3.5946 | 74.02 | 55000 | 3.3098 | 0.4304 | | 3.5946 | 74.7 | 55500 | 3.2949 | 0.4328 | | 3.5946 | 75.37 | 56000 | 3.2820 | 0.4335 | | 3.5946 | 76.04 | 56500 | 3.2686 | 0.4355 | | 3.5946 | 76.72 | 57000 | 3.2663 | 0.4359 | | 3.5946 | 77.39 | 57500 | 3.2482 | 0.4379 | | 3.5946 | 78.06 | 58000 | 3.2344 | 0.4393 | | 3.5946 | 78.73 | 58500 | 3.2281 | 0.4407 | | 3.5946 | 79.41 | 59000 | 3.2172 | 0.4412 | | 3.5946 | 80.08 | 59500 | 3.2110 | 0.4420 | | 3.4435 | 80.75 | 60000 | 3.1940 | 0.4443 | | 3.4435 | 81.43 | 60500 | 3.1837 | 0.4455 | | 3.4435 | 82.1 | 61000 | 3.1744 | 0.4469 | | 3.4435 | 82.77 | 61500 | 3.1611 | 0.4483 | | 3.4435 | 83.45 | 62000 | 3.1531 | 0.4496 | | 3.4435 | 84.12 | 62500 | 3.1524 | 0.4499 | | 3.4435 | 84.79 | 63000 | 3.1431 | 0.4501 | | 3.4435 | 85.46 | 63500 | 3.1287 | 0.4527 | | 3.4435 | 86.14 | 64000 | 3.1192 | 0.4533 | | 3.4435 | 86.81 | 64500 | 3.1107 | 0.4547 | | 3.3301 | 87.48 | 65000 | 3.1041 | 0.4553 | | 3.3301 | 88.16 | 65500 | 3.0999 | 0.4562 | | 3.3301 | 88.83 | 66000 | 3.0882 | 0.4576 | | 3.3301 | 89.5 | 66500 | 3.0777 | 0.4589 | | 3.3301 | 90.17 | 67000 | 3.0726 | 0.4588 | | 3.3301 | 90.85 | 67500 | 3.0676 | 0.4601 | | 3.3301 | 91.52 | 68000 | 3.0616 | 0.4602 | | 3.3301 | 92.19 | 68500 | 3.0523 | 0.4621 | | 3.3301 | 92.87 | 69000 | 3.0464 | 0.4624 | | 3.3301 | 93.54 | 69500 | 3.0405 | 0.4635 | | 3.2418 | 94.21 | 70000 | 3.0312 | 0.4649 | | 3.2418 | 94.89 | 70500 | 3.0209 | 0.4653 | | 3.2418 | 95.56 | 71000 | 3.0202 | 0.4657 | | 3.2418 | 96.23 | 71500 | 3.0101 | 0.4676 | | 3.2418 | 96.9 | 72000 | 3.0105 | 0.4666 | | 3.2418 | 97.58 | 72500 | 3.0023 | 0.4685 | | 3.2418 | 98.25 | 73000 | 3.0008 | 0.4680 | | 3.2418 | 98.92 | 73500 | 2.9882 | 0.4691 | | 3.2418 | 99.6 | 74000 | 2.9855 | 0.4702 | | 3.2418 | 100.27 | 74500 | 2.9790 | 0.4709 | | 3.1698 | 100.94 | 75000 | 2.9680 | 0.4716 | | 3.1698 | 101.61 | 75500 | 2.9667 | 0.4724 | | 3.1698 | 102.29 | 76000 | 2.9657 | 0.4726 | | 3.1698 | 102.96 | 76500 | 2.9623 | 0.4731 | | 3.1698 | 103.63 | 77000 | 2.9515 | 0.4745 | | 3.1698 | 104.31 | 77500 | 2.9471 | 0.4753 | | 3.1698 | 104.98 | 78000 | 2.9407 | 0.4756 | | 3.1698 | 105.65 | 78500 | 2.9388 | 0.4761 | | 3.1698 | 106.33 | 79000 | 2.9369 | 0.4766 | | 3.1698 | 107.0 | 79500 | 2.9297 | 0.4762 | | 3.1101 | 107.67 | 80000 | 2.9291 | 0.4776 | | 3.1101 | 108.34 | 80500 | 2.9139 | 0.4788 | | 3.1101 | 109.02 | 81000 | 2.9113 | 0.4790 | | 3.1101 | 109.69 | 81500 | 2.9057 | 0.4798 | | 3.1101 | 110.36 | 82000 | 2.9058 | 0.4804 | | 3.1101 | 111.04 | 82500 | 2.9019 | 0.4807 | | 3.1101 | 111.71 | 83000 | 2.8934 | 0.4818 | | 3.1101 | 112.38 | 83500 | 2.8864 | 0.4825 | | 3.1101 | 113.06 | 84000 | 2.8926 | 0.4815 | | 3.1101 | 113.73 | 84500 | 2.8812 | 0.4830 | | 3.058 | 114.4 | 85000 | 2.8740 | 0.4840 | | 3.058 | 115.07 | 85500 | 2.8730 | 0.4840 | | 3.058 | 115.75 | 86000 | 2.8694 | 0.4847 | | 3.058 | 116.42 | 86500 | 2.8694 | 0.4848 | | 3.058 | 117.09 | 87000 | 2.8601 | 0.4862 | | 3.058 | 117.77 | 87500 | 2.8547 | 0.4862 | | 3.058 | 118.44 | 88000 | 2.8538 | 0.4861 | | 3.058 | 119.11 | 88500 | 2.8494 | 0.4876 | | 3.058 | 119.78 | 89000 | 2.8430 | 0.4882 | | 3.058 | 120.46 | 89500 | 2.8436 | 0.4875 | | 3.0129 | 121.13 | 90000 | 2.8402 | 0.4884 | | 3.0129 | 121.8 | 90500 | 2.8353 | 0.4888 | | 3.0129 | 122.48 | 91000 | 2.8271 | 0.4896 | | 3.0129 | 123.15 | 91500 | 2.8236 | 0.4900 | | 3.0129 | 123.82 | 92000 | 2.8199 | 0.4913 | | 3.0129 | 124.5 | 92500 | 2.8119 | 0.4916 | | 3.0129 | 125.17 | 93000 | 2.8138 | 0.4916 | | 3.0129 | 125.84 | 93500 | 2.8089 | 0.4925 | | 3.0129 | 126.51 | 94000 | 2.8067 | 0.4928 | | 3.0129 | 127.19 | 94500 | 2.8010 | 0.4939 | | 2.9701 | 127.86 | 95000 | 2.7992 | 0.4938 | | 2.9701 | 128.53 | 95500 | 2.7953 | 0.4948 | | 2.9701 | 129.21 | 96000 | 2.7964 | 0.4942 | | 2.9701 | 129.88 | 96500 | 2.7838 | 0.4955 | | 2.9701 | 130.55 | 97000 | 2.7770 | 0.4968 | | 2.9701 | 131.22 | 97500 | 2.7800 | 0.4962 | | 2.9701 | 131.9 | 98000 | 2.7743 | 0.4972 | | 2.9701 | 132.57 | 98500 | 2.7696 | 0.4973 | | 2.9701 | 133.24 | 99000 | 2.7691 | 0.4980 | | 2.9701 | 133.92 | 99500 | 2.7612 | 0.4989 | | 2.9289 | 134.59 | 100000 | 2.7606 | 0.4987 | | 2.9289 | 135.26 | 100500 | 2.7545 | 0.4993 | | 2.9289 | 135.94 | 101000 | 2.7544 | 0.4999 | | 2.9289 | 136.61 | 101500 | 2.7550 | 0.4999 | | 2.9289 | 137.28 | 102000 | 2.7510 | 0.5001 | | 2.9289 | 137.95 | 102500 | 2.7480 | 0.5002 | | 2.9289 | 138.63 | 103000 | 2.7422 | 0.5012 | | 2.9289 | 139.3 | 103500 | 2.7419 | 0.5014 | | 2.9289 | 139.97 | 104000 | 2.7416 | 0.5009 | | 2.9289 | 140.65 | 104500 | 2.7412 | 0.5017 | | 2.8968 | 141.32 | 105000 | 2.7356 | 0.5023 | | 2.8968 | 141.99 | 105500 | 2.7303 | 0.5027 | | 2.8968 | 142.66 | 106000 | 2.7359 | 0.5029 | | 2.8968 | 143.34 | 106500 | 2.7283 | 0.5032 | | 2.8968 | 144.01 | 107000 | 2.7226 | 0.5033 | | 2.8968 | 144.68 | 107500 | 2.7247 | 0.5039 | | 2.8968 | 145.36 | 108000 | 2.7209 | 0.5044 | | 2.8968 | 146.03 | 108500 | 2.7210 | 0.5039 | | 2.8968 | 146.7 | 109000 | 2.7135 | 0.5046 | | 2.8968 | 147.38 | 109500 | 2.7139 | 0.5048 | | 2.8697 | 148.05 | 110000 | 2.7167 | 0.5050 | | 2.8697 | 148.72 | 110500 | 2.7125 | 0.5058 | | 2.8697 | 149.39 | 111000 | 2.7064 | 0.5060 | | 2.8697 | 150.07 | 111500 | 2.7024 | 0.5067 | | 2.8697 | 150.74 | 112000 | 2.7035 | 0.5067 | | 2.8697 | 151.41 | 112500 | 2.7034 | 0.5067 | | 2.8697 | 152.09 | 113000 | 2.6967 | 0.5073 | | 2.8697 | 152.76 | 113500 | 2.6982 | 0.5070 | | 2.8697 | 153.43 | 114000 | 2.6948 | 0.5079 | | 2.8697 | 154.1 | 114500 | 2.6946 | 0.5076 | | 2.8457 | 154.78 | 115000 | 2.6918 | 0.5078 | | 2.8457 | 155.45 | 115500 | 2.6917 | 0.5078 | | 2.8457 | 156.12 | 116000 | 2.6868 | 0.5086 | | 2.8457 | 156.8 | 116500 | 2.6870 | 0.5084 | | 2.8457 | 157.47 | 117000 | 2.6830 | 0.5091 | | 2.8457 | 158.14 | 117500 | 2.6824 | 0.5090 | | 2.8457 | 158.82 | 118000 | 2.6812 | 0.5092 | | 2.8457 | 159.49 | 118500 | 2.6747 | 0.5098 | | 2.8457 | 160.16 | 119000 | 2.6747 | 0.5105 | | 2.8457 | 160.83 | 119500 | 2.6750 | 0.5102 | | 2.825 | 161.51 | 120000 | 2.6761 | 0.5102 | | 2.825 | 162.18 | 120500 | 2.6670 | 0.5115 | | 2.825 | 162.85 | 121000 | 2.6740 | 0.5104 | | 2.825 | 163.53 | 121500 | 2.6700 | 0.5106 | | 2.825 | 164.2 | 122000 | 2.6629 | 0.5119 | | 2.825 | 164.87 | 122500 | 2.6642 | 0.5117 | | 2.825 | 165.54 | 123000 | 2.6664 | 0.5109 | | 2.825 | 166.22 | 123500 | 2.6626 | 0.5117 | | 2.825 | 166.89 | 124000 | 2.6561 | 0.5130 | | 2.825 | 167.56 | 124500 | 2.6612 | 0.5125 | | 2.8059 | 168.24 | 125000 | 2.6594 | 0.5123 | | 2.8059 | 168.91 | 125500 | 2.6508 | 0.5132 | | 2.8059 | 169.58 | 126000 | 2.6477 | 0.5134 | | 2.8059 | 170.26 | 126500 | 2.6527 | 0.5133 | | 2.8059 | 170.93 | 127000 | 2.6519 | 0.5136 | | 2.8059 | 171.6 | 127500 | 2.6456 | 0.5141 | | 2.8059 | 172.27 | 128000 | 2.6473 | 0.5139 | | 2.8059 | 172.95 | 128500 | 2.6426 | 0.5144 | | 2.8059 | 173.62 | 129000 | 2.6454 | 0.5137 | | 2.8059 | 174.29 | 129500 | 2.6427 | 0.5147 | | 2.788 | 174.97 | 130000 | 2.6376 | 0.5150 | | 2.788 | 175.64 | 130500 | 2.6366 | 0.5154 | | 2.788 | 176.31 | 131000 | 2.6327 | 0.5156 | | 2.788 | 176.98 | 131500 | 2.6328 | 0.5157 | | 2.788 | 177.66 | 132000 | 2.6335 | 0.5156 | | 2.788 | 178.33 | 132500 | 2.6302 | 0.5166 | | 2.788 | 179.0 | 133000 | 2.6333 | 0.5160 | | 2.788 | 179.68 | 133500 | 2.6253 | 0.5171 | | 2.788 | 180.35 | 134000 | 2.6237 | 0.5167 | | 2.788 | 181.02 | 134500 | 2.6269 | 0.5165 | | 2.7723 | 181.7 | 135000 | 2.6283 | 0.5164 | | 2.7723 | 182.37 | 135500 | 2.6255 | 0.5174 | | 2.7723 | 183.04 | 136000 | 2.6254 | 0.5175 | | 2.7723 | 183.71 | 136500 | 2.6231 | 0.5172 | | 2.7723 | 184.39 | 137000 | 2.6181 | 0.5173 | | 2.7723 | 185.06 | 137500 | 2.6260 | 0.5168 | | 2.7723 | 185.73 | 138000 | 2.6183 | 0.5176 | | 2.7723 | 186.41 | 138500 | 2.6174 | 0.5182 | | 2.7723 | 187.08 | 139000 | 2.6144 | 0.5182 | | 2.7723 | 187.75 | 139500 | 2.6152 | 0.5186 | | 2.7575 | 188.43 | 140000 | 2.6150 | 0.5183 | | 2.7575 | 189.1 | 140500 | 2.6110 | 0.5190 | | 2.7575 | 189.77 | 141000 | 2.6044 | 0.5194 | | 2.7575 | 190.44 | 141500 | 2.6083 | 0.5186 | | 2.7575 | 191.12 | 142000 | 2.6102 | 0.5189 | | 2.7575 | 191.79 | 142500 | 2.6063 | 0.5195 | | 2.7575 | 192.46 | 143000 | 2.6071 | 0.5198 | | 2.7575 | 193.14 | 143500 | 2.6090 | 0.5191 | | 2.7575 | 193.81 | 144000 | 2.6068 | 0.5200 | | 2.7575 | 194.48 | 144500 | 2.6032 | 0.5200 | | 2.7445 | 195.15 | 145000 | 2.6031 | 0.5200 | | 2.7445 | 195.83 | 145500 | 2.6007 | 0.5201 | | 2.7445 | 196.5 | 146000 | 2.5998 | 0.5203 | | 2.7445 | 197.17 | 146500 | 2.5980 | 0.5208 | | 2.7445 | 197.85 | 147000 | 2.5952 | 0.5211 | | 2.7445 | 198.52 | 147500 | 2.5977 | 0.5210 | | 2.7445 | 199.19 | 148000 | 2.5922 | 0.5212 | | 2.7445 | 199.87 | 148500 | 2.5936 | 0.5211 | | 2.7445 | 200.54 | 149000 | 2.5933 | 0.5219 | | 2.7445 | 201.21 | 149500 | 2.5887 | 0.5219 | | 2.7324 | 201.88 | 150000 | 2.5911 | 0.5215 | | 2.7324 | 202.56 | 150500 | 2.5892 | 0.5219 | | 2.7324 | 203.23 | 151000 | 2.5875 | 0.5218 | | 2.7324 | 203.9 | 151500 | 2.5898 | 0.5220 | | 2.7324 | 204.58 | 152000 | 2.5872 | 0.5223 | | 2.7324 | 205.25 | 152500 | 2.5805 | 0.5226 | | 2.7324 | 205.92 | 153000 | 2.5861 | 0.5225 | | 2.7324 | 206.59 | 153500 | 2.5839 | 0.5223 | | 2.7324 | 207.27 | 154000 | 2.5804 | 0.5232 | | 2.7324 | 207.94 | 154500 | 2.5766 | 0.5235 | | 2.7212 | 208.61 | 155000 | 2.5764 | 0.5233 | | 2.7212 | 209.29 | 155500 | 2.5801 | 0.5233 | | 2.7212 | 209.96 | 156000 | 2.5737 | 0.5241 | | 2.7212 | 210.63 | 156500 | 2.5769 | 0.5236 | | 2.7212 | 211.31 | 157000 | 2.5769 | 0.5237 | | 2.7212 | 211.98 | 157500 | 2.5748 | 0.5239 | | 2.7212 | 212.65 | 158000 | 2.5745 | 0.5230 | | 2.7212 | 213.32 | 158500 | 2.5725 | 0.5240 | | 2.7212 | 214.0 | 159000 | 2.5736 | 0.5239 | | 2.7212 | 214.67 | 159500 | 2.5675 | 0.5252 | | 2.7103 | 215.34 | 160000 | 2.5678 | 0.5245 | | 2.7103 | 216.02 | 160500 | 2.5691 | 0.5250 | | 2.7103 | 216.69 | 161000 | 2.5688 | 0.5245 | | 2.7103 | 217.36 | 161500 | 2.5681 | 0.5251 | | 2.7103 | 218.03 | 162000 | 2.5582 | 0.5255 | | 2.7103 | 218.71 | 162500 | 2.5675 | 0.5247 | | 2.7103 | 219.38 | 163000 | 2.5609 | 0.5259 | | 2.7103 | 220.05 | 163500 | 2.5625 | 0.5254 | | 2.7103 | 220.73 | 164000 | 2.5630 | 0.5254 | | 2.7103 | 221.4 | 164500 | 2.5607 | 0.5265 | | 2.7003 | 222.07 | 165000 | 2.5615 | 0.5260 | | 2.7003 | 222.75 | 165500 | 2.5660 | 0.5248 | | 2.7003 | 223.42 | 166000 | 2.5569 | 0.5263 | | 2.7003 | 224.09 | 166500 | 2.5610 | 0.5255 | | 2.7003 | 224.76 | 167000 | 2.5569 | 0.5263 | | 2.7003 | 225.44 | 167500 | 2.5534 | 0.5265 | | 2.7003 | 226.11 | 168000 | 2.5573 | 0.5259 | | 2.7003 | 226.78 | 168500 | 2.5559 | 0.5264 | | 2.7003 | 227.46 | 169000 | 2.5508 | 0.5277 | | 2.7003 | 228.13 | 169500 | 2.5500 | 0.5276 | | 2.6915 | 228.8 | 170000 | 2.5501 | 0.5270 | | 2.6915 | 229.47 | 170500 | 2.5508 | 0.5273 | | 2.6915 | 230.15 | 171000 | 2.5523 | 0.5267 | | 2.6915 | 230.82 | 171500 | 2.5464 | 0.5276 | | 2.6915 | 231.49 | 172000 | 2.5482 | 0.5271 | | 2.6915 | 232.17 | 172500 | 2.5486 | 0.5270 | | 2.6915 | 232.84 | 173000 | 2.5474 | 0.5275 | | 2.6915 | 233.51 | 173500 | 2.5483 | 0.5270 | | 2.6915 | 234.19 | 174000 | 2.5480 | 0.5276 | | 2.6915 | 234.86 | 174500 | 2.5486 | 0.5278 | | 2.6833 | 235.53 | 175000 | 2.5484 | 0.5273 | | 2.6833 | 236.2 | 175500 | 2.5436 | 0.5277 | | 2.6833 | 236.88 | 176000 | 2.5448 | 0.5278 | | 2.6833 | 237.55 | 176500 | 2.5430 | 0.5284 | | 2.6833 | 238.22 | 177000 | 2.5433 | 0.5279 | | 2.6833 | 238.9 | 177500 | 2.5398 | 0.5288 | | 2.6833 | 239.57 | 178000 | 2.5424 | 0.5282 | | 2.6833 | 240.24 | 178500 | 2.5371 | 0.5291 | | 2.6833 | 240.91 | 179000 | 2.5372 | 0.5294 | | 2.6833 | 241.59 | 179500 | 2.5368 | 0.5290 | | 2.6753 | 242.26 | 180000 | 2.5383 | 0.5289 | | 2.6753 | 242.93 | 180500 | 2.5387 | 0.5289 | | 2.6753 | 243.61 | 181000 | 2.5351 | 0.5295 | | 2.6753 | 244.28 | 181500 | 2.5340 | 0.5296 | | 2.6753 | 244.95 | 182000 | 2.5349 | 0.5289 | | 2.6753 | 245.63 | 182500 | 2.5358 | 0.5295 | | 2.6753 | 246.3 | 183000 | 2.5333 | 0.5299 | | 2.6753 | 246.97 | 183500 | 2.5363 | 0.5292 | | 2.6753 | 247.64 | 184000 | 2.5323 | 0.5298 | | 2.6753 | 248.32 | 184500 | 2.5286 | 0.5299 | | 2.6679 | 248.99 | 185000 | 2.5276 | 0.5306 | | 2.6679 | 249.66 | 185500 | 2.5249 | 0.5308 | | 2.6679 | 250.34 | 186000 | 2.5308 | 0.5302 | | 2.6679 | 251.01 | 186500 | 2.5307 | 0.5297 | | 2.6679 | 251.68 | 187000 | 2.5293 | 0.5305 | | 2.6679 | 252.36 | 187500 | 2.5255 | 0.5306 | | 2.6679 | 253.03 | 188000 | 2.5244 | 0.5312 | | 2.6679 | 253.7 | 188500 | 2.5278 | 0.5305 | | 2.6679 | 254.37 | 189000 | 2.5212 | 0.5317 | | 2.6679 | 255.05 | 189500 | 2.5256 | 0.5307 | | 2.6611 | 255.72 | 190000 | 2.5273 | 0.5303 | | 2.6611 | 256.39 | 190500 | 2.5222 | 0.5310 | | 2.6611 | 257.07 | 191000 | 2.5237 | 0.5311 | | 2.6611 | 257.74 | 191500 | 2.5258 | 0.5309 | | 2.6611 | 258.41 | 192000 | 2.5219 | 0.5313 | | 2.6611 | 259.08 | 192500 | 2.5243 | 0.5314 | | 2.6611 | 259.76 | 193000 | 2.5203 | 0.5319 | | 2.6611 | 260.43 | 193500 | 2.5205 | 0.5313 | | 2.6611 | 261.1 | 194000 | 2.5205 | 0.5322 | | 2.6611 | 261.78 | 194500 | 2.5196 | 0.5317 | | 2.655 | 262.45 | 195000 | 2.5199 | 0.5315 | | 2.655 | 263.12 | 195500 | 2.5226 | 0.5315 | | 2.655 | 263.8 | 196000 | 2.5175 | 0.5316 | | 2.655 | 264.47 | 196500 | 2.5160 | 0.5322 | | 2.655 | 265.14 | 197000 | 2.5185 | 0.5316 | | 2.655 | 265.81 | 197500 | 2.5133 | 0.5322 | | 2.655 | 266.49 | 198000 | 2.5163 | 0.5318 | | 2.655 | 267.16 | 198500 | 2.5135 | 0.5325 | | 2.655 | 267.83 | 199000 | 2.5132 | 0.5326 | | 2.655 | 268.51 | 199500 | 2.5148 | 0.5323 | | 2.6486 | 269.18 | 200000 | 2.5194 | 0.5317 | | 2.6486 | 269.85 | 200500 | 2.5162 | 0.5321 | | 2.6486 | 270.52 | 201000 | 2.5090 | 0.5332 | | 2.6486 | 271.2 | 201500 | 2.5126 | 0.5325 | | 2.6486 | 271.87 | 202000 | 2.5155 | 0.5320 | | 2.6486 | 272.54 | 202500 | 2.5099 | 0.5329 | | 2.6486 | 273.22 | 203000 | 2.5130 | 0.5325 | | 2.6486 | 273.89 | 203500 | 2.5064 | 0.5329 | | 2.6486 | 274.56 | 204000 | 2.5154 | 0.5319 | | 2.6486 | 275.24 | 204500 | 2.5097 | 0.5329 | | 2.6433 | 275.91 | 205000 | 2.5075 | 0.5334 | | 2.6433 | 276.58 | 205500 | 2.5120 | 0.5325 | | 2.6433 | 277.25 | 206000 | 2.5100 | 0.5329 | | 2.6433 | 277.93 | 206500 | 2.5115 | 0.5332 | | 2.6433 | 278.6 | 207000 | 2.5071 | 0.5332 | | 2.6433 | 279.27 | 207500 | 2.5075 | 0.5335 | | 2.6433 | 279.95 | 208000 | 2.5020 | 0.5338 | | 2.6433 | 280.62 | 208500 | 2.5025 | 0.5340 | | 2.6433 | 281.29 | 209000 | 2.5064 | 0.5333 | | 2.6433 | 281.96 | 209500 | 2.5038 | 0.5336 | | 2.6383 | 282.64 | 210000 | 2.5041 | 0.5340 | | 2.6383 | 283.31 | 210500 | 2.5075 | 0.5336 | | 2.6383 | 283.98 | 211000 | 2.5028 | 0.5333 | | 2.6383 | 284.66 | 211500 | 2.5008 | 0.5340 | | 2.6383 | 285.33 | 212000 | 2.5005 | 0.5345 | | 2.6383 | 286.0 | 212500 | 2.5020 | 0.5334 | | 2.6383 | 286.68 | 213000 | 2.5011 | 0.5344 | | 2.6383 | 287.35 | 213500 | 2.5028 | 0.5338 | | 2.6383 | 288.02 | 214000 | 2.4970 | 0.5340 | | 2.6383 | 288.69 | 214500 | 2.4995 | 0.5343 | | 2.6336 | 289.37 | 215000 | 2.5010 | 0.5343 | | 2.6336 | 290.04 | 215500 | 2.5060 | 0.5336 | | 2.6336 | 290.71 | 216000 | 2.4955 | 0.5347 | | 2.6336 | 291.39 | 216500 | 2.4972 | 0.5349 | | 2.6336 | 292.06 | 217000 | 2.4977 | 0.5349 | | 2.6336 | 292.73 | 217500 | 2.4973 | 0.5346 | | 2.6336 | 293.4 | 218000 | 2.4981 | 0.5346 | | 2.6336 | 294.08 | 218500 | 2.4941 | 0.5346 | | 2.6336 | 294.75 | 219000 | 2.4978 | 0.5350 | | 2.6336 | 295.42 | 219500 | 2.4990 | 0.5343 | | 2.6288 | 296.1 | 220000 | 2.4929 | 0.5347 | | 2.6288 | 296.77 | 220500 | 2.4937 | 0.5349 | | 2.6288 | 297.44 | 221000 | 2.4938 | 0.5349 | | 2.6288 | 298.12 | 221500 | 2.4938 | 0.5343 | | 2.6288 | 298.79 | 222000 | 2.4924 | 0.5354 | | 2.6288 | 299.46 | 222500 | 2.4953 | 0.5348 | | 2.6288 | 300.13 | 223000 | 2.4930 | 0.5347 | | 2.6288 | 300.81 | 223500 | 2.4934 | 0.5353 | | 2.6288 | 301.48 | 224000 | 2.4942 | 0.5348 | | 2.6288 | 302.15 | 224500 | 2.4960 | 0.5344 | | 2.6246 | 302.83 | 225000 | 2.4875 | 0.5357 | | 2.6246 | 303.5 | 225500 | 2.4898 | 0.5355 | | 2.6246 | 304.17 | 226000 | 2.4847 | 0.5366 | | 2.6246 | 304.84 | 226500 | 2.4970 | 0.5348 | | 2.6246 | 305.52 | 227000 | 2.4905 | 0.5356 | | 2.6246 | 306.19 | 227500 | 2.4873 | 0.5361 | | 2.6246 | 306.86 | 228000 | 2.4939 | 0.5350 | | 2.6246 | 307.54 | 228500 | 2.4910 | 0.5360 | | 2.6246 | 308.21 | 229000 | 2.4886 | 0.5355 | | 2.6246 | 308.88 | 229500 | 2.4890 | 0.5369 | | 2.6207 | 309.56 | 230000 | 2.4900 | 0.5360 | | 2.6207 | 310.23 | 230500 | 2.4885 | 0.5354 | | 2.6207 | 310.9 | 231000 | 2.4895 | 0.5358 | | 2.6207 | 311.57 | 231500 | 2.4871 | 0.5358 | | 2.6207 | 312.25 | 232000 | 2.4914 | 0.5352 | | 2.6207 | 312.92 | 232500 | 2.4843 | 0.5366 | | 2.6207 | 313.59 | 233000 | 2.4837 | 0.5365 | | 2.6207 | 314.27 | 233500 | 2.4883 | 0.5360 | | 2.6207 | 314.94 | 234000 | 2.4839 | 0.5366 | | 2.6207 | 315.61 | 234500 | 2.4854 | 0.5366 | | 2.6171 | 316.29 | 235000 | 2.4833 | 0.5367 | | 2.6171 | 316.96 | 235500 | 2.4783 | 0.5374 | | 2.6171 | 317.63 | 236000 | 2.4807 | 0.5370 | | 2.6171 | 318.3 | 236500 | 2.4824 | 0.5366 | | 2.6171 | 318.98 | 237000 | 2.4857 | 0.5361 | | 2.6171 | 319.65 | 237500 | 2.4817 | 0.5366 | | 2.6171 | 320.32 | 238000 | 2.4855 | 0.5364 | | 2.6171 | 321.0 | 238500 | 2.4834 | 0.5367 | | 2.6171 | 321.67 | 239000 | 2.4831 | 0.5363 | | 2.6171 | 322.34 | 239500 | 2.4806 | 0.5370 | | 2.6134 | 323.01 | 240000 | 2.4842 | 0.5365 | | 2.6134 | 323.69 | 240500 | 2.4830 | 0.5364 | | 2.6134 | 324.36 | 241000 | 2.4822 | 0.5367 | | 2.6134 | 325.03 | 241500 | 2.4805 | 0.5373 | | 2.6134 | 325.71 | 242000 | 2.4838 | 0.5365 | | 2.6134 | 326.38 | 242500 | 2.4776 | 0.5371 | | 2.6134 | 327.05 | 243000 | 2.4786 | 0.5376 | | 2.6134 | 327.73 | 243500 | 2.4824 | 0.5371 | | 2.6134 | 328.4 | 244000 | 2.4842 | 0.5363 | | 2.6134 | 329.07 | 244500 | 2.4790 | 0.5375 | | 2.6107 | 329.74 | 245000 | 2.4770 | 0.5372 | | 2.6107 | 330.42 | 245500 | 2.4805 | 0.5375 | | 2.6107 | 331.09 | 246000 | 2.4839 | 0.5370 | | 2.6107 | 331.76 | 246500 | 2.4802 | 0.5367 | | 2.6107 | 332.44 | 247000 | 2.4796 | 0.5373 | | 2.6107 | 333.11 | 247500 | 2.4736 | 0.5377 | | 2.6107 | 333.78 | 248000 | 2.4789 | 0.5374 | | 2.6107 | 334.45 | 248500 | 2.4761 | 0.5375 | | 2.6107 | 335.13 | 249000 | 2.4728 | 0.5379 | | 2.6107 | 335.8 | 249500 | 2.4702 | 0.5386 | | 2.608 | 336.47 | 250000 | 2.4764 | 0.5377 | | 2.608 | 337.15 | 250500 | 2.4738 | 0.5380 | | 2.608 | 337.82 | 251000 | 2.4795 | 0.5371 | | 2.608 | 338.49 | 251500 | 2.4702 | 0.5387 | | 2.608 | 339.17 | 252000 | 2.4823 | 0.5369 | | 2.608 | 339.84 | 252500 | 2.4741 | 0.5382 | | 2.608 | 340.51 | 253000 | 2.4718 | 0.5382 | | 2.608 | 341.18 | 253500 | 2.4731 | 0.5378 | | 2.608 | 341.86 | 254000 | 2.4780 | 0.5373 | | 2.608 | 342.53 | 254500 | 2.4706 | 0.5388 | | 2.6058 | 343.2 | 255000 | 2.4707 | 0.5386 | | 2.6058 | 343.88 | 255500 | 2.4725 | 0.5380 | | 2.6058 | 344.55 | 256000 | 2.4744 | 0.5382 | | 2.6058 | 345.22 | 256500 | 2.4766 | 0.5374 | | 2.6058 | 345.89 | 257000 | 2.4736 | 0.5378 | | 2.6058 | 346.57 | 257500 | 2.4731 | 0.5383 | | 2.6058 | 347.24 | 258000 | 2.4754 | 0.5377 | | 2.6058 | 347.91 | 258500 | 2.4749 | 0.5382 | | 2.6058 | 348.59 | 259000 | 2.4735 | 0.5378 | | 2.6058 | 349.26 | 259500 | 2.4716 | 0.5384 | | 2.6027 | 349.93 | 260000 | 2.4726 | 0.5378 | | 2.6027 | 350.61 | 260500 | 2.4733 | 0.5378 | | 2.6027 | 351.28 | 261000 | 2.4698 | 0.5386 | | 2.6027 | 351.95 | 261500 | 2.4702 | 0.5388 | | 2.6027 | 352.62 | 262000 | 2.4673 | 0.5390 | | 2.6027 | 353.3 | 262500 | 2.4683 | 0.5390 | | 2.6027 | 353.97 | 263000 | 2.4739 | 0.5379 | | 2.6027 | 354.64 | 263500 | 2.4743 | 0.5382 | | 2.6027 | 355.32 | 264000 | 2.4694 | 0.5388 | | 2.6027 | 355.99 | 264500 | 2.4671 | 0.5391 | | 2.6009 | 356.66 | 265000 | 2.4747 | 0.5383 | | 2.6009 | 357.34 | 265500 | 2.4703 | 0.5382 | | 2.6009 | 358.01 | 266000 | 2.4695 | 0.5388 | | 2.6009 | 358.68 | 266500 | 2.4725 | 0.5380 | | 2.6009 | 359.35 | 267000 | 2.4639 | 0.5397 | | 2.6009 | 360.03 | 267500 | 2.4686 | 0.5385 | | 2.6009 | 360.7 | 268000 | 2.4698 | 0.5386 | | 2.6009 | 361.37 | 268500 | 2.4694 | 0.5386 | | 2.6009 | 362.05 | 269000 | 2.4680 | 0.5390 | | 2.6009 | 362.72 | 269500 | 2.4728 | 0.5383 | | 2.5989 | 363.39 | 270000 | 2.4697 | 0.5385 | | 2.5989 | 364.06 | 270500 | 2.4701 | 0.5387 | | 2.5989 | 364.74 | 271000 | 2.4702 | 0.5387 | | 2.5989 | 365.41 | 271500 | 2.4687 | 0.5390 | | 2.5989 | 366.08 | 272000 | 2.4725 | 0.5382 | | 2.5989 | 366.76 | 272500 | 2.4673 | 0.5384 | | 2.5989 | 367.43 | 273000 | 2.4659 | 0.5390 | | 2.5989 | 368.1 | 273500 | 2.4686 | 0.5389 | | 2.5989 | 368.78 | 274000 | 2.4677 | 0.5382 | | 2.5989 | 369.45 | 274500 | 2.4632 | 0.5389 | | 2.5977 | 370.12 | 275000 | 2.4676 | 0.5385 | | 2.5977 | 370.79 | 275500 | 2.4699 | 0.5388 | | 2.5977 | 371.47 | 276000 | 2.4629 | 0.5394 | | 2.5977 | 372.14 | 276500 | 2.4720 | 0.5380 | | 2.5977 | 372.81 | 277000 | 2.4678 | 0.5391 | | 2.5977 | 373.49 | 277500 | 2.4643 | 0.5396 | | 2.5977 | 374.16 | 278000 | 2.4654 | 0.5395 | | 2.5977 | 374.83 | 278500 | 2.4645 | 0.5395 | | 2.5977 | 375.5 | 279000 | 2.4649 | 0.5391 | | 2.5977 | 376.18 | 279500 | 2.4639 | 0.5392 | | 2.5961 | 376.85 | 280000 | 2.4659 | 0.5389 | | 2.5961 | 377.52 | 280500 | 2.4681 | 0.5385 | | 2.5961 | 378.2 | 281000 | 2.4641 | 0.5390 | | 2.5961 | 378.87 | 281500 | 2.4658 | 0.5393 | | 2.5961 | 379.54 | 282000 | 2.4687 | 0.5388 | | 2.5961 | 380.22 | 282500 | 2.4690 | 0.5385 | | 2.5961 | 380.89 | 283000 | 2.4679 | 0.5391 | | 2.5961 | 381.56 | 283500 | 2.4612 | 0.5395 | | 2.5961 | 382.23 | 284000 | 2.4624 | 0.5395 | | 2.5961 | 382.91 | 284500 | 2.4668 | 0.5390 | | 2.5947 | 383.58 | 285000 | 2.4663 | 0.5389 | | 2.5947 | 384.25 | 285500 | 2.4654 | 0.5387 | | 2.5947 | 384.93 | 286000 | 2.4708 | 0.5385 | | 2.5947 | 385.6 | 286500 | 2.4669 | 0.5388 | | 2.5947 | 386.27 | 287000 | 2.4612 | 0.5396 | | 2.5947 | 386.94 | 287500 | 2.4666 | 0.5392 | | 2.5947 | 387.62 | 288000 | 2.4653 | 0.5393 | | 2.5947 | 388.29 | 288500 | 2.4666 | 0.5390 | | 2.5947 | 388.96 | 289000 | 2.4684 | 0.5388 | | 2.5947 | 389.64 | 289500 | 2.4660 | 0.5394 | | 2.5936 | 390.31 | 290000 | 2.4642 | 0.5395 | | 2.5936 | 390.98 | 290500 | 2.4627 | 0.5403 | | 2.5936 | 391.66 | 291000 | 2.4683 | 0.5389 | | 2.5936 | 392.33 | 291500 | 2.4667 | 0.5387 | | 2.5936 | 393.0 | 292000 | 2.4660 | 0.5389 | | 2.5936 | 393.67 | 292500 | 2.4673 | 0.5390 | | 2.5936 | 394.35 | 293000 | 2.4645 | 0.5391 | | 2.5936 | 395.02 | 293500 | 2.4693 | 0.5389 | | 2.5936 | 395.69 | 294000 | 2.4692 | 0.5385 | | 2.5936 | 396.37 | 294500 | 2.4653 | 0.5385 | | 2.5934 | 397.04 | 295000 | 2.4661 | 0.5390 | | 2.5934 | 397.71 | 295500 | 2.4630 | 0.5394 | | 2.5934 | 398.38 | 296000 | 2.4641 | 0.5390 | | 2.5934 | 399.06 | 296500 | 2.4636 | 0.5392 | | 2.5934 | 399.73 | 297000 | 2.4650 | 0.5392 | ### Framework versions - Transformers 4.20.1 - Pytorch 1.12.0+cu102 - Datasets 2.3.2 - Tokenizers 0.12.1 ### Additional configurations ``` data: dataset_name: cc100 lang: nl overwrite_cache: False validation_split_percentage: 5 max_seq_length: 512 preprocessing_num_workers: 8 mlm_probability: 0.15 line_by_line: False pad_to_max_length: False max_train_samples: -1 max_eval_samples: -1 training: do_train: True do_eval: True do_predict: True resume_from_checkpoint: False evaluation_strategy: steps eval_steps: 500 per_device_train_batch_size: 16 per_device_eval_batch_size: 16 gradient_accumulation_steps: 32 eval_accumulation_steps: 1 learning_rate: 5e-5 weight_decay: 0.0 adam_beta1: 0.9 adam_beta2: 0.999 adam_epsilon: 1e-8 max_grad_norm: 1.0 num_train_epochs: 400.0 lr_scheduler_type: linear fp16: False warmup_steps: 8000 seed: 703 ```