--- library_name: peft license: mit base_model: FacebookAI/xlm-roberta-large tags: - generated_from_trainer datasets: - biobert_json metrics: - precision - recall - f1 - accuracy model-index: - name: xml-roberta-large-ner-qlorafinetune-runs-colab-32size results: [] --- # xml-roberta-large-ner-qlorafinetune-runs-colab-32size This model is a fine-tuned version of [FacebookAI/xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) on the biobert_json dataset. It achieves the following results on the evaluation set: - Loss: 0.0784 - Precision: 0.9381 - Recall: 0.9575 - F1: 0.9477 - Accuracy: 0.9813 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0004 - train_batch_size: 32 - eval_batch_size: 32 - seed: 42 - optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - training_steps: 2141 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1 | Accuracy | |:-------------:|:------:|:----:|:---------------:|:---------:|:------:|:------:|:--------:| | 2.3478 | 0.0654 | 20 | 1.3438 | 0.0 | 0.0 | 0.0 | 0.7180 | | 1.2529 | 0.1307 | 40 | 0.8221 | 0.4886 | 0.4719 | 0.4801 | 0.8204 | | 0.741 | 0.1961 | 60 | 0.4377 | 0.7634 | 0.6193 | 0.6838 | 0.8838 | | 0.4179 | 0.2614 | 80 | 0.2434 | 0.8013 | 0.7903 | 0.7958 | 0.9309 | | 0.301 | 0.3268 | 100 | 0.1965 | 0.8393 | 0.8788 | 0.8586 | 0.9477 | | 0.2704 | 0.3922 | 120 | 0.1662 | 0.8354 | 0.8823 | 0.8583 | 0.9494 | | 0.2128 | 0.4575 | 140 | 0.1408 | 0.9054 | 0.8776 | 0.8913 | 0.9616 | | 0.1851 | 0.5229 | 160 | 0.1298 | 0.8848 | 0.8979 | 0.8913 | 0.9636 | | 0.1812 | 0.5882 | 180 | 0.1330 | 0.8804 | 0.9242 | 0.9018 | 0.9620 | | 0.1492 | 0.6536 | 200 | 0.1161 | 0.8742 | 0.9313 | 0.9018 | 0.9639 | | 0.1388 | 0.7190 | 220 | 0.0979 | 0.9238 | 0.9124 | 0.9180 | 0.9707 | | 0.1358 | 0.7843 | 240 | 0.1196 | 0.8820 | 0.9563 | 0.9177 | 0.9673 | | 0.115 | 0.8497 | 260 | 0.0878 | 0.9164 | 0.9443 | 0.9302 | 0.9737 | | 0.1177 | 0.9150 | 280 | 0.1110 | 0.8822 | 0.9358 | 0.9082 | 0.9660 | | 0.1332 | 0.9804 | 300 | 0.0890 | 0.9152 | 0.9253 | 0.9202 | 0.9726 | | 0.1079 | 1.0458 | 320 | 0.1022 | 0.8865 | 0.9379 | 0.9115 | 0.9682 | | 0.0889 | 1.1111 | 340 | 0.1028 | 0.9014 | 0.9475 | 0.9238 | 0.9685 | | 0.1061 | 1.1765 | 360 | 0.0958 | 0.9017 | 0.9520 | 0.9262 | 0.9709 | | 0.1032 | 1.2418 | 380 | 0.0795 | 0.9270 | 0.9526 | 0.9396 | 0.9777 | | 0.1034 | 1.3072 | 400 | 0.0775 | 0.9232 | 0.9520 | 0.9374 | 0.9775 | | 0.1124 | 1.3725 | 420 | 0.0888 | 0.8985 | 0.9343 | 0.9160 | 0.9725 | | 0.0936 | 1.4379 | 440 | 0.0870 | 0.9214 | 0.9466 | 0.9338 | 0.9756 | | 0.0935 | 1.5033 | 460 | 0.0747 | 0.9352 | 0.9472 | 0.9412 | 0.9777 | | 0.1005 | 1.5686 | 480 | 0.0993 | 0.9050 | 0.9579 | 0.9307 | 0.9699 | | 0.0903 | 1.6340 | 500 | 0.0887 | 0.9067 | 0.9466 | 0.9262 | 0.9734 | | 0.0817 | 1.6993 | 520 | 0.0775 | 0.9179 | 0.9588 | 0.9379 | 0.9780 | | 0.0825 | 1.7647 | 540 | 0.0784 | 0.9250 | 0.9538 | 0.9392 | 0.9785 | | 0.0921 | 1.8301 | 560 | 0.0862 | 0.9108 | 0.9635 | 0.9364 | 0.9758 | | 0.0918 | 1.8954 | 580 | 0.0805 | 0.9194 | 0.9544 | 0.9366 | 0.9757 | | 0.0888 | 1.9608 | 600 | 0.0819 | 0.9271 | 0.9533 | 0.9400 | 0.9770 | | 0.0769 | 2.0261 | 620 | 0.0764 | 0.9323 | 0.9552 | 0.9436 | 0.9790 | | 0.0683 | 2.0915 | 640 | 0.0747 | 0.9261 | 0.9579 | 0.9417 | 0.9789 | | 0.0533 | 2.1569 | 660 | 0.0708 | 0.9309 | 0.9497 | 0.9402 | 0.9788 | | 0.0804 | 2.2222 | 680 | 0.0732 | 0.9263 | 0.9580 | 0.9419 | 0.9783 | | 0.0725 | 2.2876 | 700 | 0.0792 | 0.9130 | 0.9484 | 0.9304 | 0.9767 | | 0.0562 | 2.3529 | 720 | 0.0697 | 0.9337 | 0.9586 | 0.9460 | 0.9804 | | 0.0627 | 2.4183 | 740 | 0.0756 | 0.9349 | 0.9519 | 0.9433 | 0.9793 | | 0.0813 | 2.4837 | 760 | 0.0740 | 0.9332 | 0.9587 | 0.9458 | 0.9805 | | 0.0662 | 2.5490 | 780 | 0.0738 | 0.9373 | 0.9577 | 0.9474 | 0.9811 | | 0.0728 | 2.6144 | 800 | 0.0706 | 0.9375 | 0.9576 | 0.9475 | 0.9804 | | 0.0606 | 2.6797 | 820 | 0.0728 | 0.9415 | 0.9478 | 0.9446 | 0.9798 | | 0.0683 | 2.7451 | 840 | 0.0717 | 0.9315 | 0.9496 | 0.9405 | 0.9797 | | 0.0614 | 2.8105 | 860 | 0.0685 | 0.9392 | 0.9540 | 0.9466 | 0.9810 | | 0.063 | 2.8758 | 880 | 0.0765 | 0.9287 | 0.9576 | 0.9430 | 0.9787 | | 0.06 | 2.9412 | 900 | 0.0750 | 0.9280 | 0.9579 | 0.9427 | 0.9790 | | 0.0605 | 3.0065 | 920 | 0.0749 | 0.9304 | 0.9616 | 0.9457 | 0.9802 | | 0.0475 | 3.0719 | 940 | 0.0774 | 0.9249 | 0.9610 | 0.9426 | 0.9797 | | 0.0543 | 3.1373 | 960 | 0.0845 | 0.9282 | 0.9581 | 0.9429 | 0.9779 | | 0.0479 | 3.2026 | 980 | 0.0746 | 0.9303 | 0.9520 | 0.9410 | 0.9786 | | 0.0575 | 3.2680 | 1000 | 0.0717 | 0.9370 | 0.9566 | 0.9467 | 0.9806 | | 0.0498 | 3.3333 | 1020 | 0.0765 | 0.9352 | 0.9484 | 0.9418 | 0.9788 | | 0.0539 | 3.3987 | 1040 | 0.0790 | 0.9245 | 0.9667 | 0.9451 | 0.9802 | | 0.0583 | 3.4641 | 1060 | 0.0873 | 0.9152 | 0.9532 | 0.9338 | 0.9752 | | 0.0557 | 3.5294 | 1080 | 0.0699 | 0.9316 | 0.9554 | 0.9433 | 0.9806 | | 0.0507 | 3.5948 | 1100 | 0.0820 | 0.9209 | 0.9569 | 0.9385 | 0.9766 | | 0.0472 | 3.6601 | 1120 | 0.0783 | 0.9290 | 0.9570 | 0.9428 | 0.9797 | | 0.0685 | 3.7255 | 1140 | 0.0760 | 0.9219 | 0.9597 | 0.9404 | 0.9802 | | 0.055 | 3.7908 | 1160 | 0.0691 | 0.9395 | 0.9601 | 0.9497 | 0.9824 | | 0.0645 | 3.8562 | 1180 | 0.0810 | 0.9271 | 0.9569 | 0.9418 | 0.9783 | | 0.0593 | 3.9216 | 1200 | 0.0794 | 0.9320 | 0.9563 | 0.9440 | 0.9796 | | 0.0646 | 3.9869 | 1220 | 0.0679 | 0.9385 | 0.9610 | 0.9496 | 0.9820 | | 0.0443 | 4.0523 | 1240 | 0.0727 | 0.9332 | 0.9575 | 0.9452 | 0.9809 | | 0.0358 | 4.1176 | 1260 | 0.0722 | 0.9305 | 0.9552 | 0.9427 | 0.9808 | | 0.0403 | 4.1830 | 1280 | 0.0676 | 0.9420 | 0.9629 | 0.9524 | 0.9825 | | 0.0503 | 4.2484 | 1300 | 0.0732 | 0.9334 | 0.9599 | 0.9465 | 0.9805 | | 0.0439 | 4.3137 | 1320 | 0.0704 | 0.9429 | 0.9524 | 0.9476 | 0.9809 | | 0.0454 | 4.3791 | 1340 | 0.0702 | 0.9374 | 0.9572 | 0.9472 | 0.9816 | | 0.0388 | 4.4444 | 1360 | 0.0796 | 0.9267 | 0.9558 | 0.9410 | 0.9787 | | 0.039 | 4.5098 | 1380 | 0.0754 | 0.9305 | 0.9597 | 0.9449 | 0.9806 | | 0.0461 | 4.5752 | 1400 | 0.0723 | 0.9350 | 0.9544 | 0.9446 | 0.9813 | | 0.0471 | 4.6405 | 1420 | 0.0737 | 0.9318 | 0.9576 | 0.9445 | 0.9806 | | 0.0371 | 4.7059 | 1440 | 0.0789 | 0.9290 | 0.9528 | 0.9408 | 0.9796 | | 0.0479 | 4.7712 | 1460 | 0.0761 | 0.9287 | 0.9569 | 0.9426 | 0.9805 | | 0.0436 | 4.8366 | 1480 | 0.0779 | 0.9313 | 0.9501 | 0.9406 | 0.9792 | | 0.0446 | 4.9020 | 1500 | 0.0741 | 0.9387 | 0.9607 | 0.9496 | 0.9815 | | 0.0486 | 4.9673 | 1520 | 0.0765 | 0.9299 | 0.9526 | 0.9411 | 0.9796 | | 0.0364 | 5.0327 | 1540 | 0.0768 | 0.9351 | 0.9568 | 0.9458 | 0.9804 | | 0.0359 | 5.0980 | 1560 | 0.0736 | 0.9399 | 0.9591 | 0.9494 | 0.9819 | | 0.0407 | 5.1634 | 1580 | 0.0726 | 0.9413 | 0.9601 | 0.9506 | 0.9824 | | 0.0351 | 5.2288 | 1600 | 0.0797 | 0.9249 | 0.9543 | 0.9394 | 0.9789 | | 0.0318 | 5.2941 | 1620 | 0.0734 | 0.9393 | 0.9599 | 0.9495 | 0.9812 | | 0.0312 | 5.3595 | 1640 | 0.0771 | 0.9366 | 0.9601 | 0.9482 | 0.9810 | | 0.0362 | 5.4248 | 1660 | 0.0745 | 0.9387 | 0.9502 | 0.9444 | 0.9807 | | 0.0436 | 5.4902 | 1680 | 0.0788 | 0.9299 | 0.9552 | 0.9424 | 0.9802 | | 0.0397 | 5.5556 | 1700 | 0.0740 | 0.9389 | 0.9595 | 0.9491 | 0.9818 | | 0.0378 | 5.6209 | 1720 | 0.0793 | 0.9312 | 0.9522 | 0.9416 | 0.9795 | | 0.037 | 5.6863 | 1740 | 0.0753 | 0.9398 | 0.9551 | 0.9474 | 0.9809 | | 0.0352 | 5.7516 | 1760 | 0.0807 | 0.9252 | 0.9521 | 0.9385 | 0.9791 | | 0.0279 | 5.8170 | 1780 | 0.0769 | 0.9373 | 0.9587 | 0.9479 | 0.9810 | | 0.037 | 5.8824 | 1800 | 0.0809 | 0.9304 | 0.9526 | 0.9414 | 0.9791 | | 0.031 | 5.9477 | 1820 | 0.0741 | 0.9414 | 0.9609 | 0.9510 | 0.9821 | | 0.0342 | 6.0131 | 1840 | 0.0794 | 0.9353 | 0.9558 | 0.9455 | 0.9803 | | 0.0295 | 6.0784 | 1860 | 0.0785 | 0.9364 | 0.9531 | 0.9447 | 0.9801 | | 0.0256 | 6.1438 | 1880 | 0.0804 | 0.9282 | 0.9512 | 0.9395 | 0.9794 | | 0.0203 | 6.2092 | 1900 | 0.0800 | 0.9321 | 0.9507 | 0.9413 | 0.9799 | | 0.0324 | 6.2745 | 1920 | 0.0788 | 0.9366 | 0.9568 | 0.9466 | 0.9811 | | 0.0295 | 6.3399 | 1940 | 0.0796 | 0.9352 | 0.9570 | 0.9460 | 0.9809 | | 0.0251 | 6.4052 | 1960 | 0.0780 | 0.9382 | 0.9558 | 0.9469 | 0.9814 | | 0.0274 | 6.4706 | 1980 | 0.0810 | 0.9321 | 0.9524 | 0.9421 | 0.9801 | | 0.0279 | 6.5359 | 2000 | 0.0789 | 0.9379 | 0.9583 | 0.9480 | 0.9813 | | 0.0317 | 6.6013 | 2020 | 0.0775 | 0.9394 | 0.9569 | 0.9481 | 0.9816 | | 0.0248 | 6.6667 | 2040 | 0.0788 | 0.9352 | 0.9542 | 0.9446 | 0.9807 | | 0.0273 | 6.7320 | 2060 | 0.0789 | 0.9358 | 0.9557 | 0.9456 | 0.9810 | | 0.0351 | 6.7974 | 2080 | 0.0777 | 0.9381 | 0.9573 | 0.9476 | 0.9814 | | 0.0236 | 6.8627 | 2100 | 0.0782 | 0.9378 | 0.9569 | 0.9473 | 0.9811 | | 0.0257 | 6.9281 | 2120 | 0.0787 | 0.9375 | 0.9573 | 0.9473 | 0.9811 | | 0.0254 | 6.9935 | 2140 | 0.0784 | 0.9381 | 0.9575 | 0.9477 | 0.9813 | ### Framework versions - PEFT 0.13.2 - Transformers 4.46.3 - Pytorch 2.5.1+cu121 - Datasets 3.2.0 - Tokenizers 0.20.3