| license: mit | |
| # roberta-large-movies | |
| This model is a fine-tuned version of [roberta-large](https://huggingface.co/roberta-large) on the movie competition dataset. | |
| link: https://huggingface.co/spaces/competitions/movie-genre-prediction | |
| This model is nased on a MLM (Mask language modeling) finetuning. The goal is to apply a domain transfer. It needs then to be finetuned on labels. | |
| It achieves the following results on the evaluation set: | |
| - Loss: 1.3261 | |
| - Accuracy: 0.7375 | |
| ## Model description | |
| roberta-large | |
| ## Training procedure | |
| ### Training hyperparameters | |
| The following hyperparameters were used during training: | |
| - learning_rate: 5e-05 | |
| - train_batch_size: 32 | |
| - eval_batch_size: 16 | |
| - seed: 42 | |
| - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 | |
| - lr_scheduler_type: linear | |
| - num_epochs: 30.0 | |
| - mixed_precision_training: Native AMP | |
| ### Training results | |
| | Training Loss | Epoch | Step | Validation Loss | Accuracy | | |
| |:-------------:|:-----:|:-----:|:---------------:|:--------:| | |
| | 1.7698 | 0.18 | 500 | 1.6168 | 0.6738 | | |
| | 1.7761 | 0.36 | 1000 | 1.6522 | 0.6830 | | |
| | 1.7626 | 0.54 | 1500 | 1.6534 | 0.6660 | | |
| | 1.7602 | 0.72 | 2000 | 1.6576 | 0.6787 | | |
| | 1.7587 | 0.89 | 2500 | 1.6266 | 0.6773 | | |
| | 1.7047 | 1.07 | 3000 | 1.6060 | 0.6852 | | |
| | 1.6782 | 1.25 | 3500 | 1.5990 | 0.6906 | | |
| | 1.6733 | 1.43 | 4000 | 1.5377 | 0.6967 | | |
| | 1.6664 | 1.61 | 4500 | 1.6435 | 0.6747 | | |
| | 1.6719 | 1.79 | 5000 | 1.4839 | 0.6907 | | |
| | 1.6502 | 1.97 | 5500 | 1.5351 | 0.6897 | | |
| | 1.6233 | 2.15 | 6000 | 1.6818 | 0.6763 | | |
| | 1.6127 | 2.32 | 6500 | 1.5865 | 0.6853 | | |
| | 1.6274 | 2.5 | 7000 | 1.5004 | 0.7004 | | |
| | 1.601 | 2.68 | 7500 | 1.4522 | 0.6930 | | |
| | 1.6123 | 2.86 | 8000 | 1.5371 | 0.6894 | | |
| | 1.6074 | 3.04 | 8500 | 1.5342 | 0.6952 | | |
| | 1.563 | 3.22 | 9000 | 1.5682 | 0.6876 | | |
| | 1.5746 | 3.4 | 9500 | 1.5705 | 0.6958 | | |
| | 1.5539 | 3.58 | 10000 | 1.4711 | 0.7041 | | |
| | 1.578 | 3.75 | 10500 | 1.5466 | 0.6889 | | |
| | 1.5492 | 3.93 | 11000 | 1.4629 | 0.6969 | | |
| | 1.5291 | 4.11 | 11500 | 1.4265 | 0.7200 | | |
| | 1.5079 | 4.29 | 12000 | 1.5053 | 0.6966 | | |
| | 1.5283 | 4.47 | 12500 | 1.5257 | 0.6903 | | |
| | 1.5141 | 4.65 | 13000 | 1.5063 | 0.6950 | | |
| | 1.4979 | 4.83 | 13500 | 1.5636 | 0.6956 | | |
| | 1.5294 | 5.01 | 14000 | 1.5878 | 0.6835 | | |
| | 1.4641 | 5.18 | 14500 | 1.5575 | 0.6962 | | |
| | 1.4754 | 5.36 | 15000 | 1.4779 | 0.7007 | | |
| | 1.4696 | 5.54 | 15500 | 1.4520 | 0.6965 | | |
| | 1.4655 | 5.72 | 16000 | 1.6320 | 0.6830 | | |
| | 1.4792 | 5.9 | 16500 | 1.4152 | 0.7134 | | |
| | 1.4379 | 6.08 | 17000 | 1.4900 | 0.7042 | | |
| | 1.4281 | 6.26 | 17500 | 1.5407 | 0.6990 | | |
| | 1.436 | 6.44 | 18000 | 1.5343 | 0.6914 | | |
| | 1.4342 | 6.61 | 18500 | 1.5324 | 0.7024 | | |
| | 1.4176 | 6.79 | 19000 | 1.4486 | 0.7133 | | |
| | 1.4308 | 6.97 | 19500 | 1.4598 | 0.7032 | | |
| | 1.4014 | 7.15 | 20000 | 1.5750 | 0.6938 | | |
| | 1.3661 | 7.33 | 20500 | 1.5404 | 0.6985 | | |
| | 1.3857 | 7.51 | 21000 | 1.4692 | 0.7037 | | |
| | 1.3846 | 7.69 | 21500 | 1.5511 | 0.6941 | | |
| | 1.3867 | 7.87 | 22000 | 1.5321 | 0.6925 | | |
| | 1.3658 | 8.04 | 22500 | 1.5500 | 0.7021 | | |
| | 1.3406 | 8.22 | 23000 | 1.5239 | 0.6960 | | |
| | 1.3405 | 8.4 | 23500 | 1.4414 | 0.7055 | | |
| | 1.3373 | 8.58 | 24000 | 1.5994 | 0.6784 | | |
| | 1.3527 | 8.76 | 24500 | 1.5106 | 0.6970 | | |
| | 1.3436 | 8.94 | 25000 | 1.4714 | 0.7080 | | |
| | 1.3069 | 9.12 | 25500 | 1.4990 | 0.6953 | | |
| | 1.2969 | 9.3 | 26000 | 1.4810 | 0.6964 | | |
| | 1.3009 | 9.47 | 26500 | 1.5965 | 0.6876 | | |
| | 1.3227 | 9.65 | 27000 | 1.4296 | 0.7014 | | |
| | 1.3259 | 9.83 | 27500 | 1.4137 | 0.7189 | | |
| | 1.3131 | 10.01 | 28000 | 1.5342 | 0.7020 | | |
| | 1.271 | 10.19 | 28500 | 1.4708 | 0.7113 | | |
| | 1.2684 | 10.37 | 29000 | 1.4342 | 0.7046 | | |
| | 1.2767 | 10.55 | 29500 | 1.4703 | 0.7094 | | |
| | 1.2861 | 10.73 | 30000 | 1.3323 | 0.7309 | | |
| | 1.2617 | 10.9 | 30500 | 1.4562 | 0.7003 | | |
| | 1.2551 | 11.08 | 31000 | 1.4361 | 0.7170 | | |
| | 1.2404 | 11.26 | 31500 | 1.4537 | 0.7035 | | |
| | 1.2562 | 11.44 | 32000 | 1.4039 | 0.7132 | | |
| | 1.2489 | 11.62 | 32500 | 1.4372 | 0.7064 | | |
| | 1.2406 | 11.8 | 33000 | 1.4926 | 0.7087 | | |
| | 1.2285 | 11.98 | 33500 | 1.4080 | 0.7152 | | |
| | 1.2213 | 12.16 | 34000 | 1.4031 | 0.7170 | | |
| | 1.1998 | 12.33 | 34500 | 1.3541 | 0.7223 | | |
| | 1.2184 | 12.51 | 35000 | 1.3630 | 0.7308 | | |
| | 1.2195 | 12.69 | 35500 | 1.3125 | 0.7281 | | |
| | 1.2178 | 12.87 | 36000 | 1.4257 | 0.7119 | | |
| | 1.1918 | 13.05 | 36500 | 1.4108 | 0.7153 | | |
| | 1.1664 | 13.23 | 37000 | 1.3577 | 0.7227 | | |
| | 1.1754 | 13.41 | 37500 | 1.3777 | 0.7206 | | |
| | 1.1855 | 13.59 | 38000 | 1.3501 | 0.7354 | | |
| | 1.1644 | 13.76 | 38500 | 1.3747 | 0.7207 | | |
| | 1.1709 | 13.94 | 39000 | 1.3704 | 0.7184 | | |
| | 1.1613 | 14.12 | 39500 | 1.4307 | 0.7247 | | |
| | 1.1443 | 14.3 | 40000 | 1.3190 | 0.7221 | | |
| | 1.1356 | 14.48 | 40500 | 1.3288 | 0.7331 | | |
| | 1.1493 | 14.66 | 41000 | 1.3505 | 0.7240 | | |
| | 1.1417 | 14.84 | 41500 | 1.3146 | 0.7320 | | |
| | 1.1349 | 15.02 | 42000 | 1.3546 | 0.7333 | | |
| | 1.1169 | 15.19 | 42500 | 1.3709 | 0.7247 | | |
| | 1.1187 | 15.37 | 43000 | 1.4243 | 0.7218 | | |
| | 1.118 | 15.55 | 43500 | 1.3835 | 0.7264 | | |
| | 1.1165 | 15.73 | 44000 | 1.3240 | 0.7254 | | |
| | 1.114 | 15.91 | 44500 | 1.3264 | 0.7382 | | |
| | 1.105 | 16.09 | 45000 | 1.3214 | 0.7334 | | |
| | 1.0924 | 16.27 | 45500 | 1.3847 | 0.7282 | | |
| | 1.0915 | 16.45 | 46000 | 1.3604 | 0.7317 | | |
| | 1.0968 | 16.62 | 46500 | 1.3540 | 0.7319 | | |
| | 1.0772 | 16.8 | 47000 | 1.2475 | 0.7306 | | |
| | 1.0975 | 16.98 | 47500 | 1.2636 | 0.7448 | | |
| | 1.0708 | 17.16 | 48000 | 1.4056 | 0.7182 | | |
| | 1.0654 | 17.34 | 48500 | 1.3769 | 0.7276 | | |
| | 1.0676 | 17.52 | 49000 | 1.3357 | 0.7224 | | |
| | 1.0507 | 17.7 | 49500 | 1.4088 | 0.7124 | | |
| | 1.0424 | 17.88 | 50000 | 1.3146 | 0.7315 | | |
| | 1.0524 | 18.06 | 50500 | 1.2896 | 0.7393 | | |
| | 1.0349 | 18.23 | 51000 | 1.3987 | 0.7192 | | |
| | 1.0217 | 18.41 | 51500 | 1.2938 | 0.7381 | | |
| | 1.0238 | 18.59 | 52000 | 1.2962 | 0.7387 | | |
| | 1.0292 | 18.77 | 52500 | 1.3195 | 0.7371 | | |
| | 1.0426 | 18.95 | 53000 | 1.2835 | 0.7412 | | |
| | 1.0196 | 19.13 | 53500 | 1.2346 | 0.7473 | | |
| | 1.012 | 19.31 | 54000 | 1.3666 | 0.7338 | | |
| | 1.0256 | 19.49 | 54500 | 1.3140 | 0.7365 | | |
| | 0.9824 | 19.66 | 55000 | 1.2764 | 0.7416 | | |
| | 1.0048 | 19.84 | 55500 | 1.2514 | 0.7488 | | |
| | 0.9947 | 20.02 | 56000 | 1.3351 | 0.7432 | | |
| | 0.977 | 20.2 | 56500 | 1.2854 | 0.7451 | | |
| | 0.9862 | 20.38 | 57000 | 1.3666 | 0.7285 | | |
| | 0.9699 | 20.56 | 57500 | 1.3123 | 0.7348 | | |
| | 0.977 | 20.74 | 58000 | 1.3426 | 0.7255 | | |
| | 0.9749 | 20.92 | 58500 | 1.3763 | 0.7297 | | |
| | 0.9505 | 21.09 | 59000 | 1.2372 | 0.7434 | | |
| | 0.9438 | 21.27 | 59500 | 1.4334 | 0.7159 | | |
| | 0.944 | 21.45 | 60000 | 1.2690 | 0.7508 | | |
| | 0.9427 | 21.63 | 60500 | 1.2186 | 0.7486 | | |
| | 0.9553 | 21.81 | 61000 | 1.3941 | 0.7269 | | |
| | 0.9571 | 21.99 | 61500 | 1.4163 | 0.7274 | | |
| | 0.932 | 22.17 | 62000 | 1.2717 | 0.7523 | | |
| | 0.9166 | 22.35 | 62500 | 1.2177 | 0.7396 | | |
| | 0.9301 | 22.52 | 63000 | 1.3264 | 0.7378 | | |
| | 0.9351 | 22.7 | 63500 | 1.2570 | 0.7520 | | |
| | 0.9211 | 22.88 | 64000 | 1.2639 | 0.75 | | |
| | 0.9211 | 23.06 | 64500 | 1.2377 | 0.7606 | | |
| | 0.9196 | 23.24 | 65000 | 1.2739 | 0.7485 | | |
| | 0.9062 | 23.42 | 65500 | 1.3263 | 0.7365 | | |
| | 0.8965 | 23.6 | 66000 | 1.2814 | 0.7455 | | |
| | 0.9004 | 23.78 | 66500 | 1.2109 | 0.7562 | | |
| | 0.9094 | 23.95 | 67000 | 1.2629 | 0.7528 | | |
| | 0.8937 | 24.13 | 67500 | 1.2771 | 0.7375 | | |
| | 0.8711 | 24.31 | 68000 | 1.3746 | 0.7353 | | |
| | 0.8972 | 24.49 | 68500 | 1.2529 | 0.7454 | | |
| | 0.8863 | 24.67 | 69000 | 1.3219 | 0.7359 | | |
| | 0.8823 | 24.85 | 69500 | 1.3136 | 0.7367 | | |
| | 0.8759 | 25.03 | 70000 | 1.3152 | 0.7428 | | |
| | 0.8722 | 25.21 | 70500 | 1.3108 | 0.7570 | | |
| | 0.8548 | 25.38 | 71000 | 1.3503 | 0.7368 | | |
| | 0.8728 | 25.56 | 71500 | 1.3091 | 0.7403 | | |
| | 0.8633 | 25.74 | 72000 | 1.2952 | 0.7416 | | |
| | 0.8612 | 25.92 | 72500 | 1.1612 | 0.7719 | | |
| | 0.8677 | 26.1 | 73000 | 1.2855 | 0.7450 | | |
| | 0.8526 | 26.28 | 73500 | 1.2979 | 0.7545 | | |
| | 0.8594 | 26.46 | 74000 | 1.2570 | 0.7598 | | |
| | 0.8481 | 26.64 | 74500 | 1.2337 | 0.7492 | | |
| | 0.855 | 26.81 | 75000 | 1.2875 | 0.7444 | | |
| | 0.835 | 26.99 | 75500 | 1.2270 | 0.7585 | | |
| | 0.8309 | 27.17 | 76000 | 1.2540 | 0.7389 | | |
| | 0.8326 | 27.35 | 76500 | 1.3611 | 0.7375 | | |
| | 0.8398 | 27.53 | 77000 | 1.2248 | 0.7505 | | |
| | 0.8304 | 27.71 | 77500 | 1.2403 | 0.7607 | | |
| | 0.8373 | 27.89 | 78000 | 1.1709 | 0.7611 | | |
| | 0.8462 | 28.07 | 78500 | 1.2891 | 0.7508 | | |
| | 0.8259 | 28.24 | 79000 | 1.2452 | 0.7501 | | |
| | 0.8334 | 28.42 | 79500 | 1.2986 | 0.7468 | | |
| | 0.8115 | 28.6 | 80000 | 1.2880 | 0.7515 | | |
| | 0.8205 | 28.78 | 80500 | 1.2728 | 0.7562 | | |
| | 0.8261 | 28.96 | 81000 | 1.2661 | 0.7524 | | |
| | 0.8299 | 29.14 | 81500 | 1.2592 | 0.7486 | | |
| | 0.8276 | 29.32 | 82000 | 1.2325 | 0.7530 | | |
| | 0.8112 | 29.5 | 82500 | 1.3154 | 0.7478 | | |
| | 0.8111 | 29.67 | 83000 | 1.3343 | 0.7405 | | |
| | 0.8148 | 29.85 | 83500 | 1.2806 | 0.7485 | | |
| ### Framework versions | |
| - Transformers 4.21.3 | |
| - Pytorch 1.12.1+cu116 | |
| - Datasets 2.4.0 | |
| - Tokenizers 0.12.1 | |