longformer-sep_tok

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

• Loss: 0.6673
• Claim: {'precision': 0.6793010091065715, 'recall': 0.6475832942280619, 'f1-score': 0.663063063063063, 'support': 4262.0}
• Majorclaim: {'precision': 0.9517884914463453, 'recall': 0.8480369515011548, 'f1-score': 0.8969223253541768, 'support': 2165.0}
• O: {'precision': 0.9979434024350116, 'recall': 0.9997527608373167, 'f1-score': 0.9988472622478387, 'support': 12134.0}
• Premise: {'precision': 0.8936961046684508, 'recall': 0.922003221105913, 'f1-score': 0.9076290060775357, 'support': 13039.0}
• Accuracy: 0.9098
• Macro avg: {'precision': 0.8806822519140949, 'recall': 0.8543440569181115, 'f1-score': 0.8666154141856536, 'support': 31600.0}
• Weighted avg: {'precision': 0.908789611984554, 'recall': 0.9097784810126582, 'f1-score': 0.9089366740356591, 'support': 31600.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Claim	Majorclaim	O	Premise	Accuracy	Macro avg	Weighted avg
No log	1.0	81	0.3209	{'precision': 0.5136306192453511, 'recall': 0.6675269826372595, 'f1-score': 0.580553004795429, 'support': 4262.0}	{'precision': 0.7583745194947831, 'recall': 0.6378752886836028, 'f1-score': 0.6929252383341696, 'support': 2165.0}	{'precision': 0.9967861557478368, 'recall': 0.996868303939344, 'f1-score': 0.9968272281511393, 'support': 12134.0}	{'precision': 0.9107806691449815, 'recall': 0.8455403021704119, 'f1-score': 0.8769487750556793, 'support': 13039.0}	0.8654	{'precision': 0.7948929909082382, 'recall': 0.7869527193576545, 'f1-score': 0.7868135615841043, 'support': 31600.0}	{'precision': 0.8797989523023909, 'recall': 0.8654113924050633, 'f1-score': 0.8703967313850799, 'support': 31600.0}
No log	2.0	162	0.2567	{'precision': 0.6369094231271208, 'recall': 0.5725011731581418, 'f1-score': 0.6029902384776968, 'support': 4262.0}	{'precision': 0.8129562043795621, 'recall': 0.8230946882217091, 'f1-score': 0.8179940325912326, 'support': 2165.0}	{'precision': 0.9972866304884065, 'recall': 0.9995879347288611, 'f1-score': 0.9984359565360553, 'support': 12134.0}	{'precision': 0.8891539321654864, 'recall': 0.9147940793005599, 'f1-score': 0.9017917895214335, 'support': 13039.0}	0.8949	{'precision': 0.8340765475401439, 'recall': 0.8274944688523179, 'f1-score': 0.8303030042816045, 'support': 31600.0}	{'precision': 0.8914339316361279, 'recall': 0.8949050632911393, 'f1-score': 0.8928603328205832, 'support': 31600.0}
No log	3.0	243	0.2675	{'precision': 0.6579241071428571, 'recall': 0.5532613796339747, 'f1-score': 0.6010706092276319, 'support': 4262.0}	{'precision': 0.8917869034406215, 'recall': 0.7422632794457275, 'f1-score': 0.8101840181497353, 'support': 2165.0}	{'precision': 0.9967118783394986, 'recall': 0.9992582825119499, 'f1-score': 0.997983456109305, 'support': 12134.0}	{'precision': 0.8758630507509432, 'recall': 0.9437073395199018, 'f1-score': 0.9085203780271707, 'support': 13039.0}	0.8986	{'precision': 0.8555714849184801, 'recall': 0.8096225702778885, 'f1-score': 0.8294396153784607, 'support': 31600.0}	{'precision': 0.8939642861109123, 'recall': 0.8985759493670886, 'f1-score': 0.894668981055346, 'support': 31600.0}
No log	4.0	324	0.2528	{'precision': 0.6399294843543412, 'recall': 0.681370248709526, 'f1-score': 0.66, 'support': 4262.0}	{'precision': 0.8895348837209303, 'recall': 0.8480369515011548, 'f1-score': 0.8682903759754079, 'support': 2165.0}	{'precision': 0.9971224204554797, 'recall': 0.9995055216746332, 'f1-score': 0.9983125488743466, 'support': 12134.0}	{'precision': 0.9082976236852357, 'recall': 0.8940869698596519, 'f1-score': 0.9011362757980985, 'support': 13039.0}	0.9027	{'precision': 0.8587211030539967, 'recall': 0.8557499229362415, 'f1-score': 0.8569348001619632, 'support': 31600.0}	{'precision': 0.9049240079307782, 'recall': 0.9027215189873418, 'f1-score': 0.9036775010177053, 'support': 31600.0}
No log	5.0	405	0.2990	{'precision': 0.6126, 'recall': 0.7186766776161426, 'f1-score': 0.6614122219822932, 'support': 4262.0}	{'precision': 0.947565543071161, 'recall': 0.7011547344110854, 'f1-score': 0.805946376426865, 'support': 2165.0}	{'precision': 0.9975324888962, 'recall': 0.9995055216746332, 'f1-score': 0.998518030627367, 'support': 12134.0}	{'precision': 0.9121495327102803, 'recall': 0.8982283917478334, 'f1-score': 0.9051354379999227, 'support': 13039.0}	0.8994	{'precision': 0.8674618911694103, 'recall': 0.8293913313624237, 'f1-score': 0.842753016759112, 'support': 31600.0}	{'precision': 0.9069606828488892, 'recall': 0.8993987341772152, 'f1-score': 0.9013256821128531, 'support': 31600.0}
No log	6.0	486	0.3114	{'precision': 0.6412139011257953, 'recall': 0.6147348662599719, 'f1-score': 0.6276952563488261, 'support': 4262.0}	{'precision': 0.8991759573436743, 'recall': 0.8568129330254042, 'f1-score': 0.8774834437086093, 'support': 2165.0}	{'precision': 0.99860036225918, 'recall': 0.9995879347288611, 'f1-score': 0.9990939044481055, 'support': 12134.0}	{'precision': 0.8901916572717024, 'recall': 0.9083518674744996, 'f1-score': 0.8991800789553598, 'support': 13039.0}	0.9003	{'precision': 0.8572954695000881, 'recall': 0.8448719003721842, 'f1-score': 0.8508631708652252, 'support': 31600.0}	{'precision': 0.8988542850970194, 'recall': 0.900253164556962, 'f1-score': 0.8994431431727875, 'support': 31600.0}
0.2362	7.0	567	0.3888	{'precision': 0.6953872932985204, 'recall': 0.5624120131393712, 'f1-score': 0.6218705409261902, 'support': 4262.0}	{'precision': 0.9364837398373984, 'recall': 0.851270207852194, 'f1-score': 0.8918461166223083, 'support': 2165.0}	{'precision': 0.9982714626718249, 'recall': 0.9995055216746332, 'f1-score': 0.9988881110241733, 'support': 12134.0}	{'precision': 0.8734682245654033, 'recall': 0.9402561546130839, 'f1-score': 0.9056325023084026, 'support': 13039.0}	0.9059	{'precision': 0.8759026800932868, 'recall': 0.8383609743198206, 'f1-score': 0.8545593177202686, 'support': 31600.0}	{'precision': 0.9016900648403317, 'recall': 0.9059493670886076, 'f1-score': 0.9022249881228259, 'support': 31600.0}
0.2362	8.0	648	0.4088	{'precision': 0.6334106728538283, 'recall': 0.7045987799155327, 'f1-score': 0.6671109630123293, 'support': 4262.0}	{'precision': 0.9212598425196851, 'recall': 0.8646651270207852, 'f1-score': 0.8920657612580415, 'support': 2165.0}	{'precision': 0.9982713203819559, 'recall': 0.9994231086204055, 'f1-score': 0.998846882464377, 'support': 12134.0}	{'precision': 0.9115860872308542, 'recall': 0.8864176700667229, 'f1-score': 0.8988257251730306, 'support': 13039.0}	0.9038	{'precision': 0.8661319807465809, 'recall': 0.8637761714058616, 'f1-score': 0.8642123329769446, 'support': 31600.0}	{'precision': 0.9080164253061992, 'recall': 0.9037974683544304, 'f1-score': 0.9055172784758262, 'support': 31600.0}
0.2362	9.0	729	0.4774	{'precision': 0.6054054054054054, 'recall': 0.7095260441107462, 'f1-score': 0.6533434157934537, 'support': 4262.0}	{'precision': 0.882988298829883, 'recall': 0.9062355658198614, 'f1-score': 0.8944609072258947, 'support': 2165.0}	{'precision': 0.9976968001974171, 'recall': 0.9995879347288611, 'f1-score': 0.9986414721501792, 'support': 12134.0}	{'precision': 0.9177163422214952, 'recall': 0.8604954367666232, 'f1-score': 0.8881852364931724, 'support': 13039.0}	0.8967	{'precision': 0.8509517116635501, 'recall': 0.868961245356523, 'f1-score': 0.8586577579156749, 'support': 31600.0}	{'precision': 0.903926071665382, 'recall': 0.8966772151898734, 'f1-score': 0.899355076707611, 'support': 31600.0}
0.2362	10.0	810	0.5144	{'precision': 0.6497005988023952, 'recall': 0.6109807602064758, 'f1-score': 0.6297460701330109, 'support': 4262.0}	{'precision': 0.9827904118008605, 'recall': 0.7385681293302541, 'f1-score': 0.8433544303797468, 'support': 2165.0}	{'precision': 0.9987648221343873, 'recall': 0.9995879347288611, 'f1-score': 0.9991762089134196, 'support': 12134.0}	{'precision': 0.8794587945879458, 'recall': 0.9322033898305084, 'f1-score': 0.9050632911392404, 'support': 13039.0}	0.9015	{'precision': 0.8776786568313971, 'recall': 0.8203350535240248, 'f1-score': 0.8443350001413544, 'support': 31600.0}	{'precision': 0.901362049622011, 'recall': 0.901487341772152, 'f1-score': 0.8998406476202226, 'support': 31600.0}
0.2362	11.0	891	0.5589	{'precision': 0.6431628745212886, 'recall': 0.6698732989206945, 'f1-score': 0.6562464084587979, 'support': 4262.0}	{'precision': 0.9496362618914381, 'recall': 0.7838337182448037, 'f1-score': 0.8588056680161943, 'support': 2165.0}	{'precision': 0.9974508675273415, 'recall': 0.9996703477830888, 'f1-score': 0.9985593743568636, 'support': 12134.0}	{'precision': 0.8977522137289033, 'recall': 0.9097323414372268, 'f1-score': 0.9037025750419017, 'support': 13039.0}	0.9033	{'precision': 0.8720005544172429, 'recall': 0.8407774265964535, 'f1-score': 0.8543285064684394, 'support': 31600.0}	{'precision': 0.9052526145440706, 'recall': 0.9032911392405063, 'f1-score': 0.9036751198899997, 'support': 31600.0}
0.2362	12.0	972	0.6348	{'precision': 0.668590065228299, 'recall': 0.6252932895354294, 'f1-score': 0.6462172647914647, 'support': 4262.0}	{'precision': 0.9247202441505595, 'recall': 0.8397228637413395, 'f1-score': 0.8801742919389979, 'support': 2165.0}	{'precision': 0.99777924000658, 'recall': 0.9997527608373167, 'f1-score': 0.9987650255228059, 'support': 12134.0}	{'precision': 0.8901408450704226, 'recall': 0.920929519134903, 'f1-score': 0.9052734743111314, 'support': 13039.0}	0.9058	{'precision': 0.8703075986139653, 'recall': 0.8464246083122471, 'f1-score': 0.8576075141410999, 'support': 31600.0}	{'precision': 0.9039604418893055, 'recall': 0.905759493670886, 'f1-score': 0.9045136384754975, 'support': 31600.0}
0.0312	13.0	1053	0.5935	{'precision': 0.6740142052412442, 'recall': 0.6457062412013139, 'f1-score': 0.6595566207309765, 'support': 4262.0}	{'precision': 0.9273797841020608, 'recall': 0.8729792147806005, 'f1-score': 0.8993576017130621, 'support': 2165.0}	{'precision': 0.998025666337611, 'recall': 0.9998351738915444, 'f1-score': 0.9989296006587073, 'support': 12134.0}	{'precision': 0.8968700743075884, 'recall': 0.9164046322570749, 'f1-score': 0.9065321295804567, 'support': 13039.0}	0.9090	{'precision': 0.8740724324971261, 'recall': 0.8587313155326335, 'f1-score': 0.8660939881708007, 'support': 31600.0}	{'precision': 0.9077455097960874, 'recall': 0.9089556962025317, 'f1-score': 0.9082096119384979, 'support': 31600.0}
0.0312	14.0	1134	0.6126	{'precision': 0.6902082834570266, 'recall': 0.6764429845143125, 'f1-score': 0.6832563099893353, 'support': 4262.0}	{'precision': 0.9273270283723245, 'recall': 0.8605080831408776, 'f1-score': 0.892668902731193, 'support': 2165.0}	{'precision': 0.9987641097470544, 'recall': 0.9990110433492665, 'f1-score': 0.9988875612871329, 'support': 12134.0}	{'precision': 0.9023875875574301, 'recall': 0.9188588081908122, 'f1-score': 0.9105487156102752, 'support': 13039.0}	0.9129	{'precision': 0.879671752283459, 'recall': 0.8637052297988173, 'f1-score': 0.8713403724044841, 'support': 31600.0}	{'precision': 0.9124862715934182, 'recall': 0.9129430379746836, 'f1-score': 0.9125890170597477, 'support': 31600.0}
0.0312	15.0	1215	0.6346	{'precision': 0.6470333477695972, 'recall': 0.7010793054903801, 'f1-score': 0.672972972972973, 'support': 4262.0}	{'precision': 0.9200581395348837, 'recall': 0.877136258660508, 'f1-score': 0.8980846535824072, 'support': 2165.0}	{'precision': 0.9985185185185185, 'recall': 0.9998351738915444, 'f1-score': 0.9991764124526437, 'support': 12134.0}	{'precision': 0.9112625313283208, 'recall': 0.8923230309072782, 'f1-score': 0.9016933390165459, 'support': 13039.0}	0.9068	{'precision': 0.86921813428783, 'recall': 0.8675934422374277, 'f1-score': 0.8679818445061424, 'support': 31600.0}	{'precision': 0.9097328433538203, 'recall': 0.9067721518987342, 'f1-score': 0.9080300671504381, 'support': 31600.0}
0.0312	16.0	1296	0.6317	{'precision': 0.6643092880716643, 'recall': 0.661191928671985, 'f1-score': 0.6627469426152399, 'support': 4262.0}	{'precision': 0.9574241617881852, 'recall': 0.8309468822170901, 'f1-score': 0.8897131552917903, 'support': 2165.0}	{'precision': 0.9984362139917695, 'recall': 0.9997527608373167, 'f1-score': 0.999094053697908, 'support': 12134.0}	{'precision': 0.8954160102033161, 'recall': 0.9153309302860649, 'f1-score': 0.9052639563106797, 'support': 13039.0}	0.9077	{'precision': 0.8788964185137338, 'recall': 0.8518056255031142, 'f1-score': 0.8642045269789045, 'support': 31600.0}	{'precision': 0.9080526542294312, 'recall': 0.9076898734177216, 'f1-score': 0.9075190007765268, 'support': 31600.0}
0.0312	17.0	1377	0.6472	{'precision': 0.6666666666666666, 'recall': 0.684185828249648, 'f1-score': 0.6753126447429365, 'support': 4262.0}	{'precision': 0.9678100263852243, 'recall': 0.8471131639722864, 'f1-score': 0.903448275862069, 'support': 2165.0}	{'precision': 0.998025341451374, 'recall': 0.9996703477830888, 'f1-score': 0.9988471673254282, 'support': 12134.0}	{'precision': 0.9003566820975943, 'recall': 0.9098857274330854, 'f1-score': 0.9050961245041197, 'support': 13039.0}	0.9096	{'precision': 0.8832146791502148, 'recall': 0.8602137668595271, 'f1-score': 0.8706760531086384, 'support': 31600.0}	{'precision': 0.9109630478322421, 'recall': 0.909620253164557, 'f1-score': 0.9099907564832829, 'support': 31600.0}
0.0312	18.0	1458	0.6553	{'precision': 0.672607421875, 'recall': 0.6464101360863445, 'f1-score': 0.6592486240727448, 'support': 4262.0}	{'precision': 0.9460732984293194, 'recall': 0.8346420323325635, 'f1-score': 0.8868711656441718, 'support': 2165.0}	{'precision': 0.9978613144690301, 'recall': 0.9997527608373167, 'f1-score': 0.9988061421925816, 'support': 12134.0}	{'precision': 0.8936518568132767, 'recall': 0.920929519134903, 'f1-score': 0.9070856624867805, 'support': 13039.0}	0.9083	{'precision': 0.8775484728966565, 'recall': 0.850433612097782, 'f1-score': 0.8630028985990696, 'support': 31600.0}	{'precision': 0.9074454833508309, 'recall': 0.9082594936708861, 'f1-score': 0.9074935883527716, 'support': 31600.0}
0.0058	19.0	1539	0.6835	{'precision': 0.687516356974614, 'recall': 0.6163772876583763, 'f1-score': 0.6500061858220958, 'support': 4262.0}	{'precision': 0.9501039501039501, 'recall': 0.8443418013856813, 'f1-score': 0.8941061384201515, 'support': 2165.0}	{'precision': 0.9978613144690301, 'recall': 0.9997527608373167, 'f1-score': 0.9988061421925816, 'support': 12134.0}	{'precision': 0.8858227478464009, 'recall': 0.9305928368739934, 'f1-score': 0.9076560571492688, 'support': 13039.0}	0.9089	{'precision': 0.8803260923484988, 'recall': 0.8477661716888419, 'f1-score': 0.8626436308960244, 'support': 31600.0}	{'precision': 0.9065019545676358, 'recall': 0.9088607594936708, 'f1-score': 0.9069780763350475, 'support': 31600.0}
0.0058	20.0	1620	0.6673	{'precision': 0.6793010091065715, 'recall': 0.6475832942280619, 'f1-score': 0.663063063063063, 'support': 4262.0}	{'precision': 0.9517884914463453, 'recall': 0.8480369515011548, 'f1-score': 0.8969223253541768, 'support': 2165.0}	{'precision': 0.9979434024350116, 'recall': 0.9997527608373167, 'f1-score': 0.9988472622478387, 'support': 12134.0}	{'precision': 0.8936961046684508, 'recall': 0.922003221105913, 'f1-score': 0.9076290060775357, 'support': 13039.0}	0.9098	{'precision': 0.8806822519140949, 'recall': 0.8543440569181115, 'f1-score': 0.8666154141856536, 'support': 31600.0}	{'precision': 0.908789611984554, 'recall': 0.9097784810126582, 'f1-score': 0.9089366740356591, 'support': 31600.0}

Framework versions

• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2