longformer-simple

This model is a fine-tuned version of allenai/longformer-base-4096 on the stab-gurevych-essays dataset. It achieves the following results on the evaluation set:

  • Loss: 0.3326
  • Claim: {'precision': 0.6375421311900441, 'recall': 0.6000488042947779, 'f1-score': 0.6182275298554368, 'support': 4098.0}
  • Majorclaim: {'precision': 0.8534005037783375, 'recall': 0.7853500231803431, 'f1-score': 0.8179623370352487, 'support': 2157.0}
  • O: {'precision': 0.9584632404706829, 'recall': 0.9674144756877474, 'f1-score': 0.9629180559765586, 'support': 9851.0}
  • Premise: {'precision': 0.884906500445236, 'recall': 0.9064994298745724, 'f1-score': 0.8955728286583305, 'support': 13155.0}
  • Accuracy: 0.8752
  • Macro avg: {'precision': 0.8335780939710751, 'recall': 0.8148281832593602, 'f1-score': 0.8236701878813937, 'support': 29261.0}
  • Weighted avg: {'precision': 0.8727042457708365, 'recall': 0.8751580602166706, 'f1-score': 0.8736819489681839, 'support': 29261.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 5

Training results

Training Loss Epoch Step Validation Loss Claim Majorclaim O Premise Accuracy Macro avg Weighted avg
No log 1.0 41 0.5843 {'precision': 0.48984771573604063, 'recall': 0.14128843338213762, 'f1-score': 0.21931818181818183, 'support': 4098.0} {'precision': 0.5396243701328447, 'recall': 0.5461288827074641, 'f1-score': 0.5428571428571428, 'support': 2157.0} {'precision': 0.8916069169126951, 'recall': 0.858390011166379, 'f1-score': 0.8746832169640548, 'support': 9851.0} {'precision': 0.7743724104313917, 'recall': 0.9660965412390726, 'f1-score': 0.8596746372645179, 'support': 13155.0} 0.7834 {'precision': 0.673862853303243, 'recall': 0.6279759671237634, 'f1-score': 0.6241332947259743, 'support': 29261.0} {'precision': 0.7566882370115429, 'recall': 0.7833635214107515, 'f1-score': 0.7516910901801511, 'support': 29261.0}
No log 2.0 82 0.4171 {'precision': 0.5352343493936415, 'recall': 0.39848706686188384, 'f1-score': 0.45684711148412366, 'support': 4098.0} {'precision': 0.8575712143928036, 'recall': 0.5303662494204914, 'f1-score': 0.6553995989687769, 'support': 2157.0} {'precision': 0.9516030844155844, 'recall': 0.952086082631205, 'f1-score': 0.9518445222509768, 'support': 9851.0} {'precision': 0.8249001331557922, 'recall': 0.9418472063854048, 'f1-score': 0.8795031055900621, 'support': 13155.0} 0.8389 {'precision': 0.7923271953394554, 'recall': 0.7056966513247462, 'f1-score': 0.7358985845734849, 'support': 29261.0} {'precision': 0.8293966272342979, 'recall': 0.8388640169508903, 'f1-score': 0.8281446341741303, 'support': 29261.0}
No log 3.0 123 0.3525 {'precision': 0.6357409713574097, 'recall': 0.49829184968277207, 'f1-score': 0.5586867305061559, 'support': 4098.0} {'precision': 0.7525641025641026, 'recall': 0.8164116828929068, 'f1-score': 0.7831887925283523, 'support': 2157.0} {'precision': 0.9471273523847455, 'recall': 0.9655872500253782, 'f1-score': 0.956268221574344, 'support': 9851.0} {'precision': 0.8749451192741109, 'recall': 0.9089319650323071, 'f1-score': 0.8916147794638529, 'support': 13155.0} 0.8637 {'precision': 0.8025943863950922, 'recall': 0.797305686908341, 'f1-score': 0.7974396310181762, 'support': 29261.0} {'precision': 0.8567240306977373, 'recall': 0.863675199070435, 'f1-score': 0.8587617347894375, 'support': 29261.0}
No log 4.0 164 0.3385 {'precision': 0.6185015290519877, 'recall': 0.5922401171303074, 'f1-score': 0.6050860134629769, 'support': 4098.0} {'precision': 0.7913082842915347, 'recall': 0.8103847936949466, 'f1-score': 0.8007329363261567, 'support': 2157.0} {'precision': 0.9529177057356608, 'recall': 0.9697492640341082, 'f1-score': 0.9612598108271282, 'support': 9851.0} {'precision': 0.8938411050904373, 'recall': 0.8903078677309008, 'f1-score': 0.8920709878894051, 'support': 13155.0} 0.8694 {'precision': 0.8141421560424051, 'recall': 0.8156705106475658, 'f1-score': 0.8147874371264168, 'support': 29261.0} {'precision': 0.8676102420265399, 'recall': 0.8694166296435528, 'f1-score': 0.8684387980236479, 'support': 29261.0}
No log 5.0 205 0.3326 {'precision': 0.6375421311900441, 'recall': 0.6000488042947779, 'f1-score': 0.6182275298554368, 'support': 4098.0} {'precision': 0.8534005037783375, 'recall': 0.7853500231803431, 'f1-score': 0.8179623370352487, 'support': 2157.0} {'precision': 0.9584632404706829, 'recall': 0.9674144756877474, 'f1-score': 0.9629180559765586, 'support': 9851.0} {'precision': 0.884906500445236, 'recall': 0.9064994298745724, 'f1-score': 0.8955728286583305, 'support': 13155.0} 0.8752 {'precision': 0.8335780939710751, 'recall': 0.8148281832593602, 'f1-score': 0.8236701878813937, 'support': 29261.0} {'precision': 0.8727042457708365, 'recall': 0.8751580602166706, 'f1-score': 0.8736819489681839, 'support': 29261.0}

Framework versions

  • Transformers 4.45.2
  • Pytorch 2.5.0+cu124
  • Datasets 2.19.1
  • Tokenizers 0.20.1
Downloads last month
25
Safetensors
Model size
148M params
Tensor type
F32
·
Inference Providers NEW
This model is not currently available via any of the supported third-party Inference Providers, and the model is not deployed on the HF Inference API.

Model tree for Theoreticallyhugo/longformer-simple

Finetuned
(92)
this model

Evaluation results