Hugging Face's logo

IntelLabs
/
eftnas-s1-bert-base

English
Model card Files
eftnas-s1-bert-base / eftnas-s1-bert-base-mrpc /README.md
jinjieyuan's picture
jinjieyuan
Upload model
1960aa4
preview code
|
raw
history blame
16.6 kB
metadata
language:
  - en
license: apache-2.0
tags:
  - generated_from_trainer
datasets:
  - glue
metrics:
  - accuracy
  - f1
model-index:
  - name: tryv3_20epochs
    results:
      - task:
          name: Text Classification
          type: text-classification
        dataset:
          name: GLUE MRPC
          type: glue
          config: mrpc
          split: validation
          args: mrpc
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8259803921568627
          - name: F1
            type: f1
            value: 0.880672268907563

tryv3_20epochs

This model is a fine-tuned version of bert-base-uncased on the GLUE MRPC dataset. It achieves the following results on the evaluation set:

  • Loss: 0.8979
  • Accuracy: 0.8260
  • F1: 0.8807
  • Combined Score: 0.8533

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 32
  • eval_batch_size: 128
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 30
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy F1 Combined Score
No log 1.0 115 0.4635 0.8235 0.8746 0.8490 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
No log 1.0 115 0.4038 0.8358 0.8835 0.8596 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
No log 2.0 230 0.4734 0.8235 0.875 0.8493 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
No log 2.0 230 0.3589 0.8480 0.8946 0.8713 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
No log 3.0 345 0.9022 0.8039 0.8680 0.8360 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
No log 3.0 345 0.6409 0.8505 0.8982 0.8743 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
No log 4.0 460 0.7340 0.8407 0.8885 0.8646 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
No log 4.0 460 0.6669 0.8578 0.9017 0.8798 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.6907 5.0 575 0.8398 0.8235 0.8788 0.8512 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.6907 5.0 575 0.6096 0.8799 0.9154 0.8976 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.6907 6.0 690 1.0233 0.8186 0.8733 0.8460 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.6907 6.0 690 0.8278 0.8407 0.8885 0.8646 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.6907 7.0 805 0.9062 0.8137 0.8628 0.8383 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.6907 7.0 805 0.6970 0.8456 0.8885 0.8670 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.6907 8.0 920 0.9299 0.8235 0.8763 0.8499 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.6907 8.0 920 0.7926 0.8456 0.8945 0.8700 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0889 9.0 1035 0.9827 0.8211 0.8765 0.8488 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0889 9.0 1035 0.7439 0.8529 0.8980 0.8755 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0889 10.0 1150 0.8922 0.8284 0.8810 0.8547 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0889 10.0 1150 0.6456 0.8603 0.9012 0.8808 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0889 11.0 1265 0.9563 0.8211 0.8773 0.8492 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0889 11.0 1265 0.6870 0.8554 0.8998 0.8776 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0889 12.0 1380 0.9069 0.8260 0.8790 0.8525 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0889 12.0 1380 0.5972 0.8676 0.9056 0.8866 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0889 13.0 1495 0.9532 0.8235 0.8796 0.8516 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0889 13.0 1495 0.6416 0.8652 0.9047 0.8849 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0339 14.0 1610 0.9255 0.8260 0.8786 0.8523 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0339 14.0 1610 0.7535 0.8578 0.9014 0.8796 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0339 15.0 1725 0.9056 0.8211 0.8748 0.8479 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0339 15.0 1725 0.6562 0.8603 0.9002 0.8802 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0339 16.0 1840 0.9582 0.8088 0.8704 0.8396 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0339 16.0 1840 0.7180 0.8554 0.8995 0.8774 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0339 17.0 1955 0.9102 0.8309 0.8829 0.8569 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0339 17.0 1955 0.6792 0.8627 0.9038 0.8833 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0195 18.0 2070 0.8795 0.8284 0.8810 0.8547 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0195 18.0 2070 0.6670 0.8652 0.9050 0.8851 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0195 19.0 2185 0.9087 0.8235 0.8784 0.8510 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0195 19.0 2185 0.6763 0.8627 0.9038 0.8833 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])
0.0195 20.0 2300 0.9082 0.8235 0.8784 0.8510 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 704, 2: 768, 3: 768, 4: 768, 5: 640, 6: 640, 7: 512, 8: 512, 9: 320, 10: 192, 11: 192, 12: 995, 13: 930, 14: 928, 15: 931, 16: 872, 17: 787, 18: 672, 19: 579, 20: 409, 21: 291, 22: 300, 23: 308})])
0.0195 20.0 2300 0.6762 0.8627 0.9038 0.8833 OrderedDict([(<ElasticityDim.WIDTH: 'width'>, {0: 768, 1: 768, 2: 768, 3: 768, 4: 768, 5: 768, 6: 768, 7: 768, 8: 768, 9: 768, 10: 768, 11: 768, 12: 3072, 13: 3072, 14: 3072, 15: 3072, 16: 3072, 17: 3072, 18: 3072, 19: 3072, 20: 3072, 21: 3072, 22: 3072, 23: 3072})])

Framework versions

  • Transformers 4.29.1
  • Pytorch 1.12.1
  • Datasets 2.13.1
  • Tokenizers 0.13.3