Hugging Face's logo

Hmehdi515
/
phi-3-medium-MoRA

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files Community
phi-3-medium-MoRA / README.md
Hmehdi515's picture
Hmehdi515
Hmehdi515/mora
15e57e2 verified
preview code
|
raw
history blame contribute delete
6.46 kB
metadata
base_model: microsoft/Phi-3-medium-4k-instruct
library_name: peft
license: mit
tags:
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: phi-3-medium-MoRA
    results: []

Visualize in Weights & Biases

phi-3-medium-MoRA

This model is a fine-tuned version of microsoft/Phi-3-medium-4k-instruct on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 1.7627

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 4
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 10

Training results

Training Loss Epoch Step Validation Loss
1.0968 0.1118 2500 0.7399
1.0638 0.2237 5000 0.7234
1.0536 0.3355 7500 0.7156
1.0581 0.4473 10000 0.7128
1.0618 0.5592 12500 0.7125
1.0533 0.6710 15000 0.7131
1.0664 0.7828 17500 0.7133
1.0719 0.8947 20000 0.7160
1.0628 1.0065 22500 0.7210
0.9341 1.1183 25000 0.7241
0.9468 1.2301 27500 0.7235
0.9553 1.3420 30000 0.7271
0.9557 1.4538 32500 0.7242
0.9669 1.5656 35000 0.7244
0.9627 1.6775 37500 0.7220
0.963 1.7893 40000 0.7215
0.9493 1.9011 42500 0.7207
0.8938 2.0130 45000 0.7668
0.7061 2.1248 47500 0.7739
0.7105 2.2366 50000 0.7693
0.7046 2.3485 52500 0.7716
0.7241 2.4603 55000 0.7713
0.7273 2.5721 57500 0.7669
0.7443 2.6840 60000 0.7685
0.7457 2.7958 62500 0.7664
0.7436 2.9076 65000 0.7659
0.7525 3.0195 67500 0.8617
0.465 3.1313 70000 0.8682
0.4823 3.2431 72500 0.8798
0.4865 3.3550 75000 0.8763
0.4977 3.4668 77500 0.8613
0.5088 3.5786 80000 0.8627
0.5136 3.6904 82500 0.8681
0.5128 3.8023 85000 0.8486
0.525 3.9141 87500 0.8585
0.3967 4.0259 90000 0.9826
0.3016 4.1378 92500 0.9951
0.3167 4.2496 95000 1.0293
0.3179 4.3614 97500 0.9904
0.3292 4.4733 100000 0.9947
0.3346 4.5851 102500 0.9932
0.3405 4.6969 105000 0.9715
0.344 4.8088 107500 0.9974
0.3497 4.9206 110000 0.9929
0.3226 5.0324 112500 1.1483
0.2071 5.1443 115000 1.1669
0.2131 5.2561 117500 1.1275
0.2204 5.3679 120000 1.1513
0.222 5.4798 122500 1.1549
0.2287 5.5916 125000 1.1552
0.2315 5.7034 127500 1.1370
0.2359 5.8153 130000 1.1318
0.2362 5.9271 132500 1.1461
0.1611 6.0389 135000 1.2983
0.1527 6.1507 137500 1.3192
0.1593 6.2626 140000 1.3295
0.16 6.3744 142500 1.3048
0.1647 6.4862 145000 1.3161
0.1659 6.5981 147500 1.2908
0.1666 6.7099 150000 1.3202
0.1692 6.8217 152500 1.3039
0.1711 6.9336 155000 1.2895
0.1433 7.0454 157500 1.4769
0.122 7.1572 160000 1.4877
0.1225 7.2691 162500 1.4722
0.1261 7.3809 165000 1.4794
0.1262 7.4927 167500 1.4749
0.1274 7.6046 170000 1.4719
0.1287 7.7164 172500 1.4495
0.1298 7.8282 175000 1.4753
0.1304 7.9401 177500 1.4705
0.1 8.0519 180000 1.6185
0.1038 8.1637 182500 1.6353
0.1053 8.2756 185000 1.6272
0.1054 8.3874 187500 1.6138
0.1057 8.4992 190000 1.6226
0.1061 8.6110 192500 1.6407
0.1068 8.7229 195000 1.6334
0.1082 8.8347 197500 1.6358
0.1063 8.9465 200000 1.6325
0.0936 9.0584 202500 1.7572
0.091 9.1702 205000 1.7476
0.0932 9.2820 207500 1.7529
0.0932 9.3939 210000 1.7541
0.0935 9.5057 212500 1.7595
0.0931 9.6175 215000 1.7609
0.0937 9.7294 217500 1.7647
0.0922 9.8412 220000 1.7643
0.0925 9.9530 222500 1.7627

Framework versions

  • PEFT 0.9.0
  • Transformers 4.42.4
  • Pytorch 2.3.1+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1