Model save

README.md

@@ -0,0 +1,81 @@
+---
+library_name: peft
+tags:
+- trl
+- dpo
+- generated_from_trainer
+base_model: DUAL-GPO-2/zephyr-7b-sft-new
+model-index:
+- name: zephyr-7b-gpo-new-v1-i0
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# zephyr-7b-gpo-new-v1-i0
+
+This model is a fine-tuned version of [DUAL-GPO-2/zephyr-7b-sft-new](https://huggingface.co/DUAL-GPO-2/zephyr-7b-sft-new) on the None dataset.
+It achieves the following results on the evaluation set:
+- Logits/chosen: -1.6150
+- Logits/rejected: -1.4339
+- Logps/chosen: -400.8042
+- Logps/rejected: -417.9239
+- Loss: 0.0367
+- Rewards/accuracies: 0.5930
+- Rewards/chosen: -0.1716
+- Rewards/margins: 0.0499
+- Rewards/rejected: -0.2215
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-06
+- train_batch_size: 2
+- eval_batch_size: 2
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 2
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 8
+- total_eval_batch_size: 4
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch | Step | Logits/chosen | Logits/rejected | Logps/chosen | Logps/rejected | Validation Loss | Rewards/accuracies | Rewards/chosen | Rewards/margins | Rewards/rejected |
+|:-------------:|:-----:|:----:|:-------------:|:---------------:|:------------:|:--------------:|:---------------:|:------------------:|:--------------:|:---------------:|:----------------:|
+| 0.0532        | 0.01  | 100  | -2.3286       | -2.1100         | -227.7062    | -195.3472      | 0.0535          | 0.5530             | 0.0015         | 0.0005          | 0.0011           |
+| 0.0625        | 0.03  | 200  | -2.3309       | -2.1124         | -225.1784    | -194.7161      | 0.0527          | 0.6080             | 0.0040         | 0.0023          | 0.0017           |
+| 0.0485        | 0.04  | 300  | -2.3236       | -2.1050         | -237.7424    | -214.5471      | 0.0496          | 0.5890             | -0.0085        | 0.0096          | -0.0181          |
+| 0.0361        | 0.05  | 400  | -2.3720       | -2.1493         | -251.4063    | -239.9417      | 0.0447          | 0.5990             | -0.0222        | 0.0213          | -0.0435          |
+| 0.0375        | 0.07  | 500  | -2.1960       | -1.9821         | -282.6958    | -281.3289      | 0.0417          | 0.5890             | -0.0535        | 0.0314          | -0.0849          |
+| 0.0522        | 0.08  | 600  | -1.5433       | -1.3697         | -398.6434    | -395.0714      | 0.0432          | 0.5920             | -0.1694        | 0.0292          | -0.1987          |
+| 0.0453        | 0.09  | 700  | -1.9137       | -1.7203         | -295.2011    | -297.8420      | 0.0367          | 0.5780             | -0.0660        | 0.0355          | -0.1014          |
+| 0.0293        | 0.1   | 800  | -1.6150       | -1.4339         | -400.8042    | -417.9239      | 0.0367          | 0.5930             | -0.1716        | 0.0499          | -0.2215          |
+
+
+### Framework versions
+
+- PEFT 0.7.1
+- Transformers 4.36.2
+- Pytorch 2.1.2+cu121
+- Datasets 2.14.6
+- Tokenizers 0.15.2

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ce3af3dfa1e8877a70472d219cf86f45e75f1cb08ff036a87b27509c10e5e4da
+oid sha256:7a59f59eb644904f48d5b1d259c9f92fbf2059a6003517198cbafa115f4e81ce
 size 671150064

all_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "train_loss": 0.0048277108391146795,
+    "train_runtime": 13417.0015,
+    "train_samples": 61135,
+    "train_samples_per_second": 4.557,
+    "train_steps_per_second": 0.57
+}

runs/May11_22-17-27_gpu4-119-5/events.out.tfevents.1715429920.gpu4-119-5.1214138.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:161e22fbf06d6241e1fb825e00fec25095c2c1d088836fca853a49f432c32732
-size 93332
+oid sha256:352c8bdd6623c986794ffb166034bfd54e97d147535349c2e891b38ba272ceaa
+size 102562

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "train_loss": 0.0048277108391146795,
+    "train_runtime": 13417.0015,
+    "train_samples": 61135,
+    "train_samples_per_second": 4.557,
+    "train_steps_per_second": 0.57
+}