Mojiz-DPO

README.md

@@ -16,15 +16,15 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [ahmedabdelwahed/Mojiz-sft](https://huggingface.co/ahmedabdelwahed/Mojiz-sft) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.1612
-- Rewards/chosen: 1.5401
-- Rewards/rejected: -0.4134
+- Loss: 0.0000
+- Rewards/chosen: 10.9834
+- Rewards/rejected: -4.3197
 - Rewards/accuracies: 1.0
-- Rewards/margins: 1.9534
-- Logps/rejected: -71.8662
-- Logps/chosen: -323.2329
-- Logits/rejected: -11.6886
-- Logits/chosen: -12.9104
+- Rewards/margins: 15.3031
+- Logps/rejected: -79.6788
+- Logps/chosen: -304.3463
+- Logits/rejected: -11.4276
+- Logits/chosen: -12.4846
 
 ## Model description
 
@@ -43,7 +43,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 1e-06
+- learning_rate: 1e-05
 - train_batch_size: 4
 - eval_batch_size: 8
 - seed: 42
@@ -56,16 +56,16 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
 |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
-| 1.3422        | 0.41  | 100  | 0.6447          | 0.0816         | -0.0178          | 1.0                | 0.0994          | -71.0751       | -326.1499    | -11.7285        | -12.9749      |
-| 1.9054        | 0.82  | 200  | 0.5151          | 0.3250         | -0.0755          | 1.0                | 0.4005          | -71.1904       | -325.6630    | -11.7221        | -12.9647      |
-| 1.7913        | 1.22  | 300  | 0.4000          | 0.5897         | -0.1444          | 1.0                | 0.7341          | -71.3283       | -325.1337    | -11.7152        | -12.9536      |
-| 1.6915        | 1.63  | 400  | 0.3189          | 0.8238         | -0.2053          | 1.0                | 1.0290          | -71.4500       | -324.6655    | -11.7087        | -12.9431      |
-| 1.4041        | 2.04  | 500  | 0.2596          | 1.0374         | -0.2638          | 1.0                | 1.3012          | -71.5670       | -324.2383    | -11.7029        | -12.9335      |
-| 1.1989        | 2.45  | 600  | 0.2187          | 1.2160         | -0.3160          | 1.0                | 1.5320          | -71.6715       | -323.8811    | -11.6980        | -12.9256      |
-| 1.2192        | 2.86  | 700  | 0.1915          | 1.3556         | -0.3570          | 1.0                | 1.7126          | -71.7533       | -323.6018    | -11.6939        | -12.9188      |
-| 1.1718        | 3.27  | 800  | 0.1738          | 1.4588         | -0.3884          | 1.0                | 1.8472          | -71.8162       | -323.3954    | -11.6909        | -12.9141      |
-| 0.9623        | 3.67  | 900  | 0.1643          | 1.5190         | -0.4068          | 1.0                | 1.9258          | -71.8531       | -323.2751    | -11.6892        | -12.9113      |
-| 1.0207        | 4.08  | 1000 | 0.1612          | 1.5401         | -0.4134          | 1.0                | 1.9534          | -71.8662       | -323.2329    | -11.6886        | -12.9104      |
+| 1.0153        | 0.41  | 100  | 0.2835          | 0.9457         | -0.2302          | 1.0                | 1.1759          | -71.4998       | -324.4215    | -11.7052        | -12.9375      |
+| 0.5417        | 0.82  | 200  | 0.0117          | 4.5050         | -1.4875          | 1.0                | 5.9925          | -74.0144       | -317.3029    | -11.5990        | -12.7637      |
+| 0.0938        | 1.22  | 300  | 0.0006          | 7.3787         | -2.7778          | 1.0                | 10.1565         | -76.5951       | -311.5557    | -11.5086        | -12.6163      |
+| 0.0091        | 1.63  | 400  | 0.0001          | 8.4468         | -3.1612          | 1.0                | 11.6080         | -77.3619       | -309.4195    | -11.4851        | -12.5777      |
+| 0.0598        | 2.04  | 500  | 0.0000          | 9.5528         | -3.6756          | 1.0                | 13.2285         | -78.3907       | -307.2074    | -11.4570        | -12.5323      |
+| 0.0032        | 2.45  | 600  | 0.0000          | 10.0884        | -3.9009          | 1.0                | 13.9892         | -78.8412       | -306.1363    | -11.4508        | -12.5212      |
+| 0.017         | 2.86  | 700  | 0.0000          | 10.4984        | -4.0850          | 1.0                | 14.5834         | -79.2095       | -305.3162    | -11.4376        | -12.4997      |
+| 0.002         | 3.27  | 800  | 0.0000          | 10.8150        | -4.2275          | 1.0                | 15.0425         | -79.4944       | -304.6830    | -11.4327        | -12.4924      |
+| 0.0001        | 3.67  | 900  | 0.0000          | 10.9336        | -4.2971          | 1.0                | 15.2308         | -79.6337       | -304.4458    | -11.4287        | -12.4862      |
+| 0.0002        | 4.08  | 1000 | 0.0000          | 10.9834        | -4.3197          | 1.0                | 15.3031         | -79.6788       | -304.3463    | -11.4276        | -12.4846      |
 
 
 ### Framework versions

adapter_config.json

@@ -19,8 +19,8 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "v",
-    "q"
+    "q",
+    "v"
   ],
   "task_type": "SEQ_2_SEQ_LM"
 }

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f074e5482ab7b55736ea422a7cae6b113c35fed4be27bd861a63e0cd28530d77
+oid sha256:c71b89296e74ce269aafa0db1d5cc34865fe3b136a22f6e9c937346976df1c46
 size 7098016

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4b39ddfbc580f2428e71774c75f1bf5d819d38ab012152bd6f322042e89e5044
+oid sha256:806816a3307c8886169d63dc7c362bc48af56e108a05e244d05c56b5a19b8fdd
 size 4219