Model save

README.md

@@ -17,6 +17,8 @@ should probably proofread and complete it, then remove this comment. -->
 # mistral-sft-lora-fsdp2
 
 This model is a fine-tuned version of [meta-llama/Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.8343
 
 ## Model description
 
@@ -45,13 +47,14 @@ The following hyperparameters were used during training:
 - total_eval_batch_size: 4
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
-- num_epochs: 1.0
+- num_epochs: 2.0
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss |
 |:-------------:|:-----:|:----:|:---------------:|
-| No log        | 1.0   | 1    | 1.9796          |
+| No log        | 1.0   | 1    | 1.9798          |
+| No log        | 2.0   | 2    | 1.8343          |
 
 
 ### Framework versions

adapter_config.json

@@ -23,13 +23,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "k_proj",
+    "up_proj",
     "down_proj",
-    "o_proj",
-    "gate_proj",
     "v_proj",
-    "up_proj",
-    "q_proj"
+    "gate_proj",
+    "o_proj",
+    "q_proj",
+    "k_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4bd5ae0ff2058d17dce24cc8ef059ae969f7f755deda4db1eafc05ceac6fb551
+oid sha256:08a13c9bcf2f18a7fc083503c10c21f34393d99111b5deca6147519caae65646
 size 1656902648

logs/training_log.txt

@@ -37,3 +37,22 @@
 2025-01-08 13:56:48,398 - INFO - Model saved to mistral-sft-lora-fsdp2/checkpoint-2/pytorch_model_fsdp_0
 2025-01-08 13:56:53,278 - INFO - Saving Optimizer state to mistral-sft-lora-fsdp2/checkpoint-2/optimizer_0
 2025-01-08 13:56:59,275 - INFO - Optimizer state saved in mistral-sft-lora-fsdp2/checkpoint-2/optimizer_0
+2025-01-08 13:57:23,252 - INFO - Loss improved from 1.97976 to 1.83431
+2025-01-08 13:57:23,252 - INFO - Loss improved from 1.97976 to 1.83431
+2025-01-08 13:57:23,252 - INFO - Loss improved from 1.97976 to 1.83431
+2025-01-08 13:57:23,253 - INFO - Step 2/2 (100.0%), epoch: 2.0000, step_time: 205.04s, elapsed_time: 599.83s
+2025-01-08 13:57:23,254 - INFO - Evaluation Results:
+  eval_loss: 1.8343
+  eval_runtime: 23.7205
+  eval_samples_per_second: 0.3370
+  eval_steps_per_second: 0.0840
+  epoch: 2.0000
+  elapsed_time: 599.83s
+  step_time: 205.04s
+2025-01-08 13:57:23,255 - INFO - Loss improved from 1.97976 to 1.83431
+2025-01-08 13:58:21,011 - INFO - Saving model to mistral-sft-lora-fsdp2/checkpoint-2/pytorch_model_fsdp_0
+2025-01-08 13:58:24,262 - INFO - Model saved to mistral-sft-lora-fsdp2/checkpoint-2/pytorch_model_fsdp_0
+2025-01-08 13:58:29,156 - INFO - Saving Optimizer state to mistral-sft-lora-fsdp2/checkpoint-2/optimizer_0
+2025-01-08 13:58:35,600 - INFO - Optimizer state saved in mistral-sft-lora-fsdp2/checkpoint-2/optimizer_0
+2025-01-08 13:58:36,036 - INFO - Step 2/2 (100.0%), epoch: 2.0000, step_time: 72.78s, elapsed_time: 672.61s
+2025-01-08 13:58:36,037 - INFO - Training completed in 672.61 seconds

runs/Jan08_13-47-23_gpu-server/events.out.tfevents.1736344266.gpu-server.882619.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8dc1e98fd384467f592194e00f17315e978da77c8188ec70ec9d163ed6d145e6
-size 5873
+oid sha256:ccf149023c1ed3b470f6fa07c6796be4504f503ed24f0f958974f0e5810ca0c4
+size 6487