End of training

README.md

@@ -12,7 +12,7 @@ model-index:
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/noc-lab/PMC_LLAMA2_7B_trainer_lora/runs/twrymb8f)
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/noc-lab/PMC_LLAMA2_7B_trainer_lora/runs/qm80zll8)
 # PMC_LLAMA2_7B_trainer_lora
 
 This model is a fine-tuned version of [chaoyi-wu/PMC_LLAMA_7B](https://huggingface.co/chaoyi-wu/PMC_LLAMA_7B) on an unknown dataset.
@@ -39,10 +39,10 @@ The following hyperparameters were used during training:
 - eval_batch_size: 48
 - seed: 123
 - distributed_type: multi-GPU
-- num_devices: 2
+- num_devices: 3
 - gradient_accumulation_steps: 8
-- total_train_batch_size: 768
-- total_eval_batch_size: 96
+- total_train_batch_size: 1152
+- total_eval_batch_size: 144
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_ratio: 0.03

adapter_config.json

@@ -20,8 +20,8 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "v_proj",
-    "q_proj"
+    "q_proj",
+    "v_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

trainer_peft.log

@@ -759,3 +759,123 @@
 2024-06-02 10:14 - Setup optimizer
 2024-06-02 10:15 - Continue  training!!
 2024-06-02 20:22 - Training complete!!!
+2024-06-02 20:22 - Training complete!!!
+2024-06-04 12:41 - Cuda check
+2024-06-04 12:41 - True
+2024-06-04 12:41 - 3
+2024-06-04 12:41 - Configue Model and tokenizer
+2024-06-04 12:41 - Cuda check
+2024-06-04 12:41 - True
+2024-06-04 12:41 - 3
+2024-06-04 12:41 - Configue Model and tokenizer
+2024-06-04 12:41 - Cuda check
+2024-06-04 12:41 - True
+2024-06-04 12:41 - 3
+2024-06-04 12:41 - Configue Model and tokenizer
+2024-06-04 12:42 - Memory usage in   0.00 GB
+2024-06-04 12:42 - Memory usage in   0.00 GB
+2024-06-04 12:42 - Memory usage in   0.00 GB
+2024-06-04 12:42 - Dataset loaded successfully:
+ train-Jingmei/Pandemic_Books
+ test -Jingmei/Pandemic
+2024-06-04 12:42 - Dataset loaded successfully:
+ train-Jingmei/Pandemic_Books
+ test -Jingmei/Pandemic
+2024-06-04 12:42 - Dataset loaded successfully:
+ train-Jingmei/Pandemic_Books
+ test -Jingmei/Pandemic
+2024-06-04 12:44 - Cuda check
+2024-06-04 12:44 - True
+2024-06-04 12:44 - 3
+2024-06-04 12:44 - Configue Model and tokenizer
+2024-06-04 12:44 - Cuda check
+2024-06-04 12:44 - True
+2024-06-04 12:44 - 3
+2024-06-04 12:44 - Configue Model and tokenizer
+2024-06-04 12:44 - Cuda check
+2024-06-04 12:44 - True
+2024-06-04 12:44 - 3
+2024-06-04 12:44 - Configue Model and tokenizer
+2024-06-04 12:44 - Memory usage in   0.00 GB
+2024-06-04 12:44 - Memory usage in   0.00 GB
+2024-06-04 12:44 - Memory usage in   0.00 GB
+2024-06-04 12:44 - Dataset loaded successfully:
+ train-Jingmei/Pandemic_Books
+ test -Jingmei/Pandemic_WHO
+2024-06-04 12:44 - Dataset loaded successfully:
+ train-Jingmei/Pandemic_Books
+ test -Jingmei/Pandemic_WHO
+2024-06-04 12:44 - Dataset loaded successfully:
+ train-Jingmei/Pandemic_Books
+ test -Jingmei/Pandemic_WHO
+2024-06-04 12:46 - Tokenize data: DatasetDict({
+    train: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 5966
+    })
+    test: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 8264
+    })
+})
+2024-06-04 12:46 - Tokenize data: DatasetDict({
+    train: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 5966
+    })
+    test: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 8264
+    })
+})
+2024-06-04 12:46 - Tokenize data: DatasetDict({
+    train: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 5966
+    })
+    test: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 8264
+    })
+})
+2024-06-04 12:51 - Split data into chunks:DatasetDict({
+    train: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 388202
+    })
+    test: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 198960
+    })
+})
+2024-06-04 12:51 - Setup PEFT
+2024-06-04 12:51 - Setup optimizer
+2024-06-04 12:51 - Split data into chunks:DatasetDict({
+    train: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 388202
+    })
+    test: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 198960
+    })
+})
+2024-06-04 12:51 - Setup PEFT
+2024-06-04 12:51 - Split data into chunks:DatasetDict({
+    train: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 388202
+    })
+    test: Dataset({
+        features: ['input_ids', 'attention_mask'],
+        num_rows: 198960
+    })
+})
+2024-06-04 12:51 - Setup PEFT
+2024-06-04 12:51 - Setup optimizer
+2024-06-04 12:51 - Setup optimizer
+2024-06-04 12:51 - Continue  training!!
+2024-06-04 12:51 - Continue  training!!
+2024-06-04 12:51 - Continue  training!!
+2024-06-04 12:52 - Training complete!!!
+2024-06-04 12:52 - Training complete!!!

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:860d6c6d3a29d05d74c27f2cf37f2b7e151e4a8ab1bb5291ab978a1b08e48ad3
+oid sha256:7f8a403943bec37ec2560eeff417047a57bceb3fe0df6c36af4ee718a0545951
 size 5176