Update README.md

README.md

@@ -33,43 +33,30 @@ Trying to get better at medical Q & A
 
 ### Model Sources [optional]
 
-<!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
+- **Repository:** [Tonic/mistralmed]
 - **Paper [optional]:** [More Information Needed]
 - **Demo [optional]:** [Tonic/MistralMed_Chat]
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+This model can be used the same way you normally use mistral
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+This model can do better in medical question and answer scenarios.
 
 ### Downstream Use [optional]
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+This model is intended to be further fine tuned.
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+- Do Not Use As Is
+- Fine Tune This Model Further
+- For Educational Purposes Only
+- Benchmark your model usage
+- Evaluate the model before use
 
 Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
@@ -77,19 +64,49 @@ Users (both direct and downstream) should be made aware of the risks, biases and
 
 Use the code below to get started with the model.
 
-[More Information Needed]
+[Tonic/MistralMED_Chat](https://huggingface.co/Tonic/MistralMED_Chat)
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+[MedQuad](https://huggingface.co/datasets/keivalya/MedQuad-MedicalQnADataset/viewer/default/train)
 
 ### Training Procedure 
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+Dataset({
+    features: ['qtype', 'Question', 'Answer'],
+    num_rows: 16407
+})
+
+[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 12628, 264, 2718, 12271, 5122, 272, 14164, 5746, 9283, 302, 272, 2787, 12271, 390, 264, 2692, 908, 395, 9623, 304, 6836, 3069, 28723, 13, 3260, 908, 1023, 6685, 272, 2718, 1423, 24329, 304, 272, 908, 1580, 347, 624, 302, 272, 2296, 5936, 262, 674, 647, 464, 3134, 647, 464, 28721, 495, 28730, 410, 262, 296, 647, 464, 19928, 647, 464, 14876, 28730, 9122, 647, 464, 28713, 16939, 647, 464, 3134, 28730, 720, 11009, 352, 647, 464, 267, 1805, 416, 647, 464, 3134, 28730, 9122, 14303, 13, 1014, 9623, 1580, 347, 624, 302, 272, 2296, 28747, 5936, 861, 647, 464, 5128, 28730, 11023, 28730, 1408, 647, 464, 11023, 28730, 4395, 647, 464, 16239, 263, 647, 464, 274, 9312, 647, 464, 28599, 647, 464, 2383, 411, 647, 464, 7449, 28730, 4837, 8524, 647, 464, 3537, 28730, 13102, 7449, 647, 464, 10470, 28713, 647, 464, 13952, 28730, 266, 28730, 2453, 314, 647, 464, 3537, 28730, 8502, 28730, 11023, 647, 464, 3537, 28730, 7502, 28730, 11023, 647, 464, 4101, 3591, 1421, 13, 13, 27332, 15255, 12271, 28747, 13, 3195, 460, 272, 19724, 354, 393, 1082, 721, 402, 4475, 294, 689, 6519, 300, 3250, 17428, 325, 9162, 28755, 28731, 1550, 13, 13, 27332, 11736, 288, 9283, 28747, 13, 28741, 331, 19742, 1683, 288, 17428, 28725, 481, 358, 721, 282, 17428, 28725, 442, 1683, 20837, 636, 721, 282, 17428, 6948, 6556, 1837, 304, 27729, 5827, 2818, 356, 2425, 472, 28723, 23331, 28733, 21255, 314, 3076, 695, 10747, 28725, 1259, 390, 16779, 294, 8731, 17653, 28725, 993, 347, 4525, 916, 2948, 10139, 28723, 5800, 7193, 506, 4894, 369, 13147, 494, 361, 262, 28725, 264, 7876, 1307, 298, 3363, 2856, 799, 7692, 282, 18257, 28725, 349, 5645, 1835, 393, 15155, 28790, 297, 11781, 311, 28725, 736, 349, 708, 6740, 5566, 298, 1760, 871, 11935, 938, 354, 5827, 302, 393, 15155, 297, 10589, 28723, 13, 2]
+
+MistralForCausalLM(
+  (model): MistralModel(
+    (embed_tokens): Embedding(32000, 4096)
+    (layers): ModuleList(
+      (0-31): 32 x MistralDecoderLayer(
+        (self_attn): MistralAttention(
+          (q_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)
+          (k_proj): Linear4bit(in_features=4096, out_features=1024, bias=False)
+          (v_proj): Linear4bit(in_features=4096, out_features=1024, bias=False)
+          (o_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)
+          (rotary_emb): MistralRotaryEmbedding()
+        )
+        (mlp): MistralMLP(
+          (gate_proj): Linear4bit(in_features=4096, out_features=14336, bias=False)
+          (up_proj): Linear4bit(in_features=4096, out_features=14336, bias=False)
+          (down_proj): Linear4bit(in_features=14336, out_features=4096, bias=False)
+          (act_fn): SiLUActivation()
+        )
+        (input_layernorm): MistralRMSNorm()
+        (post_attention_layernorm): MistralRMSNorm()
+      )
+    )
+    (norm): MistralRMSNorm()
+  )
+  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
+)
 
 #### Preprocessing [optional]
 
@@ -98,50 +115,29 @@ Use the code below to get started with the model.
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Training regime:** 
+config = LoraConfig(
+    r=8,
+    lora_alpha=16,
+    target_modules=[
+        "q_proj",
+        "k_proj",
+        "v_proj",
+        "o_proj",
+        "gate_proj",
+        "up_proj",
+        "down_proj",
+        "lm_head",
+    ],
+    bias="none",
+    lora_dropout=0.05,  # Conventional
+    task_type="CAUSAL_LM",
+)
 
 #### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Data Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
+trainable params: 21260288 || all params: 3773331456 || trainable%: 0.5634354746703705
+TrainOutput(global_step=1000, training_loss=0.47226515007019043, metrics={'train_runtime': 3143.4141, 'train_samples_per_second': 2.545, 'train_steps_per_second': 0.318, 'total_flos': 1.75274075357184e+17, 'train_loss': 0.47226515007019043, 'epoch': 0.49})
 [More Information Needed]
 
 ## Environmental Impact
@@ -156,53 +152,180 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 - **Compute Region:** [More Information Needed]
 - **Carbon Emitted:** [More Information Needed]
 
-## Technical Specifications [optional]
+## Training Results
+
+ [1000/1000 52:20, Epoch 0/1]
+Step	Training Loss
+  50	0.474200
+  100	0.523300
+  150	0.484500
+  200	0.482800
+  250	0.498800
+  300	0.451800
+  350	0.491800
+  400	0.488000
+  450	0.472800
+  500	0.460400
+  550	0.464700
+  600	0.484800
+  650	0.474600
+  700	0.477900
+  750	0.445300
+  800	0.431300
+  850	0.461500
+  900	0.451200
+  950	0.470800
+1000	0.454900
 
 ### Model Architecture and Objective
 
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
+PeftModelForCausalLM(
+  (base_model): LoraModel(
+    (model): MistralForCausalLM(
+      (model): MistralModel(
+        (embed_tokens): Embedding(32000, 4096)
+        (layers): ModuleList(
+          (0-31): 32 x MistralDecoderLayer(
+            (self_attn): MistralAttention(
+              (q_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=4096, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=4096, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)
+              )
+              (k_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=4096, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=1024, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)
+              )
+              (v_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=4096, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=1024, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)
+              )
+              (o_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=4096, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=4096, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)
+              )
+              (rotary_emb): MistralRotaryEmbedding()
+            )
+            (mlp): MistralMLP(
+              (gate_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=4096, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=14336, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=4096, out_features=14336, bias=False)
+              )
+              (up_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=4096, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=14336, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=4096, out_features=14336, bias=False)
+              )
+              (down_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=14336, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=4096, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=14336, out_features=4096, bias=False)
+              )
+              (act_fn): SiLUActivation()
+            )
+            (input_layernorm): MistralRMSNorm()
+            (post_attention_layernorm): MistralRMSNorm()
+          )
+        )
+        (norm): MistralRMSNorm()
+      )
+      (lm_head): Linear(
+        in_features=4096, out_features=32000, bias=False
+        (lora_dropout): ModuleDict(
+          (default): Dropout(p=0.05, inplace=False)
+        )
+        (lora_A): ModuleDict(
+          (default): Linear(in_features=4096, out_features=8, bias=False)
+        )
+        (lora_B): ModuleDict(
+          (default): Linear(in_features=8, out_features=32000, bias=False)
+        )
+        (lora_embedding_A): ParameterDict()
+        (lora_embedding_B): ParameterDict()
+      )
+    )
+  )
+)
 #### Hardware
 
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
+A100
 
-## More Information [optional]
 
-[More Information Needed]
 
 ## Model Card Authors [optional]
 
-[More Information Needed]
+[Tonic](https://huggingface.co/Tonic)
 
 ## Model Card Contact
 
-[More Information Needed]
+[Tonic](https://huggingface.co/Tonic)
 
 
 ## Training procedure