Update README.md

README.md

@@ -1,70 +1,83 @@
 ---
 library_name: peft
 base_model: stabilityai/stablelm-3b-4e1t
+license: mit
+language:
+- en
+metrics:
+- bleu
+- bertscore
+- accuracy
+tags:
+- medical
 ---
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+Welcome to StableMed , it's a stable 3b llm - alpha fine tuned model for Medical Question and Answering.
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+This is a stable 3b finetune for medical QnA using MedQuad. 
+It's intended for education in public health and sanitation, 
+specifically to improve our understanding of outreach and communication.
 
 
 
-- **Developed by:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** [Tonic](https://huggingface.co/Tonic)
+- **Shared by [optional]:** [Tonic](https://huggingface.co/Tonic)
+- **Model type:** stable LM 3b - Alpha
+- **Language(s) (NLP):** English
+- **License:** MIT
+- **Finetuned from model [optional]:** [stabilityai/stablelm-3b-4e1t](https://huggingface.co/stabilityai/stablelm-3b-4e1t)
 
 ### Model Sources [optional]
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
+- **Repository:** [Tonic/stablemed](https://huggingface.co/Tonic/stablemed)
 - **Demo [optional]:** [More Information Needed]
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+Use this model for educational purposes only , do not use for decision support in the wild.
+
+Use this model for Medical Q n A.
+
+Use this model as a educational tool for "miniature" models.
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+Medical Question and Answering
 
-[More Information Needed]
 
 ### Downstream Use [optional]
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+Finetune this model to work in a network or swarm of medical finetunes.
 
-[More Information Needed]
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+do not use this model in the wild.
 
-[More Information Needed]
+do not use this model directly.
+
+do not use this model for real world decision support.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-[More Information Needed]
+[We use Giskard for evaluation - Coming Soon!]
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. 
+
+DO NOT USE THIS MODEL WITHOUT EVALUATION
+
+DO NOT USE THIS MODEL WITHOUT BENCHMARKING
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+DO NOT USE THIS MODEL WITHOUT FURTHER FINETUNING
 
 ## How to Get Started with the Model
 
@@ -76,66 +89,115 @@ Use the code below to get started with the model.
 
 ### Training Data
 
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-[More Information Needed]
+[Dataset](https://huggingface.co/datasets/keivalya/MedQuad-MedicalQnADataset) 
+
+```json
+output
+Dataset({
+    features: ['qtype', 'Question', 'Answer'],
+    num_rows: 16407
+})
+```
 
 ### Training Procedure 
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+```json
+trainable params: 12940288 || all params: 1539606528 || trainable%: 0.8404931886596937
+```
+
+Using Lora 
 
 #### Preprocessing [optional]
 
-[More Information Needed]
+Original:
+
+```json
+StableLMEpochForCausalLM(
+  (model): StableLMEpochModel(
+    (embed_tokens): Embedding(50304, 2560)
+    (layers): ModuleList(
+      (0-31): 32 x DecoderLayer(
+        (self_attn): Attention(
+          (q_proj): Linear4bit(in_features=2560, out_features=2560, bias=False)
+          (k_proj): Linear4bit(in_features=2560, out_features=2560, bias=False)
+          (v_proj): Linear4bit(in_features=2560, out_features=2560, bias=False)
+          (o_proj): Linear4bit(in_features=2560, out_features=2560, bias=False)
+          (rotary_emb): RotaryEmbedding()
+        )
+        (mlp): MLP(
+          (gate_proj): Linear4bit(in_features=2560, out_features=6912, bias=False)
+          (up_proj): Linear4bit(in_features=2560, out_features=6912, bias=False)
+          (down_proj): Linear4bit(in_features=6912, out_features=2560, bias=False)
+          (act_fn): SiLU()
+        )
+        (input_layernorm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
+        (post_attention_layernorm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
+      )
+    )
+    (norm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
+  )
+  (lm_head): Linear(in_features=2560, out_features=50304, bias=False)
+)
+```
 
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Training regime:**  <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
 #### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Data Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
+```json
+TrainOutput(global_step=2051, training_loss=0.6156479549198718, metrics={'train_runtime': 22971.4974, 'train_samples_per_second': 0.357, 'train_steps_per_second': 0.089, 'total_flos': 6.5950444363776e+16, 'train_loss': 0.6156479549198718, 'epoch': 0.5})
+```
+
+## Results
+
+| Value | Measurement |
+|-------|-------------|
+| 50    | 1.427000    |
+| 100   | 0.763200    |
+| 150   | 0.708200    |
+| 200   | 0.662300    |
+| 250   | 0.650900    |
+| 300   | 0.617400    |
+| 350   | 0.602900    |
+| 400   | 0.608900    |
+| 450   | 0.596100    |
+| 500   | 0.602000    |
+| 550   | 0.594700    |
+| 600   | 0.584700    |
+| 650   | 0.611000    |
+| 700   | 0.558700    |
+| 750   | 0.616300    |
+| 800   | 0.568700    |
+| 850   | 0.597300    |
+| 900   | 0.607400    |
+| 950   | 0.563200    |
+| 1000  | 0.602900    |
+| 1050  | 0.594900    |
+| 1100  | 0.583000    |
+| 1150  | 0.604500    |
+| 1200  | 0.547400    |
+| 1250  | 0.586600    |
+| 1300  | 0.554300    |
+| 1350  | 0.581000    |
+| 1400  | 0.578900    |
+| 1450  | 0.563200    |
+| 1500  | 0.556800    |
+| 1550  | 0.570300    |
+| 1600  | 0.599800    |
+| 1650  | 0.556000    |
+| 1700  | 0.592500    |
+| 1750  | 0.597200    |
+| 1800  | 0.559100    |
+| 1850  | 0.586100    |
+| 1900  | 0.581100    |
+| 1950  | 0.589400    |
+| 2000  | 0.581100    |
+| 2050  | 0.533100    |
 
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
 
 ## Environmental Impact
 
@@ -153,50 +215,167 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+with LORA :
+
+```json
+PeftModelForCausalLM(
+  (base_model): LoraModel(
+    (model): StableLMEpochForCausalLM(
+      (model): StableLMEpochModel(
+        (embed_tokens): Embedding(50304, 2560)
+        (layers): ModuleList(
+          (0-31): 32 x DecoderLayer(
+            (self_attn): Attention(
+              (q_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=2560, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=2560, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=2560, out_features=2560, bias=False)
+              )
+              (k_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=2560, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=2560, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=2560, out_features=2560, bias=False)
+              )
+              (v_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=2560, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=2560, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=2560, out_features=2560, bias=False)
+              )
+              (o_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=2560, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=2560, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=2560, out_features=2560, bias=False)
+              )
+              (rotary_emb): RotaryEmbedding()
+            )
+            (mlp): MLP(
+              (gate_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=2560, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=6912, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=2560, out_features=6912, bias=False)
+              )
+              (up_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=2560, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=6912, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=2560, out_features=6912, bias=False)
+              )
+              (down_proj): Linear4bit(
+                (lora_dropout): ModuleDict(
+                  (default): Dropout(p=0.05, inplace=False)
+                )
+                (lora_A): ModuleDict(
+                  (default): Linear(in_features=6912, out_features=8, bias=False)
+                )
+                (lora_B): ModuleDict(
+                  (default): Linear(in_features=8, out_features=2560, bias=False)
+                )
+                (lora_embedding_A): ParameterDict()
+                (lora_embedding_B): ParameterDict()
+                (base_layer): Linear4bit(in_features=6912, out_features=2560, bias=False)
+              )
+              (act_fn): SiLU()
+            )
+            (input_layernorm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
+            (post_attention_layernorm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
+          )
+        )
+        (norm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
+      )
+      (lm_head): Linear(
+        in_features=2560, out_features=50304, bias=False
+        (lora_dropout): ModuleDict(
+          (default): Dropout(p=0.05, inplace=False)
+        )
+        (lora_A): ModuleDict(
+          (default): Linear(in_features=2560, out_features=8, bias=False)
+        )
+        (lora_B): ModuleDict(
+          (default): Linear(in_features=8, out_features=50304, bias=False)
+        )
+        (lora_embedding_A): ParameterDict()
+        (lora_embedding_B): ParameterDict()
+      )
+    )
+  )
+)
+```
 
 ### Compute Infrastructure
 
-[More Information Needed]
+GCS
 
 #### Hardware
 
-[More Information Needed]
+T4
 
 #### Software
 
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
+transformers
+peft
+torch
+datasets
 
 ## Model Card Authors [optional]
 
-[More Information Needed]
+[Tonic](https://huggingface.co/Tonic)
 
 ## Model Card Contact
 
-[More Information Needed]
-
+[Tonic](https://huggingface.co/Tonic)
 
 ## Training procedure
 
@@ -216,4 +395,4 @@ The following `bitsandbytes` quantization config was used during training:
 ### Framework versions
 
 
-- PEFT 0.6.2.dev0
+- PEFT 0.6.2.dev0