Update README.md

README.md

@@ -3,199 +3,219 @@ library_name: transformers
 tags: []
 ---
 
-# Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+## Model Description
 
+<!-- Provide a longer summary of what this model is/does. -->
+LoRA adapter weights from fine-tuning [BioMobileBERT](https://huggingface.co/nlpie/bio-mobilebert) on the MIMIC-III mortality prediction task. The [PEFT](https://github.com/huggingface/peft) was used and the model was trained for a maximum of 5 epochs with early stopping, full details can be found at the [github repo](https://github.com/nlpie-research/efficient-ml). 
 
+<!-- - **Developed by:** Niall Taylor -->
+<!-- - **Shared by [Optional]:** More information needed -->
+- **Model type:** Language model LoRA adapter
+- **Language(s) (NLP):** en
+- **License:** apache-2.0
+- **Parent Model:** Llama-2-7b-hf
+- **Resources for more information:** 
+    - [GitHub Repo](https://github.com/nlpie-research/efficient-ml)
+    - [Associated Paper](https://arxiv.org/abs/2402.10597)
 
-## Model Details
+<!-- # Uses -->
 
-### Model Description
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-<!-- Provide a longer summary of what this model is. -->
+<!-- ## Direct Use -->
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
 
-### Model Sources [optional]
 
-<!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+<!-- ## Downstream Use [Optional] -->
 
-## Uses
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
+ 
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+# How to Get Started with the Model
 
-### Direct Use
+Use the code below to get started with the model.
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+<details>
+<summary> Click to expand </summary>
 
-[More Information Needed]
 
-### Downstream Use [optional]
+```python
+from peft import AutoPeftModelForCausalLM, AutoPeftModelForSequenceClassification
+from transformers import AutoTokenizer
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+model_name = "NTaylor/bio-mobilebert-mimic-mp-lora"
 
-[More Information Needed]
+# load using AutoPeftModelForSequenceClassification
+model = AutoPeftModelForSequenceClassification.from_pretrained(lora_id)
 
-### Out-of-Scope Use
+# use base llama tokenizer
+tokenizer = AutoTokenizer.from_pretrained("nlpie/bio-mobilebert")
+
+# example input
+text = "Clinical note..."
+inputs = tokenizer(text, return_tensors="pt")
+outputs = reloaded_model(**inputs)
+# extract prediction from outputs based on argmax of logits
+pred = torch.argmax(outputs.logits, axis = -1)
+print(f"Prediction is: {pred}") # binary classification: 1 for mortality
+```
+
+
+</details>
+
+## Out-of-Scope Use
 
 <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
 
-[More Information Needed]
+This model and LoRA weights were trained on the MIMIC-III dataset and are not intended for use on other datasets, nor be used in any real clinical setting. The experiments were conducted as a means of exploring the potential of LoRA adapters for clinical NLP tasks, and the model should not be used for any other purpose.
 
-## Bias, Risks, and Limitations
+
+<!-- # Bias, Risks, and Limitations -->
 
 <!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-[More Information Needed]
+<!-- Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. -->
+
 
-### Recommendations
+<!-- ## Recommendations -->
 
 <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-## How to Get Started with the Model
 
-Use the code below to get started with the model.
 
-[More Information Needed]
+<!-- 
+# Training Details
 
-## Training Details
+## Training Data
 
-### Training Data
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+<!-- More information on training data needed -->
 
-[More Information Needed]
 
-### Training Procedure 
+<!-- ## Training Procedure -->
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-#### Preprocessing [optional]
+<!-- ### Preprocessing
 
-[More Information Needed]
+More information needed -->
 
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
+<!-- ### Speeds, Sizes, Times -->
 
 <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
-[More Information Needed]
-
-## Evaluation
+<!-- More information needed -->
+ 
+<!-- # Evaluation -->
 
 <!-- This section describes the evaluation protocols and provides the results. -->
 
-### Testing Data, Factors & Metrics
+<!-- ## Testing Data, Factors & Metrics
 
-#### Testing Data
+### Testing Data -->
 
-<!-- This should link to a Dataset Card if possible. -->
+<!-- This should link to a Data Card if possible. -->
 
-[More Information Needed]
+<!-- More information needed -->
 
-#### Factors
+
+<!-- ### Factors -->
 
 <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
 
-[More Information Needed]
+<!-- More information needed -->
 
-#### Metrics
+<!-- ### Metrics -->
 
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
+<!-- More information needed -->
 
-#### Summary
+<!-- ## Results 
 
+More information needed
 
+# Model Examination
 
-## Model Examination [optional]
+More information needed
 
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
+# Environmental Impact -->
 
 <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
+<!-- 
 Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+- **Hardware Type:** More information needed
+- **Hours used:** More information needed
+- **Cloud Provider:** More information needed
+- **Compute Region:** More information needed
+- **Carbon Emitted:** More information needed
 
-## Technical Specifications [optional]
+# Technical Specifications [optional]
 
-### Model Architecture and Objective
+## Model Architecture and Objective
 
-[More Information Needed]
+More information needed
 
-### Compute Infrastructure
+## Compute Infrastructure
 
-[More Information Needed]
+More information needed
 
-#### Hardware
+### Hardware
 
-[More Information Needed]
+More information needed
 
-#### Software
+### Software
 
-[More Information Needed]
+More information needed --> 
 
-## Citation [optional]
+# Citation
 
 <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
 **BibTeX:**
+``````
+@misc{taylor2024efficiency,
+      title={Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks}, 
+      author={Niall Taylor and Upamanyu Ghose and Omid Rohanian and Mohammadmahdi Nouriborji and Andrey Kormilitzin and David Clifton and Alejo Nevado-Holgado},
+      year={2024},
+      eprint={2402.10597},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+``````
+<!-- **APA:** -->
+
+<!-- More information needed -->
+
+<!-- # Glossary [optional] -->
 
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
 
-## Glossary [optional]
+<!-- More information needed -->
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+<!-- # More Information [optional] -->
 
-[More Information Needed]
+<!-- More information needed -->
 
-## More Information [optional]
+<!-- # Model Card Authors [optional] -->
 
-[More Information Needed]
+<!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. -->
 
-## Model Card Authors [optional]
+<!-- More information needed -->
 
-[More Information Needed]
+<!-- # Model Card Contact -->
 
-## Model Card Contact
+<!-- More information needed -->
 
-[More Information Needed]