Model Description

LoRA adapter weights from fine-tuning BioMobileBERT on the MIMIC-III mortality prediction task. The PEFT was used and the model was trained for a maximum of 5 epochs with early stopping, full details can be found at the github repo.

Model type: Language model LoRA adapter
Language(s) (NLP): en
License: apache-2.0
Parent Model: BioMobileBERT
Resources for more information:
- GitHub Repo
- Associated Paper

How to Get Started with the Model

Use the code below to get started with the model.

Click to expand

from peft import AutoPeftModelForCausalLM, AutoPeftModelForSequenceClassification
from transformers import AutoTokenizer

model_name = "NTaylor/bio-mobilebert-mimic-mp-lora"

# load using AutoPeftModelForSequenceClassification
model = AutoPeftModelForSequenceClassification.from_pretrained(lora_id)

# use base llama tokenizer
tokenizer = AutoTokenizer.from_pretrained("nlpie/bio-mobilebert")

# example input
text = "Clinical note..."
inputs = tokenizer(text, return_tensors="pt")
outputs = reloaded_model(**inputs)
# extract prediction from outputs based on argmax of logits
pred = torch.argmax(outputs.logits, axis = -1)
print(f"Prediction is: {pred}") # binary classification: 1 for mortality

Out-of-Scope Use

This model and LoRA weights were trained on the MIMIC-III dataset and are not intended for use on other datasets, nor be used in any real clinical setting. The experiments were conducted as a means of exploring the potential of LoRA adapters for clinical NLP tasks, and the model should not be used for any other purpose.

Citation

BibTeX:

@misc{taylor2024efficiency,
      title={Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks}, 
      author={Niall Taylor and Upamanyu Ghose and Omid Rohanian and Mohammadmahdi Nouriborji and Andrey Kormilitzin and David Clifton and Alejo Nevado-Holgado},
      year={2024},
      eprint={2402.10597},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}