MedMobile: A mobile-sized language model with expert-level clinical
  capabilities
Authors: name: Krithik Vishwanath, name: Jaden Stryker, name: Anton Alaykin, name: Daniel Alexander Alber, name: Eric Karl Oermann
Organization: NYU Langone Health OLAB

Language models (LMs) have demonstrated expert-level reasoning and recall
abilities in medicine. However, computational costs and privacy concerns are
mounting barriers to wide-scale implementation. We introduce a parsimonious
adaptation of phi-3-mini, MedMobile, a 3.8 billion parameter LM capable of
running on a mobile device, for medical applications. We demonstrate that
MedMobile scores 75.7% on the MedQA (USMLE), surpassing the passing mark for
physicians (~60%), and approaching the scores of models 100 times its size. We
subsequently perform a careful set of ablations, and demonstrate that chain of
thought, ensembling, and fine-tuning lead to the greatest performance gains,
while unexpectedly retrieval augmented generation fails to demonstrate
significant improvements