Towards a Personal Health Large Language Model
Authors: name: Justin Cosentino; name: Anastasiya Belyaeva; name: Xin Liu; name: Nicholas A. Furlotte; name: Zhun Yang; name: Chace Lee; name: Erik Schenck; name: Yojan Patel; name: Jian Cui; name: Logan Douglas Schneider; name: Robby Bryant; name: Ryan G. Gomes; name: Allen Jiang; name: Roy Lee; name: Yun Liu; name: Javier Perez; name: Jameson K. Rogers; name: Cathy Speed; name: Shyam Tailor; name: Megan Walker; name: Jeffrey Yu; name: Tim Althoff; name: Conor Heneghan; name: John Hernandez; name: Mark Malhotra; name: Leor Stern; name: Yossi Matias; name: Greg S. Corrado; name: Shwetak Patel; name: Shravya Shetty; name: Jiening Zhan; name: Shruthi Prabhakara; name: Daniel McDuff; name: Cory Y. McLean
Published: 2024-06-10 17:16:49

Abstract: In health, most large language model (LLM) research has focused on clinical
tasks. However, mobile and wearable devices, which are rarely integrated into
such tasks, provide rich, longitudinal data for personal health monitoring.
Here we present Personal Health Large Language Model (PH-LLM), fine-tuned from
Gemini for understanding and reasoning over numerical time-series personal
health data. We created and curated three datasets that test 1) production of
personalized insights and recommendations from sleep patterns, physical
activity, and physiological responses, 2) expert domain knowledge, and 3)
prediction of self-reported sleep outcomes. For the first task we designed 857
case studies in collaboration with domain experts to assess real-world
scenarios in sleep and fitness. Through comprehensive evaluation of
domain-specific rubrics, we observed that Gemini Ultra 1.0 and PH-LLM are not
statistically different from expert performance in fitness and, while experts
remain superior for sleep, fine-tuning PH-LLM provided significant improvements
in using relevant domain knowledge and personalizing information for sleep
insights. We evaluated PH-LLM domain knowledge using multiple choice sleep
medicine and fitness examinations. PH-LLM achieved 79% on sleep and 88% on
fitness, exceeding average scores from a sample of human experts. Finally, we
trained PH-LLM to predict self-reported sleep quality outcomes from textual and
multimodal encoding representations of wearable data, and demonstrate that
multimodal encoding is required to match performance of specialized
discriminative models. Although further development and evaluation are
necessary in the safety-critical personal health domain, these results
demonstrate both the broad knowledge and capabilities of Gemini models and the
benefit of contextualizing physiological data for personal health applications
as done with PH-LLM.