ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical
  Prediction?
Authors: name: Canyu Chen, name: Jian Yu, name: Shan Chen, name: Che Liu, name: Zhongwei Wan, name: Danielle Bitterman, name: Fei Wang, name: Kai Shu

Large Language Models (LLMs) hold great promise to revolutionize current
clinical systems for their superior capacities on medical text processing tasks
and medical licensing exams. Meanwhile, traditional ML models such as SVM and
XGBoost have still been mainly adopted in clinical prediction tasks. An
emerging question is Can LLMs beat traditional ML models in clinical
prediction? Thus, we build a new benchmark ClinicalBench to comprehensively
study the clinical predictive modeling capacities of both general-purpose and
medical LLMs, and compare them with traditional ML models. ClinicalBench
embraces three common clinical prediction tasks, two databases, 14
general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through
extensive empirical investigation, we discover that both general-purpose and
medical LLMs, even with different model scales, diverse prompting or
fine-tuning strategies, still cannot beat traditional ML models in clinical
prediction yet, shedding light on their potential deficiency in clinical
reasoning and decision-making. We call for caution when practitioners adopt
LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap
between LLMs' development for healthcare and real-world clinical practice.