Evaluation of ChatGPT and Gemini large language models for pharmacometrics with NONMEM
Authors: Shin Euibeom , Yu Yifan , Bies Robert , Ramanathan Murali 
Published: February 2025

To assess ChatGPT 4.0 (ChatGPT) and Gemini Ultra 1.0 (Gemini) large language models on NONMEM coding tasks relevantto pharmacometrics and clinical pharmacology. ChatGPT and Gemini were assessed on tasks mimicking real-world applications of NONMEM. The tasks ranged from providing a curriculum for learning NONMEM, an overview of NONMEM codestructure to generating code. Prompts in lay language to elicit NONMEM code for a linear pharmacokinetic (PK) modelwith oral administration and a more complex model with two parallel first-order absorption mechanisms were investigated.Reproducibility and the impact of “temperature” hyperparameter settings were assessed. The code was reviewed by twoNONMEM experts. ChatGPT and Gemini provided NONMEM curriculum structures combining foundational knowledgewith advanced concepts (e.g., covariate modeling and Bayesian approaches) and practical skills including NONMEM codestructure and syntax. ChatGPT provided an informative summary of the NONMEM control stream structure and outlinedthe key NONMEM Translator (NM-TRAN) records needed. ChatGPT and Gemini were able to generate code blocks forthe NONMEM control stream from the lay language prompts for the two coding tasks. The control streams contained focalstructural and syntax errors that required revision before they could be executed without errors and warnings. The codeoutput from ChatGPT and Gemini was not reproducible, and varying the temperature hyperparameter did not reduce theerrors and omissions substantively. Large language models may be useful in pharmacometrics for efficiently generating aninitial coding template for modeling projects. However, the output can contain errors and omissions that require correction.Citations: 1. ChatGPT Version Jan 2024 (2024) https:// chat. openai. com/.Accessed Mar 20242. Gemini Team, Anil R, Borgeaud S et al (2023) Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805.https:// doi. org/ 10. 48550/ arXiv. 2312. 11805. Accessed Dec 012023. https://ui.adsabs.harvard.edu/abs/2023arXiv231211805G3. Llama 2: open source, free for research and commercial use (2024)https://llama.meta.com/llama2/4. Touvron H, Martin L, Stone K, et al. Llama 2: Open Foundationand Fine-Tuned Chat Models. 2023: arXiv:2307.09288. https://doi. org/ 10. 48550/ arXiv. 2307. 09288. Accessed Jul 01 2023.https://ui.adsabs.harvard.edu/abs/2023arXiv230709288T5. Meet Claude (2024) https://www.anthropic.com/claude6. Orru G, Piarulli A, Conversano C, Gemignani A (2023) Humanlike problem-solving abilities in large language models usingChatGPT. Front Artif Intell 6:1199350. https://doi.org/10.3389/frai.2023.11993507. Roumeliotis KI, Tselikas ND (2023) ChatGPT and Open-AI Models: A Preliminary Review. Future Internet 15(6):1928. Radford A, Narasimhan K, Salimans T, Sutskever I (2018)Improving language understanding by generative pre-training9. Owen JS, Fiedler-Kelly J (2014) Introduction to population pharmacokinetic/pharmacodynamic analysis with nonlinear mixedeffects models. Wiley10. Pétricoul O, Cosson V, Fuseau E, Marchand M (2007) Populationmodels for drug absorption and enterohepatic recycling. Pharmacometrics: the science of quantitative pharmacology 345–38211. Bauer RJ (2019) NONMEM tutorial part I: description of commands and options, with simple examples of population analysis.CPT Pharmacometrics Syst Pharmacol 8(8):525–537. https://doi.org/10.1002/psp4.1240412. Sun H, Fadiran EO, Jones CD et al (1999) Population pharmacokinetics: a regulatory perspective. Clin Pharmacokinet 37:41–5813 Cloesmeijer ME, Janssen A, Koopman SF, Cnossen MH, Mathôt RA,consortium S, (2024) ChatGPT in pharmacometrics? Potential opportunities and limitations. British J Clin Pharmacol 90(1):360–36514. Shin E, Ramanathan M (2024) Evaluation of prompt engineeringstrategies for pharmacokinetic data analysis with the ChatGPT largelanguage model. J Pharmacokinet Pharmacodyn 51(2):101–10815. Bard Large language model. 2023. https://bard.google.com16. Fidler M, Wilkins JJ, Hooijmaijers R et al (2019) NonlinearMixed-Effects Model Development and Simulation Using nlmixrand Related R Open-Source Packages. CPT Pharmacometrics SystPharmacol 8(9):621–633. https://doi.org/10.1002/psp4.1244517. Bonate PL, Barrett JS, Ait-Oudhia S et al (2023) Training thenext generation of pharmacometric modelers: a multisector perspective. J Pharmacokinet Pharmacodyn. https://doi.org/10.1007/s10928-023-09878-418. Shin E, Ramanathan M (2023) Evaluation of prompt engineeringstrategies for pharmacokinetic data analysis with the ChatGPTlarge language model. J Pharmacokinet Pharmacodyn. https://doi.org/10.1007/s10928-023-09892-619. mrgsolve: Simulate from ODE-Based Models. R package version1.4.1. Metrum Research Group; 2024. https://github.com/metrumresearchgroup/mrgsolve20. Fidler M, Hooijmaijers R, Schoemaker R, Wilkins JJ, Xiong Y,Wang W (2021) R and nlmixr as a gateway between statisticsand pharmacometrics. CPT Pharmacometrics Syst Pharmacol10(4):283–285. https://doi.org/10.1002/psp4.1261821. nlmixr: an R package for population PKPD modeling. 2019.https://nlmixrdevelopment.github.io/nlmixr/index.html22. Anonymous. Monolix documentation. Lixoft-SimulationsPlus.Accessed March 13, 2024, 2024. https:// monol ix. lixoft. com/single-page/23. Stan Reference Manual. NumFOCUS; 2011. https://mc-stan.org/docs/reference-manual/24. Cloesmeijer ME, Janssen A, Koopman SF, Cnossen MH, MathotRAA, Symphony consortium (2024) ChatGPT in pharmacometrics? Potential opportunities and limitations. Br J Clin Pharmacol90(1):360–365. https://doi.org/10.1111/bcp.1589525. Frieder S, Pinchetti L, Chevalier A et al (2023) Mathematicalcapabilities of ChatGPT. arXiv. arXiv:2301.13867v226. Yuan Z, Yuan H, Tan C, Wang W, Huang S. How well do largelanguage models perform in arithmetic tasks? arXiv. 2023:arXiv:2304.0201527. Alkaissi H, McFarlane SI (2023) Artificial Hallucinations inChatGPT: Implications in Scientific Writing. Cureus J Med Sci15(2).https://doi.org/10.7759/cureus.3517928. Beutel G, Geerits E, Kielstein JT (2023) Artificial hallucination: GPT on LSD? Crit Care 27(1):148. https://doi.org/10.1186/s13054-023-04425-6