Evaluating Advanced Large Language Models for Pulmonary Disease Diagnosis Using Portable Spirometer Data: A Comparative Analysis of Gemini-1.5 Pro, GPT-4o, and Claude-3.5 Sonnet

Authors: jin-hyun  park
chinock  cheong
sanghee  kang
inpyo  lee
sungjin  lee
kisang  yoon
hwamin  lee
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-07-05 14:27:08

Abstract: Pulmonary function tests (PFTs) are vital for diagnosing various pulmonary conditions, including chronic obstructive pulmonary disease (COPD) and asthma. Traditional PFTs, conducted using laboratory-based spirometers, are accurate but costly and require skilled technicians. Recent advancements in portable spirometry and large language models (LLMs) offer promising alternatives for remote diagnostics and clinical decision support. This study evaluates the performance of three advanced LLMs: Gemini 1.5 Pro, GPT 4o, and Claude 3.5 Sonnet in understanding and interpreting PFTs data. The models were assessed using three prompt types: zero shot, guidelines enhanced, and few shot, and their performance was measured in terms of accuracy, precision, recall, F1 score, and processing speed. Results indicate that Claude 3.5 Sonnet consistently outperformed the other models across all metrics, demonstrating superior comprehension and classification abilities. Error analysis revealed specific areas for improvement, particularly in logical reasoning and adherence to guidelines. The findings highlight the potential 
of LLMs to enhance diagnostic processes and reduce healthcare costs, while also emphasizing the need for further research to address data privacy, interoperability, and ethical considerations for clinical integration. Future efforts should focus on leveraging open-source models and expanding datasets to optimize LLMs for real-world medical applications.

Main paper: 
 i. introduction: 
 Pulmonary function tests (PFTs) are essential diagnostic tools used to assess lung health and diagnose various pulmonary conditions, including chronic obstructive pulmonary disease (COPD), asthma, and restrictive lung diseases [1] [2] [3] . The Global Initiative for Chronic Obstructive Lung Disease (GOLD) report highlights that COPD is a leading cause of morbidity and mortality worldwide, necessitating accurate diagnostic methods [4] . Traditionally, PFTs are conducted using laboratory-based spirometers, which provide precise measurements but come with high costs and require skilled technicians. This makes the approach impractical and leads to substantial increases in medical expenses for patients directed to tertiary hospitals [5] .Several studies have indicated that measurements obtained using portable spirometers are nearly identical to those from traditional, laboratory-based spirometers [6, 7] . This suggests that portable spirometers, with their cost-effectiveness, portability, and ease of use, can serve as a viable alternative to conventional PFTs devices [8] . Recently, the integration of portable spirometry with mobile applications has enabled patients to easily measure their lung capacity at home and transmit the results to healthcare providers. This advancement goes beyond the development of remote treatment plans, extending to the creation of diverse software services related to pulmonary health [9] [10] [11] .Moreover, with the advent of large language models (LLMs), the medical field has seen substantial advancements [12] , including the development of medical domain LLMs capable of passing medical licensing exams [13] , as well as applications in disease prediction and diagnostic consultation chatbots [14] . Ongoing research is also aimed at creating LLMs that can perform tasks beyond simple algorithmic classification of pulmonary function test data, generating pulmonary function test charts at the proficiency level of medical residents [15] . Such innovations have the potential to significantly reduce unnecessary medical expenditures and alleviate the burden of frequent hospital visits for patients.Therefore, the aim of this study is to conduct foundational research by evaluating and comparing the comprehension of pulmonary function test data among three leading state-of-theart large parameter LLMs, namely the Gemini 1.5 Pro [16] , the GPT 4o [17] and the Claude 3.5 Sonnet [18] , which have demonstrated top tier performance across various assessments.The methodology section provides a comprehensive overview of the experimental design, data collection, preprocessing, and the specific approaches used to evaluate the performance of three advanced LLMs in interpreting PFTs data.The study involved a total of 200 participants who were recruited from various primary healthcare settings. Participants included individuals across different age groups, genders, and varying health statuses to ensure a representative sample. Each participant provided informed consent before participating in the study, ensuring ethical standards were maintained. Data were collected using the portable spirometer BULO M, developed by BREATHINGS Co., Ltd. This device is known for its accuracy and portability, which makes it ideal for use in both clinical and home settings. The spirometer measured two key pulmonary parameters: forced vital capacity (FVC) and forced expiratory volume in one second (FEV1). These parameters are critical in diagnosis of pulmonary diseases.The predicted normal values for FVC and FEV1, used to classify pulmonary disease types, were calculated using Dr. Choi's equation [19] , a widely accepted formula for predicting normal values based on the Asian population and considering age, sex, height, and weight. This normalization was essential to account for individual variations and ensure that the comparisons made across participants were valid.Using the predicted normal values for FVC and FEV1, the data were classified into four pulmonary disease categories: normal, restrictive, obstructive, or combined. The classification followed established diagnostic guidelines:• Normal: FEV1/FVC ratio of 70% or higher and FVC 80% or higher of the predicted normal value for FVC; labeled as 'normal'.• Restrictive: FEV1/FVC ratio of 70% or higher and FVC less than 80% of the predicted normal value for FVC; labeled as 'restrictive'.• Obstructive: FEV1/FVC ratio of less than 70% and FVC 80% or higher of the predicted normal value for FVC; labeled as 'obstructive'.• Combined: Meeting the criteria for both restrictive and obstructive conditions; labeled as 'combined'.The ground truth labels for each participant were determined based on the above contents.To facilitate the evaluation of the LLMs, the numerical PFTs data were transformed into standardized sentences. These sentences followed a specific structure, as outlined in the study's guidelines, to ensure consistency and clarity. Each sentence included the measured FVC and FEV1 values and the predicted normal values for these measurements to maintain uniformity.The study evaluated three state of the art LLMs: Gemini 1.5 Pro (developed by Google), GPT 4o (developed by OpenAI), and Claude 3.5 Sonnet (developed by Anthropic). These models were chosen based on their superior performance across various natural language processing tasks and their availability via application programming interfaces.Each model was accessed through its respective application programming interface and configured with specific parameters to ensure consistent performance across all tests:• Temperature: Set to 0.0 to ensure deterministic outputs.This parameter controls the randomness of the model's predictions, with a lower temperature resulting in more consistent and predictable responses.• Top p: Set to 1.0, which means the entire probability distribution of possible tokens was considered during the generation of responses. This setting ensures that no potential answers are excluded from consideration.Prompt engineering is a crucial aspect of evaluating LLMs, as the quality and structure of prompts can significantly impact the models' performances. This study employed three types of prompts to assess the models' abilities to interpret PFTs data and generate accurate clinical rationales.Each case was processed in a separate session to ensure that prior inputs did not influence subsequent responses, maintaining the integrity and independence of each test case.Zero shot prompting involves providing the model with PFTs data without any additional context or examples. This type of prompt assesses the model's inherent ability to understand and interpret the data based solely on its pre-existing knowledge. Each model was given a prompt structured as follows, corresponding to Fig. 1: • Entry of transformed PFTs data.• A request to generate a diagnosis (normal, restrictive, obstructive, or combined).• A request to provide a clinical rationale in four sentences.To improve the models' performance, zero shot prompt were enhanced with specific diagnostic guidelines and the chain of thought (CoT) technique [20] . The guidelines provided explicit criteria for classifying pulmonary diseases, while the CoT technique involved breaking down the reasoning process into sequential steps. The guidelines enhanced prompt structure included the following, corresponding to Fig. 1: • The same PFTs data as provided in the zero shot prompt.• Detailed diagnostic guidelines to assist the model in making accurate classifications.• CoT sentences to guide the model through the reasoning process step by step. Few shot prompting involves providing the model with a small number of example cases in addition to the primary prompt. This approach helps the model understand the task better by learning from the examples. For this study, four example datasets were included in the prompts. The structure of the few shot prompt, as shown in Fig. 1 , was as follows:• The same PFTs data input as the previous two prompts.• The same diagnostic guidelines and CoT sentences input as the previous prompts.• Input of four example responses.The quantitative performance of the LLMs in classifying different types of pulmonary diseases was assessed through a variety of evaluation metrics:• Accuracy: The proportion of correct predictions out of the total predictions.• Precision: The ratio of true positives to the sum of true positives and false positives, indicating the model's ability to avoid false positives and its reliability in positive predictions.• Recall: The ratio of true positives to the sum of true positives and false negatives, reflecting the model's ability to identify all relevant instances and its effectiveness in capturing actual positives.• F1 Score: The harmonic mean of precision and recall, providing a balanced measure of the model's performance, particularly useful when there is an uneven class distribution.These metrics were selected to ensure a comprehensive understanding of how well the models performed in distinguishing between the four categories of pulmonary conditions.The clinical rationales generated by the LLMs were analyzed for different types of errors, including:• Comparison Operation Errors (COE): These errors occurred when the model's logical operations involving numerical comparisons were flawed, despite the calculations being correct.• Guideline Deviations (GD): These errors happened when the model deviated from the provided diagnostic guidelines, either by offering alternative explanations or additional clinical opinions that were not aligned with the guidelines.• Misdiagnoses (MD): In these cases, the model generated a clinical rationale that aligned with the diagnostic guidelines, but the final diagnosis was incorrect.• Miscalculations (MC): These errors involved incorrect calculations during the reasoning process, which could lead to incorrect conclusions.In this section, we present the detailed results of our evaluation of the three LLMs: Gemini 1.5 Pro, GPT 4o, and Claude 3.5 Sonnet, on their comprehension of PFTs data. The evaluation was conducted across three types of prompts: zero shot, guidelines enhanced, and few shot. This includes the quantitative performance metrics, such as accuracy, precision, recall, and F1 score, along with the time efficiency and the number of tokens generated by the three models for 200 iterations to assess their processing speed and efficiency under different prompt types. Additionally, we present a comprehensive error analysis to provide insights into the models' capabilities and limitations.The zero shot performance of the models was evaluated by providing them with only the PFTs results without any additional context. Table I summarizes the accuracy, precision, recall, and F1 score for each model across the four pulmonary disease types: normal, restrictive, obstructive, and combined. Claude 3.5 Sonnet outperformed the other models in the zero shot setting across all disease types, achieving the highest accuracy, precision, recall, and F1 score. This indicates that Claude 3.5 Sonnet has a superior inherent ability to understand and classify pulmonary diseases based solely on the provided PFTs data. The guidelines enhanced prompt included specific diagnostic criteria and the CoT technique to aid the models in their reasoning process. Table II presents the performance metrics for each model using the guidelines enhanced prompt.With the inclusion of diagnostic guidelines and CoT, overall accuracy showed minimal change, but there was an improvement in the classification of the 'combined' type. The Claude 3.5 Sonnet maintained its lead, achieving the highest scores across all metrics and disease types. The enhancements provided clearer criteria and reasoning steps, which helped the models make more accurate diagnoses.The few shot prompt included four example datasets to provide additional context and improve the models' understanding. Table III details the performance metrics for each model with the few shot prompt.The few shot prompting further improved the performance of all models, with Claude 3.5 Sonnet continuing to outperform Gemini 1.5 Pro and GPT 4o. The additional examples provided context that helped the models better understand the classification task, leading to higher accuracy and more reliable predictions. In contrast, Claude 3.5 Sonnet took the most time for all prompt types, particularly with the few shot prompt, where the total time was 1090.10 seconds and the average per iteration was 5.04 seconds.In this study, we conducted a detailed error analysis to understand the types and frequencies of errors made by Gemini 1.5 Pro, GPT 4o, and Claude 3.5 Sonnet in interpreting PFTs data. Fig. 2 illustrates the error rate for each model across the different prompt types (zero shot, guidelines enhanced, and few shot). The results reveal that Claude 3.5 Sonnet consistently outperformed both Gemini 1.5 Pro and GPT 4o in all prompt settings, achieving the lowest error rate.In the zero shot setting, Claude 3.5 Sonnet achieved an error rate of 20%, significantly lower than GPT 4o (26%) and Gemini 1.5 Pro (45%). This indicates that Claude 3.5 Sonnet has a superior inherent ability to understand and classify pulmonary disease types based on PFTs data without additional context. The introduction of diagnostic guidelines and the CoT technique resulted in improved performance for all models. Claude 3.5 Sonnet again demonstrated the lowest error rate at 19%, followed by GPT 4o at 27% and Gemini 1.5 Pro at 52%. Despite the addition of guidelines, the error rate for both Gemini 1.5 Pro and GPT 4o slightly increased. The inclusion of four example datasets further reduced the error rate across all models. Claude 3.5 Sonnet achieved the lowest error rate at 16%, GPT 4o improved to 18%, and Gemini 1.5 Pro showed a notable reduction to 28%. The few shot learning approach significantly enhanced the models' ability to leverage additional context, resulting in more accurate classifications.Fig. 3 categorizes the examples of errors generated by the LLMs into four types. COE referred to instances where the calculations were correct, but the logical operation involving comparisons with values such as 70 or 80 was flawed. GD indicated cases where the LLMs deviated from the provided diagnostic guidelines during the second and third prompts, providing alternative explanations or additional clinical opinions.MD occurred when the clinical rationale aligns with the diagnostic guidelines, but the "diagnosis:" field contained an incorrect answer. Lastly, MC denoted calculation errors, which were observed only in the Gemini 1.5 Pro model. These errors cooccurred with 3 instances of MD, 6 instances of GD, and 10 instances of COE. The Gemini 1.5 Pro displayed a more balanced distribution of error types, including significant instances of COEs, GDs, and MDs. This indicates that while the model had some understanding of the guidelines, it frequently made logical errors and provided incorrect diagnoses despite correct rationales, compounded further by instances of MCs.The GPT 4o, with relatively fewer errors, showed a predominant issue with COEs, suggesting that the model's main challenge was in the logical interpretation of numerical data rather than following the guidelines or providing accurate diagnoses.The Claude 3.5 Sonnet generated the least number of errors overall, with the majority being GDs. The tendency of Claude 3.5 Sonnet to deviate slightly from the guidelines while still offering clinically relevant insights indicates its advanced understanding of the context, albeit with a propensity to provide more information than necessary.The results highlight the potential of advanced LLMs, particularly the Claude 3.5 Sonnet, in the medical field for diagnosing pulmonary diseases. Although Claude 3 Opus is a different model from the Claude 3.5 Sonnet and is not specialized for the medical domain, it demonstrated performance [21] on the PubMedQA dataset [22] nearly equivalent to Google's Med-PaLM 2 [23] , which is designed specifically for medical applications. This suggests that the contexts of our experiment and those results are similar.The ability of these models to accurately interpret PFTs data and generate clinical rationales can significantly enhance diagnostic processes [24] , reduce healthcare costs [25] , and improve patient outcomes [26, 27] . Integrating such AI technologies into clinical settings can provide healthcare professionals with reliable decision support tools [28] , leading to more efficient and accurate diagnoses [29] , and potentially advancing to specific tasks such as summarizing risk levels based on preoperative pulmonary function test results [30] .However, implementing LLMs in clinical settings requires careful consideration of data privacy and integration with existing systems [31] . Key steps include ensuring security through local deployment and anonymization, as well as complying with healthcare regulations like the Health Insurance Portability and Accountability Act (HIPAA) and establishing interoperability with electronic health record (EHR) systems using a standardized application programming interface (API) [32, 33] . Additionally, fine tuning LLMs with domain specific data and incorporating human oversight are crucial for enhancing accuracy and maintaining clinical standards [34] . Therefore, balancing technological advancements with ethical and practical considerations will be critical for the successful integration of LLMs in healthcare [35] .Furthermore, for the future development of LLMs capable of generating pulmonary function test charts for use in real world medical settings, it is essential to utilize open sourced LLMs [36] . Related research has demonstrated the application of large parameter LLMs, such as GPT 4, to clinical data for diagnosing Alzheimer's disease and generating clinical rationales, subsequently using knowledge distillation to train open sourced models [37] .The key aspect is that large parameter LLMs generate instruction following data for training open sourced models [38] . This study's significance lies in comparing three state of the art models, which currently demonstrate superior performance, using PFTs data, and analyzing their errors to provide insights into the understanding of pulmonary diseases by large parameter LLMs.Additionally, exploring and fine tuning other open sourced LLMs can help identify models with even greater potential for clinical applications [39] . Further efforts should focus on minimizing specific types of errors, such as COEs and GDs, to enhance the models' reliability and accuracy [40] .This study evaluated the performance of three state of the art LLMs: Gemini 1.5 Pro, GPT 4o, and Claude 3.5 Sonnet in interpreting PFTs data and generating clinical rationales for diagnosing pulmonary diseases. Notably, there are no recent studies that directly compare the performance of these latest models on specific clinical tasks, making this research a novel contribution to the field. The Claude 3.5 Sonnet consistently outperformed the Gemini 1.5 Pro and the GPT 4o across all evaluation metrics and prompt types. It demonstrated the highest accuracy, precision, recall, and F1 score, indicating its superior ability to understand and classify PFTs data. The model's performance was enhanced by the inclusion of diagnostic guidelines and few shot examples, showcasing its capacity to learn effectively from additional context. The Gemini 1.5 Pro and the GPT 4o also showed improvements with enhanced prompts, but their overall performance remained below that of the Claude 3.5 Sonnet.From the perspective of error analysis, the Gemini 1.5 Pro exhibited a balanced distribution of errors, including COEs, GDs, and MDs, indicating a need for better logical reasoning and adherence to guidelines. The GPT 4o, while demonstrating fewer errors overall, struggled primarily with COEs, suggesting a need for improved numerical interpretation capabilities.While this study provides valuable insights, it is limited by the sample size and the specific models evaluated. Future research should involve larger and more diverse datasets to better generalize the findings. Additionally, further efforts should focus on minimizing specific types of errors, such as COEs and GDs, to enhance the models' reliability and accuracy.This research was supported by the Bio&Medical Technology Development Program of the National Research Foundation (NRF) funded by the Korean government (MSIT) (No. RS-2024-00440371) and by the NRF grant funded by the Korean government (MSIT) (No. RS-2024-00457381). 
 
 Appendix:  



Reviews: review: title: title: The paper considers large language models for pulmonary disease diagnosis. It is well-written paper and covers a versatile analysis.
paper_summary: paper_summary: In the paper pulmonary function tests (PFTs) which are crucial for the accurate diagnosis and prognosis of pulmonary diseases are considered. Large language models (LLMs) are considered to overcome the diffuculties of traditional spirometers. With the software provided, the integration of portable spirometry with mobile applications has enabled patients to easily measure their lung capacity at home and transmit the results to healthcare providers. For this application three different LLMs, Gemini-1.5 Pro, GPT-4o, and Claude-3.5 Sonnet, are considered to understand and interpret PFTs data. Data from 200 participants are used and the models are tested through zero-shot, guidelines-enhanced, and few-shot prompts. Performance metrics included accuracy, precision, recall, and F1-score for disease classification, along with error analysis of clinical rationale. Results indicate that Claude-3.5 Sonnet consistently outperformed the other models.
strength_weakness: strength_weakness: The words at the end of the lines are cut improperly, making it difficult to follow, please fix this small problem.
questions: questions: I have no questions for the authors.
title: title: Good and Interesting Paper
paper_summary: paper_summary: The paper evaluated the performance of three leading state-of-the-art Advanced Large Language Models (LLM), Gemini-1.5 Pro, GPT-4o, and Claude-3.5 Sonnet, for pulmonary disease diagnosis using data from portable spirometers. The study included 200 participants and the models were tested through zero-shot, guidelines-enhanced, and few-shot prompts. The numerical data were transformed into standardized sentences following a specific structure for consistency and clarity. Performance metrics included accuracy, precision, recall, and F1-score, with Claude-3.5 Sonnet outperforming the other models.
strength_weakness: strength_weakness: 1.	Relatively small dataset. Additional data must be included to ensure the generalizability of the models.
2.	The paper does not report the time required for the LLMs to evaluate the data. Time metrics can be beneficial in order to assess the efficiency of the models in case of time-sensitive decision-making.

Formatting issues.
- The margins on the first page are not correctly set according to the rest of the paper.
- The paper does not enable hyphenation.
- Section II-E is not formatted in a column-wise manner. Please check the entire manuscript for this.
- Tables are very large, with large margins and font size.
questions: questions: 1.	How representative are these cases of the broader population? Is it possible for demographic differences to impact the performance of such models?
2.	What specific steps would be necessary to implement these LLMs in a clinical setting, particularly concerning data privacy and integration with existing medical systems?
title: title: Evaluating Advanced Large Language Models for Pulmonary Disease Diagnosis Using Portable Spirometer Data: A Comparative Analysis of Gemini 1.5 Pro, GPT 4o , and Claude 3.5 Sonnet
paper_summary: paper_summary: •	The study aims to evaluate and compare the performance of three advanced LLMs—Gemini-1.5 Pro, GPT-4o, and Claude-3.5 Sonnet—in understanding and interpreting PFTs data.
strength_weakness: strength_weakness: •	Relevant references are less.
questions: questions: •	Tables I, II, III in the captions don’t match with Tables 1, 2, 3 in the discussion.