bluuebunny/crossref_metadata_2025_split huggingface dataset:
Can ChatGPT provide accurate financial toxicity resources to patients with cancer?
Authors: Patel Jharna Mehul , Miller Erin Leigh , Cantor Tal , Chino Fumiko , Thom Bridgette , Stasenko Marina , Aviki Emeline Mariam 
Published: May 2024

e13638 Background: Artificial Intelligence (AI) has potential benefits in medicine, including improving patient education, but its use in mitigating financial toxicity faced by patients with cancer is unknown. We aimed to investigate the accuracy of ChatGPT, an open access AI, responses to common financial concerns among patients with cancer. Methods: ChatGPT, an open access AI, was used to query 50 questions developed using oncologic society websites. These questions encompassed medical (n=25) and non-medical financial (n=25) inquiries related to breast, endometrial, lung, colon, and prostate cancers. The answers were scored by cancer financial toxicity experts with the scale: 1) correct and comprehensive, 2) correct but not comprehensive, 3) some correct, some incorrect components and 4) completely incorrect. Score discrepancies were resolved with a third reviewer. In secondary analysis, all questions were queried 3 times to assess if ChatGPT could provide more comprehensive answers on subsequent inquiries. The proportion of responses earning each score were calculated overall and within each question category and disease site. Results: Overall, in the first queries, 84% of questions were scored as correct but not comprehensive (n=42/50) and 6% (n=3/50) of questions were scored as having both correct and incorrect components. Approximately 28% (n=14/50) of questions had discrepant scores that required third party review, and 10% (n=5/50) of questions never achieved a consensus. All non-medical questions were scored as correct but not comprehensive (n=25/25) with a concordance rate of 96% (n=24/25), with only 1 question sent to a third reviewer. Additional queries of non-medical questions did not yield different answers. For the medical questions, approximately 68% (n=17/25) of questions were scored as correct but not comprehensive and 12% (n=3/25) were scored as having correct and incorrect components. 52% (n=13/25) of questions were sent to a third reviewer, and 20% (n=5/25) of questions never achieved a consensus score. While additional queries of medical questions yielded different answers, the quality of the responses did not improve. For each disease site, prostate cancer questions had the highest number of discordant questions with 40% (n=4/10) that were sent to a third reviewer for scoring but a 100% (n=10/10) of responses had a score that showed correct but not comprehensive answers. Reviewer feedback was universal that ChatGPT responses did not provide helpful information to patients, even in the situations in which there was technically no inaccurate information. Conclusions: ChatGPT does not consistently provide accurate or helpful answers to financial toxicity concerns commonly asked by patients with cancer. Additional input from experts is necessary to aid in AI learning before this resource should be used by patients facing financial toxicity.
Auto-machine learning for opportunistic thyroid nodule detection in lung cancer screening chest CT.
Authors: Patel Sumir , Kocaman Veysel , Sayici Mehmet Burak , Patel Nikhil 
Published: May 2024

e13639 Background: Automated Machine Learning (Auto-ML) in medical imaging is a process that allows non-experts to utilize machine learning techniques, opening the door for non-coder physician-driven exploitation of the technology. Auto-ML was applied for opportunistic detection of thyroid nodules in the context of low-dose lung cancer screening chest CT, facilitated by an innovative platform integration. By leveraging scans originally intended for lung cancer screening, suspicious appearing asymptomatic thyroid nodules can also be screened for where technically feasible. Methods: CT scans from the National Lung Screening Trial (NLST) dataset were utilized. This dataset contains low-dose chest CT examinations initially used for lung cancer screening. Annotation was carried out within a no-code web-based platform, Gesund.ai. A board-certified diagnostic radiologist annotated 100 CT examinations. A case was labeled "suspicious" if it presented a thyroid nodule of 1 cm or greater in diameter, while cases with no such nodules were deemed "not suspicious." The Auto-ML feature was then engaged within the software platform for parameter adjustments to refine the model without the need for manual coding. This phase was characterized by an iterative exploration of model performance, facilitated by the platform’s robust validation tools. Results: Upon training and validation, the Auto-ML model, cultivated from 100 annotated cases and assessed against a separate set of 130 cases, demonstrated a discernment accuracy of 0.51 in identifying "suspicious" thyroid nodules with a sensitivity and precision of 0.51 and 0.62 respectively. The AUC for this class stands at 0.69, indicating a moderate ability to distinguish between the presence and absence of thyroid nodules. This metric, while foundational, offers insight into the model's potential efficacy in real-world diagnostic scenarios, reflecting the initial capabilities of the platform's Auto-ML integration in enhancing diagnostic processes. Conclusions: The findings from this study highlight the potential of Auto-ML in revolutionizing opportunistic screening for thyroid nodules via lung cancer CT. The initial accuracy rate underscores the necessity for further refinement and validation of the model, but also demonstrates the promising potential of Auto-ML. Institutional Auto-ML would include extraction of appropriate studies through the no-code software platform integrated directly within the PACS framework. This process would ensure data immobility and foster a secure environment for dataset creation. Annotation would be carried out by multiple physician imagers within the same facility, enhancing the data’s accuracy and comprehensiveness specific for the institution’s patient population. This exploration highlights the strategic role of Auto-ML in advancing cancer patient care through innovative technological integration.
Dynamic machine learning model to forecast patient availability for clinical trials.
Authors: Kulkarni Rajeev , Vaidya Vivek Prabhakar , Parmar Dhaval , Tibrewal Abhishek , Parikh Ravi Bharat 
Published: May 2024

e13641 Background: Patient eligibility is a routine concept in clinical research, comprised of simple or complex set of inclusion or exclusion criteria applied in a specific sequence. Our methodology introduces the concept of ‘availability’ as a narrower term that is closer to the actual number that will be considered for a specific study and who may decide to participate. This specific research develops Machine Learning (ML) model that dynamically predicts likelihood of Multiple Myeloma progression and departure from targeted outcomes to sustain therapy and to indicate potential availability for clinical trials – to reduce screening efforts and improve screening precision of clinical research organizations. Methods: The cohort for training these models was drawn from the ConcertAI Oncology Research database, further enriched by key variables derived from unstructured data sources. Eligible patients for the analysis had a diagnosis of Multiple Myeloma made after Jan 1, 2020. Progressions were labelled using International Myeloma Working Group Criteria. Patients that did not satisfy the criteria were censored. Labels were assigned to patients after initial diagnosis of MM, and the duration between diagnosis to label was computed to build time to event (TTE) models. Over 2000 features were created to build this model, e.g., laboratory values, vitals, biomarkers, medications, ECOG, stage, etc. Variable creation preserved temporality of lab variables and vitals and noted presence of other variables. Several ML methods were employed to build the TTE models, with gradient boosting resulting in the highest c-index (0.8) based on 12 clinically relevant features. A temporal validation of the model was performed on a simulated index date. Results: Cohort used for training the model comprised of N = 4315 patients with average age at diagnosis of 72.3 years, and ~11% of the patients labelled as progressing. Temporal validation was performed on N = 1874 patients, with ~9% progression rate. The area under the curve (AUC) and weighted F1 score for this cohort was 87% and 89% respectively. Conclusions: Patient availability assessments are complex, not supported effectively by existing solutions and often require 100’s of hours from highly trained clinical personnel. An AI model predicting Multiple Myeloma progression built using a large-scale real-world dataset demonstrated 2-4x improvement in forecasting patient availability over other approaches. Integrating these solutions into clinical workflows can improve the effectiveness of research teams; enable more patients to have clinical trials as viable options at the right time; and allow care teams to have forward visibility to choices of standard-of-care and clinical trial alternatives in the best interest of patient outcomes. [Table: see text]
Results of the COMPARE-GPT study: Comparison of medication package inserts and GPT-4 cancer drug information.
Authors: Hundal Jasmin , Teplinsky Eleonora 
Published: May 2024

e13646 Background: The integration of artificial intelligence (AI) technologies has opened new avenues for information dissemination. GPT-4 (GPT) is a large language model developed by open AI. Concerns about its accuracy exist, especially in health communications. We conducted a study to evaluate the accuracy of GPT responses to questions about anti-neoplastic agents for solid tumor malignancies. Methods: We evaluated GPT responses to four questions (Table 1) for each solid tumor drug newly approved or approved for a new indication between 2020 and 2022. The responses from GPT were saved and then compared to the information provided in the medication package insert. For discordant results, a second search was run using a new GPT session. The study was performed by two physicians who ensured that the comparison was both rigorous and unbiased. Results: 53 unique antineoplastic agents were included. GPT responses regarding FDA approval & mechanism of action were correct for each drug (100%). When asked about common adverse reactions, GPT provided incorrect responses for 47% (25/53) of the medications and correct responses for 53% (28/43). The inaccurate responses were either missing certain side effects (88%), provided inaccurate incidence rates (8%), and/or included side effects not listed on the package insert (20%). When a second search was conducted with the same question, GPT provided different responses in 76% (19/25), same responses in 16% (4/25) and now correct responses in 8% (2/25). When asked about drug warnings and precautions, GPT provided incorrect responses for 68% (36/53) of the medications and correct responses for 32% (17/53). The inaccurate responses were missing certain warnings and precautions (89%) and/or included warnings and precautions not listed on the package insert (25%). When a second search was conducted with the same question, GPT provided different responses in 53% (19/36), same responses in 39% (14/36) and now correct responses in 8% (3/36). Conclusions: Analysis of solid tumor drugs approved for use in 2020-2022 evaluated GPT's ability to provide drug information. GPT accurately identified FDA approved indications & mechanisms of action but had significant limitations in accurately reporting a comprehensive list of adverse reactions, drug warnings and precautions. GPT occasionally included drug side effects that are not listed in the package insert. Significant variability in output was demonstrated in repeated searches. Given the inconsistencies in the data identified, GPT should not be used as a primary source of medical information and may potentially cause harm. [Table: see text]
Technology enhanced multidisciplinary cancer conferences in gynecologic oncology.
Authors: Agyemang Amma , Ibekwe Opuruiche , Gaudioso Carmelo , Attwood Kristopher , Werner Sarah , Nwogu Chukwumere E. , Frederick Peter Jonathan 
Published: May 2024

e13652 Background: Multidisciplinary Cancer Conferences (MCCs) improve quality of care and patient outcomes. The goals of our study were to investigate the impact of an information technology platform, Navify Tumor Board Solution (NTBS) on the (1) efficiency of preparing and conducting MCCs and (2) educational quality of the gynecologic oncology MCCs. Methods: We conducted a prospective, mixed methods study involving participants of the biweekly gynecologic oncology MCC at a comprehensive cancer center. NTBS was introduced to the gynecologic oncology MCC for the preparation and presentation of tumor board cases. Data were collected before and after the introduction of NTBS. We used a validated observational tool to assess the impact of the platform on the quality of case presentation and discussion, as well as decision making (as measured by a composite mean score). We also evaluated compliance of care plans with national guidelines, changes in care plans, and concordance of actual treatment received with MCC recommendations. The survey consisted of five qualitative and two demographic questions. Interviews lasting 10 minutes were recorded using Microsoft Teams. Analyses were conducted using SAS v9.4 (Cary, NC) and Dedoose v9.0.17, (Los Angeles, CA). Results: Pre- and Post-NTBS cohorts consisted of 49 and 60 patients, respectively. When comparing pre- versus post-NTBS cohorts, there was a difference in the mean composite score (50.57 vs 54.11; p=0.016). While there was no statistical difference between cohorts, patient care plans were changed in 13% of the cases and were 92% concordant with treatment received. Additionally, MCC recommendations were 100% compliant with national guidelines. Surveys of 13 MCC participants revealed that post-NTBS, the presentation format was more cohesive, concise, and structured. Common themes from gynecologic oncology fellows’ responses included improvement of case preparation time and simplified process while using NTBS. However, trainees reported limited use of NTBS features, difficulty incorporating graphical, and lack of a comprehensive list of NCCN guidelines. All respondents had educational goals for their participation and reported a high educational value of the MCC. These goals included knowledge acquisition of clinical diagnosis, management of standard of care versus complex situations, sentinel clinical trials underlying NCCN treatment guidelines, primary literature review, management of preparation time and flow of discussion, and preparation for oral boards. Conclusions: NTBS contributed positively to the gynecologic oncology MCC and clinical care for patients. Overall, NTBS had a positive impact on case formatting, preparation time and quality of trainee education. Comprehensive user training in the NTBS features could further enhance its utilization within the MCC.
Comparative analysis of ChatGPT-4.0’s management of six gastrointestinal cancers according to the NCCN guidelines.
Authors: Bresler Tamir E. , Wilson Tyler , Htway Zin , Slomowitz Samuel , Fujita Manabu 
Published: May 2024

e13654 Background: The integration of Natural Language Processing (NLP) into healthcare holds tremendous promise. ChatGPT-4.0 (OpenAI, San Francisco, CA) is a widely recognized large language model that can comprehend and generate text, answer questions, and perform other language-related tasks. However, pitfalls and errors have been described in its clinical application. We explored the ability of ChatGPT-4.0 (ChatGPT) to guide clinical decision-making in 6 gastrointestinal cancers using the National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines as a framework. Methods: We reviewed the NCCN Guidelines for Anal Squamous Cell Carcinoma (AN), Small Bowel Adenocarcinoma (SB), Ampullary Adenocarcinoma (AA), Biliary Tract Cancers (BT), Pancreatic Adenocarcinoma (PN), and Gastric Cancer (GA). Up to 2 clinical questions were designed for each decision-making page. Questions were categorized by type ( Wo rkup, Treatment, Surveillance, Diagnostics, or References). ChatGPT was queried in a reproducible fashion. To account for variable prompt engineering of our non-validated assessment tool, up to 3 follow-up questions were allowed. Responses were rated on a Likert scale: 5) Correct; 4) Correct, with missing information requiring clarification; 3) Correct, but unable to complete answer; 2) Partially incorrect; 1) Absolutely incorrect. Subgroup analysis was conducted on Correctness (defined as scores 1-2 vs 3-5) and Accuracy (scores 1-3 vs 4-5). Variance between ChatGPT responses to each cancer was analyzed. Descriptive statistics were used, and significance was tested with binary logistic regression. Results: A total of 270 questions were generated (range per cancer 32-68). The score frequency distribution was: 5) 45.2%; 4) 19.3%; 3) 13.3%; 2) 13.7%; and 1) 8.5%. On subgroup analysis, Correctness was seen in 210 (77.8%) of questions, and Accuracy with 174 (64.4%). The difference in Correctness scores between cancers was not statistically significant, and there was no statistically significant difference in scores by question type. There was a statistically significant difference in the Accuracy of ChatGPT between cancers (Table). Conclusions: ChatGPT was significantly more likely to provide accurate responses to questions regarding GA and PN versus AN or SB. It demonstrates a limited capacity to assist with complex clinical decision-making in 6 gastrointestinal cancers. However, the Accuracy level is below the acceptable threshold for implementation into clinical use. Further analysis of the expanding capabilities of ChatGPT and other NLP-based tools is warranted in this rapidly evolving domain. Future studies would benefit from a validating grading instrument. [Table: see text]
Implementing large language model-based artificial intelligence (AI) technology in proposing effective treatment plans in patients with cancer.
Authors: Santoki Aditya , Jones Rachel , Peter Manas , Mathew Aju 
Published: May 2024

e13660 Background: Language Learning Models (LLM), are programs that can respond to queries without previous exposure to the input. This is because the programs are actually generative mathematical models trained on hundreds of terabytes of textual data. The output is thus a statistical assessment of the corpus of human generated text likely indicating the correct answer. ChatGPT is an example of an LLM that can understand questions and synthesize responses based on probability. However Chat GPT’s potential to accurately assist physicians with a diagnosis, remains yet to be determined. The goal of this study is to elucidate AI’s potential in proposing both feasible and accurate treatment plans specifically for patients with cancer. Methods: Fifty patients at a medical oncology practice were recruited for this study. For each patient, the physician’s shorthand notes were recorded and organized. ChatGPT was then asked to come up with a potential treatment plan using these notes according to a predetermined set workflow for each individual. The responses were then reviewed by a medical oncologist for accuracy and feasibility via a qualtrics survey. Results: ChatGPT correctly proposed a treatment plan in agreement with the physician for a total of 18 cancer patients allowing for the agreement rate to be 36%. For the rest of the 32 patients, the physician disagreed with the proposed ChatGPT treatment plan on the basis of lack of personalization, inhibitive cost, incorrect recommendation, and greater alternative options. Thirteen patients fell into the ‘Lack of Personalization in Treatment' category, twelve in the ‘Incorrect Recommendation’ category, four in ‘Inhibitive Cost' category, and three in the ‘Greater Alternative Options’ category. Most patients however fell in the category of personalization, suggesting ChatGPT’s unfamiliarity with the subject of generating personal treatment plans. Conclusions: In the current study, ChatGPT was expected to create a treatment plan with only the clinician’s abridged shorthand notes and without any familiarity with the physician’s personal preferences for treatment. This knowledge gap can account for an agreement rate less than 50%. Despite its flaws, ChatGPT was, however, able to generate an adjusted and corrected version of the treatment plan when prompted further outside the set workflow. This highlights the invaluable role of clinical experience in synthesizing an accurate treatment plan. With its encroachment into the medical field, ChatGPT has shown potential in proposing feasible and accurate treatment plans, especially in the oncology clinical setting. Despite the low agreement rate obtained from this study, ChatGPT has shown promise in developing an agreeable treatment plan with the oversight of the attending physician.
Real-world implementation of an EHR-integrated, SMS text remote patient monitoring platform at an academic cancer center.
Authors: Shah Sumit , Kadurugamuwa Gayani , Holmes Tisch Alison , Lu Monika , Modarresi Shiva , Johnston Caleb , Colbert James 
Published: May 2024

e13669 Background: Remote monitoring studies in oncology have shown improved quality of life, decreased healthcare utilization, and even improved overall survival in an oncologic population, however applications in a real-world setting have been limited. Barriers to successful deployment include low patient engagement rates with smartphone apps, lack of integration with the EHR and burdensome clinical workflows for staff. Methods: We sought to assess the feasibility of an EHR-integrated, SMS text conversational AI agent to automate the collection of ePRO data as well as the assessment and triage of common patient symptoms for patients with GI and GU tumors receiving anti-cancer therapy. This program was developed collaboratively by a health system digital health team, nursing informatics, IT support resources, oncology nurses, medical oncologists, and representatives from Memora Health. Core components included 1) weekly ePRO collection based on the NIH’s CTCAE, 2) NLP activated symptom triage for 16 common symptoms and toxicities 3) weekly symptom check-in 4) weekly medication adherence assessment for patients on oral therapy. Patients received all messages through SMS on their phones, and all data collected were routed into the EHR. In-EHR visualizations allowed nurse coordinators to track ePRO symptoms over time and symptoms that met pre-specified criteria triggered EHR-based alerts. Results: Of 99 patients ordered into this SMS program, 83 enrolled (84% enrollment rate), with 55 Uro Onc patients; 44 GI Onc patients. A total of 2,583 surveys were sent to patients. 98% of patients completed at least 1 survey. Overall patient survey completion rate was 62%. Medication adherence surveys had the highest completion rate at 81%. ePRO completion rates were higher for GU (65%) vs GI (39%) patients (p<0.01). 71% of ePRO surveys generated a symptom alert to the nursing staff. Patient satisfaction was high with 93% rating this SMS program as “excellent” or “good” (n=78). Care team surveys found that 3 of the 6 Oncologists believe ePRO collection is valuable and 3 of 6 used ePRO data to adjust a care plan. Nurse coordinators had a mixed view of the program with a majority reporting that there were too many symptom alerts. Conclusions: We successfully deployed an EHR-integrated SMS program to collect ePRO data, track patient symptoms, and monitor adherence to oral therapy. We believe having a physician champion influenced patient engagement in the GU patient population. Patients had a favorable impression of this program and survey completion rates demonstrated ease of use for patients. Clinical staff, particularly nurses, felt that there were too many alerts. Additional workflow changes may reduce alert burden and better incorporate ePRO learnings into clinical practice.