bluuebunny/crossref_metadata_2025 huggingface dataset:
Retrieval-augmented large language models for clinical trial screening.
Authors: Tan Ryan , Ho Si Xian , Oo Shiyun Vivianna Fequira , Chua Shi Ling , Zaw Ma Wai Wai , Tan Daniel Shao-Weng
Published: May 2024

e13611

Background: Clinical trial screening is currently manual and laborious. We tested several large language models enhanced with retrieval-augmented generation (RAG-LLM) to assess their performance on this task. Methods: We extracted eligibility criteria of 184 oncology trials with FDA approval notifications between 8 January 2020 and 18 January 2024, as well as information on cancer staging and performance status scoring for the RAG-LLM vector database. A medical oncologist and 2 senior clinical trial coordinators developed a test set of 975 synthetic patient profiles which included primary site, stage, prior therapy, tumor mutations and one additional clinical feature. Each profile was paired with one of the 184 trials and annotated for ground-truth eligibility and the reason for it. This process was repeated for another validation set of 240 longer and more challenging profiles paired with 8 ongoing trials. We developed RAG-LLMs with 4 leading LLMs (Zephyr-7B, Med42, GPT 3.5, GPT4) and evaluated their accuracy in determining trial eligibility as well as retrieval-augmented generation assessment (RAGAs) metrics. A response was deemed accurate only if it correctly assigned both trial eligibility and the reason for assignment. Results: GPT4 performed best and achieved an accuracy of 95.18% and 80.00% on the test and validation set respectively with a mean inference time of 10.95 seconds. It also demonstrated the highest answer relevancy, context relevancy and faithfulness (Table). Conclusions: Our results demonstrate potential for RAG-LLMs to assist with trial screening at scale. Further evaluation in real-world cohorts utilizing electronic records and full protocol data with tracking of impact on trial enrolment can be explored within secure firewalls. [Table: see text]
Clustering analysis for predicting racial disparities in genitourinary cancer–related financial burden.
Authors: Zhang Yanjia , Khosla Atulya Aman , Chhabra Sufal , Batra Nitya , Ganiyani Mohammad Arfat , Goyal Anmol , Bellur Shreyas S , Singh Rohit , Jatwani Karan , Ramamoorthy Venkataraghavan , Rubens Muni , Saxena Anshul , Garje Rohan
Published: May 2024

e13612

Background: Healthcare costs in the United States (US) are significantly impacted by genitourinary (GU) cancers. Prostate cancer treatment alone cost the US more than $22 billion in 2020. We sought to utilize machine learning (ML)-based classification to investigate differences in the total cost of healthcare expenditure among patients with GU cancer in the US. Methods: The study included eligible adult GU cases from the 2019–2021 Medical Expenditure Panel Survey. Renal cell, prostate, urothelial, testicular, and penile cancers were identified using ICD-10 CM codes. Patients were clustered using demographic information such as age, race, sex, marital status, highest degree, insurance status, perceived health status, and perceived mental health status. The analysis accounted for the complex survey design and sampling weights. The cluster groups were taken into account when conducting descriptive and regression analyses. GU cancer cases were divided into low-, medium-, and high-risk healthcare expenditure clusters. Results: The study included a representative sample of 9,061,181 cases of GU cancer in the US between 2019 and 2021. The average age of the sample was 71.6 years (95% CI: 70.7, 72.5). There were 7.9% Hispanic (Hisp), 74.8% non-Hispanic White (NHW), 13.8% non-Hispanic Black (NHB), and 3.5% individuals from Asian or other races. Overall, the mean total healthcare expenditures within low-, med-, and high-risk clusters were $17974 (13816, 22133), $20496 (13100, 27893), and $30512 (21389, 39635), respectively. Regression analysis showed that, after adjusting for sampling weight, cluster, gender, family income, age, insurance status, and survey year, the mean total expenditure was higher by $11209.70 and $13100.81 in med- and high-risk clusters when compared to the low-risk cluster (ref.). Within each race or ethnicity, the proportion of Hisp or NHB individuals was higher in med- and high-risk clusters. Med- and high-risk clusters were more likely to contain people with cognitive impairments, poor health status, and financial difficulties in their families. Conclusions: We used an unsupervised clustering algorithm that can distinguish between GU cases with high and low expenditure based solely on sociodemographic variables. We demonstrate that our findings were consistent for all three data cohorts. This approach can predict the disparity in utilization in any insured population. [Table: see text]
Evaluating generalizability of landmark randomized controlled trials in common metastatic cancers using machine learning-based emulated trials.
Authors: Orcutt Xavier , Long Qi , Mamtani Ronac , Parikh Ravi Bharat
Published: May 2024

e13618

Background: Randomized controlled trial (RCT) results often lack generalizability to real-world oncology patients (pts). We present TrialTranslator, a machine learning (ML) framework to evaluate the generalizability of Phase III oncology RCTs by emulating trials across prognostic risk groups. Methods: TrialTranslator was applied to 11 landmark RCTs encompassing the most prevalent advanced cancers – non-small cell lung (aNSCLC), breast (mBC), prostate (mPC), and colorectal (mCRC). Flatiron Health’s EHR-derived de-identified database was used for model development and trial emulation. In Stage I, ML models (gradient boosted [GBM], random forest, support vector machine and penalized Cox) were trained to predict survival from time of metastatic diagnosis. Models used 130 longitudinal demographic, cancer-related, lab, and biomarker features. In Stage II, trial emulation, pts meeting RCT cancer and biomarker eligibility criteria were stratified into low, medium, and high-risk groups based on the top-performing ML model. We used inverse probability of treatment weighted survival analysis for each risk group to compare the median overall survival (mOS) or median progression free survival (mPFS) between experimental and control arms. Results: GBM models achieved the highest time-dependent area under the receiver operating curve (AUC) in the test set in all cancer types. For example, the 1-year AUC for the aNSCLC GBM vs the Lung Cancer Prognostic Index (validated Cox model) was 0.783 vs 0.689. ML-predicted low-risk pts typically had similar mOS or mPFS as RCT populations (Table). In emulated trials for low-risk pts, real-world survival benefit – the absolute difference in mOS or mPFS between experimental and control arms – was similar to RCT results. In contrast, for high-risk pts, survival benefits were not statistically or clinically significant. For example, in the FLAURA trial, RCT mPFS for experimental arm (osimertinib) vs control (gefitinib or erlotinib) was 18.9 mos vs 10.2 months; mPFS was 17.1 vs 10.3 months for low-risk pts and 5.1 vs 4.2 months for high-risk pts. Conclusions: ML frameworks like TrialTranslator may facilitate individual patient-level estimation of real-world treatment benefits of novel oncology agents. TrialTranslator is accessible at trialtranslator.com. [Table: see text]
AI-generated synthetic clinical-genomic data for precision oncology research: Validation using a case study on lung adenocarcinoma.
Authors: Theodorou Brandon , Das Amar , Eagle Craig , Sun Jimeng , Hardin Aaron , Zhang Nicole , Yay Donderici Elifnur
Published: May 2024

e13627

Background: The analysis of genomic variants is crucial in precision oncology research, offering insights into cancer risks and progression, especially in diverse types such as lung adenocarcinoma (LUAD). However, such research often grapples with balancing patient privacy with the need for comprehensive, high-quality genomic datasets. Our project addresses this by creating synthetic clinical-genomic data, which maintains patient confidentiality and provides a rich resource for genomic cancer research. Methods: Leveraging the GuardantINFORM database, which includes anonymized genomic data and structured payer claims, we focused on generating synthetic data for LUAD patient cohorts. This approach involves processing real patient data into a format compatible with Medisyn’s generative AI models, ensuring the synthetic data retains the original's statistical properties, and processing the output back into the original database structure and format. This method plays a crucial role in maintaining patient privacy and serves as a valuable tool for research by enabling the generation of realistic patients with desired properties on demand. Results: Our synthetic data closely mirrors real-world genomic and claims variable distributions, evidenced by a 0.994 R2 correlation between real and synthetic data along with comparable Oncoprints. Importantly, privacy tests show that patient confidentiality is effectively maintained despite this effective performance. The synthetic data's utility was then demonstrated in a study replicating real-world findings: LUAD patients with KRAS G12C in combination with STK11 mutations showed a significantly higher risk of early mortality. This underscores the potential of synthetic data in advancing cancer research. Conclusions: This research offers a promising avenue for the cancer research community. By providing a method to share privatized, synthetic genomic data, which can be combined and generated on demand, we enable broader, more responsible data sharing. This approach protects patient privacy and offers a rich dataset for groundbreaking research, potentially accelerating advances in cancer diagnosis and treatment. [Table: see text]
Assessing ChatGPT's potential as a clinical resource for medical oncologists: An evaluation with board-style questions and real-world patient cases.
Authors: Lee Hyun , Hamedi Zahra , Oliver David , Lin Hui , Athar Uzma , Vohra Hiba Zainab , Daniels Peter , Kothadia Sejal
Published: May 2024

e13628

Background: ChatGPT is a conversational artificial intelligence (AI) model that learns from massive text-based datasets and then responds to user input, which often involves completing tasks or answering questions. Recent studies showed ChatGPT’s success in passing multiple specialty medical licensing and board examinations, showcasing its promising capabilities in the medical domain. Here, we investigated ChatGPT's potential as a swift and reliable information source for medical oncologists using board examination style questions and real patient cases. Methods: We randomly selected 121 board-style questions from the American Society of Clinical Oncology Self-Evaluation Program (ASCO SEP). The questions were entered into ChatGPT in both multiple-choice (MC) and open-ended (OE) prompts. ChatGPT’s answers and explanations were evaluated for accuracy and concordance. Non-inferiority analysis was performed with power of 80% at α = 0.05 and non-inferiority margin set at 70% correct answers given the historical board exam pass rate of about 65% correct answers. For subgroup analysis, the questions were categorized by tested competency and primary tumor pathology. ChatGPT was also given 10 questions derived from real patient cases. We compared its responses to the answers provided by experienced oncologists to determine accuracy and practical applicability. Results: ChatGPT answered 75 (62.0%) MC queries correctly. Among the correctly answered queries, 2 responses contained faulty explanations. Such inaccurate or discordant explanations were found in 26 of the 46 incorrectly answered queries. In OE prompts, ChatGPT answered 53 (43.8%) questions correctly with correct explanations for all. Of the 68 incorrect responses, 32 of them contained inaccurate or discordant explanations. Subgroup analysis suggested varying performance across the categories. The best performance was seen with malignant hematology (81.8% of MC and 72.8% of OE prompts answered correctly) while the weakest performance was seen with genitourinary malignancies (60% of MC and 20% of OE prompts answered correctly). As for the real-world patient case questions, responses from ChatGPT and the clinicians were concordant in 5 questions. None of the discordant responses contained inaccurate information while 80% of the concordant responses contained sufficient details to assist with patient management decisions. Conclusions: ChatGPT's performance fell short of the non-inferiority margin, highlighting the challenges with incorporating AI in the rapidly evolving field of medical oncology. Despite the limitations, ChatGPT’s partial success, in both board-style and real-world patient care questions, affirms its potential for clinical utility in future.
Can ChatGPT provide accurate financial toxicity resources to patients with cancer?
Authors: Patel Jharna Mehul , Miller Erin Leigh , Cantor Tal , Chino Fumiko , Thom Bridgette , Stasenko Marina , Aviki Emeline Mariam
Published: May 2024

e13638

Background: Artificial Intelligence (AI) has potential benefits in medicine, including improving patient education, but its use in mitigating financial toxicity faced by patients with cancer is unknown. We aimed to investigate the accuracy of ChatGPT, an open access AI, responses to common financial concerns among patients with cancer. Methods: ChatGPT, an open access AI, was used to query 50 questions developed using oncologic society websites. These questions encompassed medical (n=25) and non-medical financial (n=25) inquiries related to breast, endometrial, lung, colon, and prostate cancers. The answers were scored by cancer financial toxicity experts with the scale: 1) correct and comprehensive, 2) correct but not comprehensive, 3) some correct, some incorrect components and 4) completely incorrect. Score discrepancies were resolved with a third reviewer. In secondary analysis, all questions were queried 3 times to assess if ChatGPT could provide more comprehensive answers on subsequent inquiries. The proportion of responses earning each score were calculated overall and within each question category and disease site. Results: Overall, in the first queries, 84% of questions were scored as correct but not comprehensive (n=42/50) and 6% (n=3/50) of questions were scored as having both correct and incorrect components. Approximately 28% (n=14/50) of questions had discrepant scores that required third party review, and 10% (n=5/50) of questions never achieved a consensus. All non-medical questions were scored as correct but not comprehensive (n=25/25) with a concordance rate of 96% (n=24/25), with only 1 question sent to a third reviewer. Additional queries of non-medical questions did not yield different answers. For the medical questions, approximately 68% (n=17/25) of questions were scored as correct but not comprehensive and 12% (n=3/25) were scored as having correct and incorrect components. 52% (n=13/25) of questions were sent to a third reviewer, and 20% (n=5/25) of questions never achieved a consensus score. While additional queries of medical questions yielded different answers, the quality of the responses did not improve. For each disease site, prostate cancer questions had the highest number of discordant questions with 40% (n=4/10) that were sent to a third reviewer for scoring but a 100% (n=10/10) of responses had a score that showed correct but not comprehensive answers. Reviewer feedback was universal that ChatGPT responses did not provide helpful information to patients, even in the situations in which there was technically no inaccurate information. Conclusions: ChatGPT does not consistently provide accurate or helpful answers to financial toxicity concerns commonly asked by patients with cancer. Additional input from experts is necessary to aid in AI learning before this resource should be used by patients facing financial toxicity.
Auto-machine learning for opportunistic thyroid nodule detection in lung cancer screening chest CT.
Authors: Patel Sumir , Kocaman Veysel , Sayici Mehmet Burak , Patel Nikhil
Published: May 2024

e13639

Background: Automated Machine Learning (Auto-ML) in medical imaging is a process that allows non-experts to utilize machine learning techniques, opening the door for non-coder physician-driven exploitation of the technology. Auto-ML was applied for opportunistic detection of thyroid nodules in the context of low-dose lung cancer screening chest CT, facilitated by an innovative platform integration. By leveraging scans originally intended for lung cancer screening, suspicious appearing asymptomatic thyroid nodules can also be screened for where technically feasible. Methods: CT scans from the National Lung Screening Trial (NLST) dataset were utilized. This dataset contains low-dose chest CT examinations initially used for lung cancer screening. Annotation was carried out within a no-code web-based platform, Gesund.ai. A board-certified diagnostic radiologist annotated 100 CT examinations. A case was labeled "suspicious" if it presented a thyroid nodule of 1 cm or greater in diameter, while cases with no such nodules were deemed "not suspicious." The Auto-ML feature was then engaged within the software platform for parameter adjustments to refine the model without the need for manual coding. This phase was characterized by an iterative exploration of model performance, facilitated by the platform’s robust validation tools. Results: Upon training and validation, the Auto-ML model, cultivated from 100 annotated cases and assessed against a separate set of 130 cases, demonstrated a discernment accuracy of 0.51 in identifying "suspicious" thyroid nodules with a sensitivity and precision of 0.51 and 0.62 respectively. The AUC for this class stands at 0.69, indicating a moderate ability to distinguish between the presence and absence of thyroid nodules. This metric, while foundational, offers insight into the model's potential efficacy in real-world diagnostic scenarios, reflecting the initial capabilities of the platform's Auto-ML integration in enhancing diagnostic processes. Conclusions: The findings from this study highlight the potential of Auto-ML in revolutionizing opportunistic screening for thyroid nodules via lung cancer CT. The initial accuracy rate underscores the necessity for further refinement and validation of the model, but also demonstrates the promising potential of Auto-ML. Institutional Auto-ML would include extraction of appropriate studies through the no-code software platform integrated directly within the PACS framework. This process would ensure data immobility and foster a secure environment for dataset creation. Annotation would be carried out by multiple physician imagers within the same facility, enhancing the data’s accuracy and comprehensiveness specific for the institution’s patient population. This exploration highlights the strategic role of Auto-ML in advancing cancer patient care through innovative technological integration.
Dynamic machine learning model to forecast patient availability for clinical trials.
Authors: Kulkarni Rajeev , Vaidya Vivek Prabhakar , Parmar Dhaval , Tibrewal Abhishek , Parikh Ravi Bharat
Published: May 2024

e13641

Background: Patient eligibility is a routine concept in clinical research, comprised of simple or complex set of inclusion or exclusion criteria applied in a specific sequence. Our methodology introduces the concept of ‘availability’ as a narrower term that is closer to the actual number that will be considered for a specific study and who may decide to participate. This specific research develops Machine Learning (ML) model that dynamically predicts likelihood of Multiple Myeloma progression and departure from targeted outcomes to sustain therapy and to indicate potential availability for clinical trials – to reduce screening efforts and improve screening precision of clinical research organizations. Methods: The cohort for training these models was drawn from the ConcertAI Oncology Research database, further enriched by key variables derived from unstructured data sources. Eligible patients for the analysis had a diagnosis of Multiple Myeloma made after Jan 1, 2020. Progressions were labelled using International Myeloma Working Group Criteria. Patients that did not satisfy the criteria were censored. Labels were assigned to patients after initial diagnosis of MM, and the duration between diagnosis to label was computed to build time to event (TTE) models. Over 2000 features were created to build this model, e.g., laboratory values, vitals, biomarkers, medications, ECOG, stage, etc. Variable creation preserved temporality of lab variables and vitals and noted presence of other variables. Several ML methods were employed to build the TTE models, with gradient boosting resulting in the highest c-index (0.8) based on 12 clinically relevant features. A temporal validation of the model was performed on a simulated index date. Results: Cohort used for training the model comprised of N = 4315 patients with average age at diagnosis of 72.3 years, and ~11% of the patients labelled as progressing. Temporal validation was performed on N = 1874 patients, with ~9% progression rate. The area under the curve (AUC) and weighted F1 score for this cohort was 87% and 89% respectively. Conclusions: Patient availability assessments are complex, not supported effectively by existing solutions and often require 100’s of hours from highly trained clinical personnel. An AI model predicting Multiple Myeloma progression built using a large-scale real-world dataset demonstrated 2-4x improvement in forecasting patient availability over other approaches. Integrating these solutions into clinical workflows can improve the effectiveness of research teams; enable more patients to have clinical trials as viable options at the right time; and allow care teams to have forward visibility to choices of standard-of-care and clinical trial alternatives in the best interest of patient outcomes. [Table: see text]
Results of the COMPARE-GPT study: Comparison of medication package inserts and GPT-4 cancer drug information.
Authors: Hundal Jasmin , Teplinsky Eleonora
Published: May 2024

e13646

Background: The integration of artificial intelligence (AI) technologies has opened new avenues for information dissemination. GPT-4 (GPT) is a large language model developed by open AI. Concerns about its accuracy exist, especially in health communications. We conducted a study to evaluate the accuracy of GPT responses to questions about anti-neoplastic agents for solid tumor malignancies. Methods: We evaluated GPT responses to four questions (Table 1) for each solid tumor drug newly approved or approved for a new indication between 2020 and 2022. The responses from GPT were saved and then compared to the information provided in the medication package insert. For discordant results, a second search was run using a new GPT session. The study was performed by two physicians who ensured that the comparison was both rigorous and unbiased. Results: 53 unique antineoplastic agents were included. GPT responses regarding FDA approval & mechanism of action were correct for each drug (100%). When asked about common adverse reactions, GPT provided incorrect responses for 47% (25/53) of the medications and correct responses for 53% (28/43). The inaccurate responses were either missing certain side effects (88%), provided inaccurate incidence rates (8%), and/or included side effects not listed on the package insert (20%). When a second search was conducted with the same question, GPT provided different responses in 76% (19/25), same responses in 16% (4/25) and now correct responses in 8% (2/25). When asked about drug warnings and precautions, GPT provided incorrect responses for 68% (36/53) of the medications and correct responses for 32% (17/53). The inaccurate responses were missing certain warnings and precautions (89%) and/or included warnings and precautions not listed on the package insert (25%). When a second search was conducted with the same question, GPT provided different responses in 53% (19/36), same responses in 39% (14/36) and now correct responses in 8% (3/36). Conclusions: Analysis of solid tumor drugs approved for use in 2020-2022 evaluated GPT's ability to provide drug information. GPT accurately identified FDA approved indications & mechanisms of action but had significant limitations in accurately reporting a comprehensive list of adverse reactions, drug warnings and precautions. GPT occasionally included drug side effects that are not listed in the package insert. Significant variability in output was demonstrated in repeated searches. Given the inconsistencies in the data identified, GPT should not be used as a primary source of medical information and may potentially cause harm. [Table: see text]
Technology enhanced multidisciplinary cancer conferences in gynecologic oncology.
Authors: Agyemang Amma , Ibekwe Opuruiche , Gaudioso Carmelo , Attwood Kristopher , Werner Sarah , Nwogu Chukwumere E. , Frederick Peter Jonathan
Published: May 2024

e13652

Background: Multidisciplinary Cancer Conferences (MCCs) improve quality of care and patient outcomes. The goals of our study were to investigate the impact of an information technology platform, Navify Tumor Board Solution (NTBS) on the (1) efficiency of preparing and conducting MCCs and (2) educational quality of the gynecologic oncology MCCs. Methods: We conducted a prospective, mixed methods study involving participants of the biweekly gynecologic oncology MCC at a comprehensive cancer center. NTBS was introduced to the gynecologic oncology MCC for the preparation and presentation of tumor board cases. Data were collected before and after the introduction of NTBS. We used a validated observational tool to assess the impact of the platform on the quality of case presentation and discussion, as well as decision making (as measured by a composite mean score). We also evaluated compliance of care plans with national guidelines, changes in care plans, and concordance of actual treatment received with MCC recommendations. The survey consisted of five qualitative and two demographic questions. Interviews lasting 10 minutes were recorded using Microsoft Teams. Analyses were conducted using SAS v9.4 (Cary, NC) and Dedoose v9.0.17, (Los Angeles, CA). Results: Pre- and Post-NTBS cohorts consisted of 49 and 60 patients, respectively. When comparing pre- versus post-NTBS cohorts, there was a difference in the mean composite score (50.57 vs 54.11; p=0.016). While there was no statistical difference between cohorts, patient care plans were changed in 13% of the cases and were 92% concordant with treatment received. Additionally, MCC recommendations were 100% compliant with national guidelines. Surveys of 13 MCC participants revealed that post-NTBS, the presentation format was more cohesive, concise, and structured. Common themes from gynecologic oncology fellows’ responses included improvement of case preparation time and simplified process while using NTBS. However, trainees reported limited use of NTBS features, difficulty incorporating graphical, and lack of a comprehensive list of NCCN guidelines. All respondents had educational goals for their participation and reported a high educational value of the MCC. These goals included knowledge acquisition of clinical diagnosis, management of standard of care versus complex situations, sentinel clinical trials underlying NCCN treatment guidelines, primary literature review, management of preparation time and flow of discussion, and preparation for oral boards. Conclusions: NTBS contributed positively to the gynecologic oncology MCC and clinical care for patients. Overall, NTBS had a positive impact on case formatting, preparation time and quality of trainee education. Comprehensive user training in the NTBS features could further enhance its utilization within the MCC.
Comparative analysis of ChatGPT-4.0’s management of six gastrointestinal cancers according to the NCCN guidelines.
Authors: Bresler Tamir E. , Wilson Tyler , Htway Zin , Slomowitz Samuel , Fujita Manabu
Published: May 2024

e13654

Background: The integration of Natural Language Processing (NLP) into healthcare holds tremendous promise. ChatGPT-4.0 (OpenAI, San Francisco, CA) is a widely recognized large language model that can comprehend and generate text, answer questions, and perform other language-related tasks. However, pitfalls and errors have been described in its clinical application. We explored the ability of ChatGPT-4.0 (ChatGPT) to guide clinical decision-making in 6 gastrointestinal cancers using the National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines as a framework. Methods: We reviewed the NCCN Guidelines for Anal Squamous Cell Carcinoma (AN), Small Bowel Adenocarcinoma (SB), Ampullary Adenocarcinoma (AA), Biliary Tract Cancers (BT), Pancreatic Adenocarcinoma (PN), and Gastric Cancer (GA). Up to 2 clinical questions were designed for each decision-making page. Questions were categorized by type ( Wo rkup, Treatment, Surveillance, Diagnostics, or References). ChatGPT was queried in a reproducible fashion. To account for variable prompt engineering of our non-validated assessment tool, up to 3 follow-up questions were allowed. Responses were rated on a Likert scale: 5) Correct; 4) Correct, with missing information requiring clarification; 3) Correct, but unable to complete answer; 2) Partially incorrect; 1) Absolutely incorrect. Subgroup analysis was conducted on Correctness (defined as scores 1-2 vs 3-5) and Accuracy (scores 1-3 vs 4-5). Variance between ChatGPT responses to each cancer was analyzed. Descriptive statistics were used, and significance was tested with binary logistic regression. Results: A total of 270 questions were generated (range per cancer 32-68). The score frequency distribution was: 5) 45.2%; 4) 19.3%; 3) 13.3%; 2) 13.7%; and 1) 8.5%. On subgroup analysis, Correctness was seen in 210 (77.8%) of questions, and Accuracy with 174 (64.4%). The difference in Correctness scores between cancers was not statistically significant, and there was no statistically significant difference in scores by question type. There was a statistically significant difference in the Accuracy of ChatGPT between cancers (Table). Conclusions: ChatGPT was significantly more likely to provide accurate responses to questions regarding GA and PN versus AN or SB. It demonstrates a limited capacity to assist with complex clinical decision-making in 6 gastrointestinal cancers. However, the Accuracy level is below the acceptable threshold for implementation into clinical use. Further analysis of the expanding capabilities of ChatGPT and other NLP-based tools is warranted in this rapidly evolving domain. Future studies would benefit from a validating grading instrument. [Table: see text]
Implementing large language model-based artificial intelligence (AI) technology in proposing effective treatment plans in patients with cancer.
Authors: Santoki Aditya , Jones Rachel , Peter Manas , Mathew Aju
Published: May 2024

e13660

Background: Language Learning Models (LLM), are programs that can respond to queries without previous exposure to the input. This is because the programs are actually generative mathematical models trained on hundreds of terabytes of textual data. The output is thus a statistical assessment of the corpus of human generated text likely indicating the correct answer. ChatGPT is an example of an LLM that can understand questions and synthesize responses based on probability. However Chat GPT’s potential to accurately assist physicians with a diagnosis, remains yet to be determined. The goal of this study is to elucidate AI’s potential in proposing both feasible and accurate treatment plans specifically for patients with cancer. Methods: Fifty patients at a medical oncology practice were recruited for this study. For each patient, the physician’s shorthand notes were recorded and organized. ChatGPT was then asked to come up with a potential treatment plan using these notes according to a predetermined set workflow for each individual. The responses were then reviewed by a medical oncologist for accuracy and feasibility via a qualtrics survey. Results: ChatGPT correctly proposed a treatment plan in agreement with the physician for a total of 18 cancer patients allowing for the agreement rate to be 36%. For the rest of the 32 patients, the physician disagreed with the proposed ChatGPT treatment plan on the basis of lack of personalization, inhibitive cost, incorrect recommendation, and greater alternative options. Thirteen patients fell into the ‘Lack of Personalization in Treatment' category, twelve in the ‘Incorrect Recommendation’ category, four in ‘Inhibitive Cost' category, and three in the ‘Greater Alternative Options’ category. Most patients however fell in the category of personalization, suggesting ChatGPT’s unfamiliarity with the subject of generating personal treatment plans. Conclusions: In the current study, ChatGPT was expected to create a treatment plan with only the clinician’s abridged shorthand notes and without any familiarity with the physician’s personal preferences for treatment. This knowledge gap can account for an agreement rate less than 50%. Despite its flaws, ChatGPT was, however, able to generate an adjusted and corrected version of the treatment plan when prompted further outside the set workflow. This highlights the invaluable role of clinical experience in synthesizing an accurate treatment plan. With its encroachment into the medical field, ChatGPT has shown potential in proposing feasible and accurate treatment plans, especially in the oncology clinical setting. Despite the low agreement rate obtained from this study, ChatGPT has shown promise in developing an agreeable treatment plan with the oversight of the attending physician.
Real-world implementation of an EHR-integrated, SMS text remote patient monitoring platform at an academic cancer center.
Authors: Shah Sumit , Kadurugamuwa Gayani , Holmes Tisch Alison , Lu Monika , Modarresi Shiva , Johnston Caleb , Colbert James
Published: May 2024

e13669

Background: Remote monitoring studies in oncology have shown improved quality of life, decreased healthcare utilization, and even improved overall survival in an oncologic population, however applications in a real-world setting have been limited. Barriers to successful deployment include low patient engagement rates with smartphone apps, lack of integration with the EHR and burdensome clinical workflows for staff. Methods: We sought to assess the feasibility of an EHR-integrated, SMS text conversational AI agent to automate the collection of ePRO data as well as the assessment and triage of common patient symptoms for patients with GI and GU tumors receiving anti-cancer therapy. This program was developed collaboratively by a health system digital health team, nursing informatics, IT support resources, oncology nurses, medical oncologists, and representatives from Memora Health. Core components included 1) weekly ePRO collection based on the NIH’s CTCAE, 2) NLP activated symptom triage for 16 common symptoms and toxicities 3) weekly symptom check-in 4) weekly medication adherence assessment for patients on oral therapy. Patients received all messages through SMS on their phones, and all data collected were routed into the EHR. In-EHR visualizations allowed nurse coordinators to track ePRO symptoms over time and symptoms that met pre-specified criteria triggered EHR-based alerts. Results: Of 99 patients ordered into this SMS program, 83 enrolled (84% enrollment rate), with 55 Uro Onc patients; 44 GI Onc patients. A total of 2,583 surveys were sent to patients. 98% of patients completed at least 1 survey. Overall patient survey completion rate was 62%. Medication adherence surveys had the highest completion rate at 81%. ePRO completion rates were higher for GU (65%) vs GI (39%) patients (p<0.01). 71% of ePRO surveys generated a symptom alert to the nursing staff. Patient satisfaction was high with 93% rating this SMS program as “excellent” or “good” (n=78). Care team surveys found that 3 of the 6 Oncologists believe ePRO collection is valuable and 3 of 6 used ePRO data to adjust a care plan. Nurse coordinators had a mixed view of the program with a majority reporting that there were too many symptom alerts. Conclusions: We successfully deployed an EHR-integrated SMS program to collect ePRO data, track patient symptoms, and monitor adherence to oral therapy. We believe having a physician champion influenced patient engagement in the GU patient population. Patients had a favorable impression of this program and survey completion rates demonstrated ease of use for patients. Clinical staff, particularly nurses, felt that there were too many alerts. Additional workflow changes may reduce alert burden and better incorporate ePRO learnings into clinical practice.
Health information and technology engagement for clinical research and healthcare purposes (HITECH) survey of patients with a history of cancer.
Authors: Hamparsumian Anahid , Naeim Arash
Published: May 2024

e13672

Background: Digital technology offers oncology clinicians and researchers the ability to more closely monitor patients on therapy, including with electronic patient-reported outcomes (ePROs) and real-time passively collected data (for example: heart rate, oxygen saturation, etc.), via smartphone and wearable devices. In order to realize the potential for these technologies to improve patient experience and outcomes, it is critical that we consider the perspectives of patients to design usable and useful digital monitoring programs. Methods: We conducted a cross-sectional survey of cancer patients from a large academic medical institution to study willingness to use digital devices to collect data and to share discrete types of data for the purposes of medical care or research. A survey including baseline characteristics and questions pertaining to digital technology use was sent to participants enrolled in the UCLA Mental Health and Health Technology Research Registry, from November 2023 to January 2024. Participants were included if they were patients at UCLA, age 18 years and older, owned a smartphone and/or wearable device, and self-reported a history of cancer. The primary outcomes of interest were willingness to collect and share digital device data. We report summary statistics for our population. Results: A total of 690 patients participated in our survey. The majority were female (58.5%), 76.7% non-Hispanic white, 8.8% Hispanic, 5.8% Asian, and 2.0% black. Median age was 69 years old. Nearly all participants reported experience using a patient portal (98.9%). Willingness to collect data using a digital device, for the purposes of clinical care was reported by 93.3% of patients. Similarly, 92.0% were willing to use a device to collect health data for research purposes. Sharing patterns were similar regardless of whether data was intended to be for clinical care or research purposes (Table). Conclusions: The overwhelming majority of patients who own a smartphone and/or wearable are willing to collect health data on their digital device for either clinical care or research. There is a preference for sharing more objective clinical information (vitals and physical activity measurements), and less inclination to share subjective information (daily activity, symptoms, and mood). This could be attributable to the greater effort required to input information as well as skepticism of the usefulness of the subjective data. Notably, we must recognize that some patients are not well-represented in this study population. Their perspectives should also be explored to prevent widening of the digital divide. [Table: see text]
Unraveling the impact of social determinants of health on HER2+ early breast cancer screening and treatment in a county oncology setting in China.
Authors: Ji Yinghua , Zhou Feidu , Qu Honglan , Wu Qianfu , Liang Wei , Liu Mengyou , Wang Juan , Luo Jun , Tan Xiang , Dai Guohua , Hu Guinv , Meng Qiuli , He Wenbo , Wang Hongjian , Qin Zien , Luo Guoxiang , Tang Huali , Ren Yun , Liu Hui , Tian Yinguo
Published: May 2024

e13677

Background: HER2 positive breast cancer (BC) accounts for 15-20% of all BC cases. Various barriers, including social, demographic, and economic factors, may hinder BC screening and treatment. We examined the barriers to screening and treatment for patients (pts) with HER2 positive early BC (eBC) in a county setting in China. Methods: This analysis investigated patient profiles, treatment patterns, and social determinants of health (SDOH) indicators among newly diagnosed pts with HER2+ eBC from in 26 hospitals in a real-world county setting. Univariate (UVA) and multivariate logistic regression analysis (MVA) were performed to explore the association between SDOH and treatment. Results: 878 pts with HER+ eBC were identified, with a median age of 53 years. 12.8% received BC screening, 18.5% received neo-adjuvant targeted therapy (NATT), 9.9% received breast-conserving surgery (BCS), 23.5% received radiotherapy (RT); 87.8% received adjuvant targeted therapy (ATT), and 52.5% received dual-targeted therapy. The UVA and MVA found differences in treatment patterns associated with age, employment, education level, insurance type, and regions. Pts in the central China regions had a higher likelihood of receiving BC screening. Pts who received NATT were more likely to be located in the western. Younger pts and those in the north-eastern areas were primarily likely to receive BCS. RT was relatively common among pts with a higher-level education, employee medical insurance, and residents in the north-eastern. Pts from eastern regions were more likely to receive ATT, while younger pts were more likely to receive dual-targeted therapy. Conclusions: In the county oncology setting, we observed inequities in screening and treatment based on social determinants analyzed. Promoting standardization of BC screening and treatment should take the sociodemographic disparities into consideration. To address these disparities, it is necessary to raise public awareness, promote high-quality, balanced development and integrate urban-rural areas in compulsory education, allocate more quality medical resources to the county level and ensure their even distribution among regions. [Table: see text]
Effects of socioeconomic status (SES) and healthcare resource (HCR) availability on survival in Black patients (pts) versus (vs) White pts with multiple myeloma (MM): A SEER Medicare analysis.
Authors: Li Sophia S. , Schuldt Robert , Zafar Faiza , To Tu My , Yellow-Duke Archibong , Levine Alina , Girvan Allicia , Mikhael Joseph
Published: May 2024

e13678

Background: In MM, disparities exist for overall survival (OS) by race, and many factors may contribute to these differences. This study evaluates whether SES and HCR availability impacts OS in older pts with newly diagnosed MM (NDMM) based on race. Methods: This retrospective study used linked SEER-Medicare claims (diagnosis [Dx] 6/1/2013–12/31/2017) and Area Health Resource Files (AHRFs; from 2018) to identify non-Hispanic White and non-Hispanic Black pts aged ≥66 years with NDMM. AHRFs included county-level HCR availability data (e.g. number of long-term hospital beds, medical doctors [MDs]). Continuous Medicare A and B enrollment until 12/31/2019 or preceding death was required. Pts with other cancers, Dx from death data, missing county information and enrollment in health maintenance organizations or clinical trials were excluded. Kaplan-Meier was used for unadjusted median OS and Cox proportional hazards for OS risk by race. Age, sex, race, Charlson Comorbidity Index (CCI), poverty, education, and HCR availability were included in the adjusted mixed effects Cox model. Results: We identified 6,609 pts with NDMM; 15.6% were Black. For Black pts vs White pts, median age at Dx was 76 vs 77 years, 57.1% vs 46.0% were female, and mean CCI was 2.1 vs 1.4 (p<.001 for all). Within census tracts, mean percentage of pts living under the poverty line and received <12 years of education was 21.2% vs 11.3% (p<.001) and 19.0% vs 11.3% (p<.001) for Black pts vs White pts, respectively. Unadjusted median OS for Black pts vs White pts was 2.76 vs 3.01 years (hazard ratio [HR]: 1.08; p=.063). In the adjusted model, several SES and HCR availability factors were significant. OS HR by race reversed directionality when adjusted for demographics, SES and HCR availability, but this was not significant (HR: 0.92; p=.12; Table). Conclusions: We found that Black pts with NDMM in SEER-Medicare claims were younger, had higher CCI, completed less education, and were more likely to live in census tracts with higher poverty percentages vs White pts. Several factors relating to SES and HCR availability significantly influenced OS and may drive disparities in care. Further research on larger datasets is needed to better understand these disparities. [Table: see text]
Racial and ethnic disparities in hospital outcomes following hospitalizations for mastectomy for breast cancer.
Authors: Sandoval-Leon Ana Cristina , Chamorro Yolcar , Rubens Muni , Roy Mukesh , Carcas Lauren , Dempsey Naomi , Ahluwalia Manmeet Singh , Mahtani Reshma L.
Published: May 2024

e13691

Background: Racial and ethnic disparities in breast cancer (BC) treatment and survival have been well documented. White (W) and Asian (A) women are more likely to have localized BC (BC) than Black (B) or Hispanic (H) women. Furthermore, the incidence of triple negative BC is higher in B women who also have the highest death rates from BC. In this study, we aimed to identify an association between race/ethnicity and outcomes following hospitalization for mastectomy using a large national database. Methods: This study isa retrospective analysis of the National Inpatient Sample database, the largest all-payer inpatient database in the United States, of hospitalizations between 2016-2019. Patients (pts) ≥18 years of age, admitted for mastectomy for BC were included and stratified by race/ethnicity. The outcomes of the study were in-hospital mortality rate, prolonged hospital length of stay, surgical site infection, bleeding complication, cardiac complication, respiratory complication, acute kidney injury, urinary tract infection, and wound dehiscence. Hospitalizations and outcomes were identified using ICD-10 diagnosis and procedure codes. Logistic regression analysis was done to identify an association between race and outcomes. Results: A total of 302,208 pts who were hospitalized for mastectomy were included. Of these 218,092 (72.2%) pts were W, 36,242 (12.0%) were B, 25,130 (8.3%) were Hispanics, and 22,744 (7.5%) were of another race. Among B pts, adverse outcomes were significantly higher (p<0.001), including in-hospital mortality rate (0.2%), prolonged hospital length of stay (64.1%), surgical site infection (0.8%), bleeding complication (4.2%), cardiac complication (0.8%), respiratory complication (0.3%), and acute kidney injury (1.4%). After adjusting for various factors including age, insurance, hospital region, hypertension, diabetes, obesity, peripheral vascular disease, alcohol use, Elixhauser comorbidity index, whether pts underwent axillary lymph node dissection and others, logistic regression was performed. This analysis identified that B pts had significantly higher odds for mortality (adjusted odds ratio [aOR], 2.99, 95% CI: 1.62-5.73), prolonged hospital length of stay (aOR, 1.32, 95% CI: 1.18-1.49), and bleeding complication (aOR, 1.28, 95% CI: 1.13-1.48). Conclusions: Results of our study show that adverse hospital outcomes following mastectomy for BC were significantly higher among Black patients. These outcomes remained significant after adjusting for clinical covariates, raising the possibility of other contributing factors to these differences in outcomes. Study limitations include the lack of information on stage at diagnosis, receptor status, and that Hispanics were categorized under race. Further studies are needed to develop effective interventions to reduce these disparities.