bluuebunny/crossref_metadata_2025_split huggingface dataset:
Investigating large language model (LLM) performance using in-context learning (ICL) for interpretation of ESMO and NCCN guidelines for lung cancer.
Authors: Iivanainen Sanna , Lagus Jarkko , Viertolahti Henri , Sippola Lauri , Koivunen Jussi 
Published: May 2024

e13637 Background: The recent development of advanced LLMs has been suggested to improve patient care across several areas such as clinical-decision support or helping to answer patients’ questions. Hallucinations have been identified as a blocker for the use of LLMs in routine clinical practice. ICL and Retrieval Augmented Generation (RAG) could improve the LLM performance and reduce hallucinations, consecutively making the use of LLMs possible in clinical practice. Methods: A method using ICL and RAG was developed on top of health AI platform (Gosta MedKit) to interpret the most recent ESMO (Dec 2022 for NSCLC, Mar 2021 for SCLC) and NCCN (Nov 2023 for NSCLC and SCLC) clinical guidelines for lung cancer. Guidelines (including tables and diagrams) were curated into a text format, and split and stored into a vector database. OpenAI’s GPT4 Turbo model version gpt-4-1106-preview (GPT4-T), having the knowledge cutoff in April 2023, was used in all implementations. 11 questions about SCLC and 13 questions about NSCLC treatment recommendations and definitions were developed to evaluate the performance of different settings: GPT4-T (existing knowledge of the model), ICL with maximum context (ICL-MC) length (128k tokens) and ICL with RAG (ICL-RAG) heuristically including only the most relevant parts from vector database. Question prompts were generated for different settings and guidelines (ESMO, NCCN and both combined) and two oncologists evaluated 216 different responses and their alignment with ESMO and NCCN guidelines. Results: For responses using ESMO guidelines having oncologists’ consensus, ICL-MC and ICL-RAG respectively provided accurate responses for 83.3% and 79.2% of questions vs. 62.5% for GPT4-T. For responses using NCCN guidelines having oncologist consensus, ICL-RAG provided accurate responses for 83.3%, GPT4-T for 75.0% and ICL-MC for 33.3% of questions. When more flexibility was allowed in results interpretation (alignment either with ESMO or NCCN), GPT4-T provided accurate response for 87.5% vs. 70.8% with ICL-RAG and 58.3% with ICL-MC. No consensus around hallucinations was reported for ICL approaches, whereas GPT4-T hallucinated the response for 4.2% of questions with ESMO guidelines. Conclusions: ICL seems to improve the LLM performance for stricter tasks such as providing responses according to specific guidelines and reducing hallucinations. ICL outperformed GPT4-T in case of ESMO guidelines. This highlights the importance of taking local and latest guidelines into account when LLMs are used across different health systems and regulatory environments. In line with earlier studies, longer context for ICL makes models forget crucial information, which can be mitigated with the use of RAG to improve ICL performance and reduce costs when using the models.
PENTAGON (Predicting Clinical Trials in Gynecologic Oncology): A retrospective study assessing study design factors that affect enrollment in gynecologic cancer trials.
Authors: Pirzadah Humza , Klein Margaret , Magharehabed Yasmeen , Chapple Andrew G , Castellano Tara , Jernigan Amelia 
Published: May 2024

e13647 Background: Obstacles to attaining a diverse clinical trial population contribute to poor representation in medicine, increasing health disparities by limiting generalizability and applicability of findings. Our objective is to evaluate trial enrollment trends according to patient and trial demographics for those who have sought care at our multi-center, mixed-setting Gynecologic Oncology practice. Additionally, we aim to determine specific trial criteria that promote or impede enrollment for patients and physicians. Methods: An IRB-approved, retrospective cohort study evaluating all patients who screened positive for a clinical trial through the Gyn Onc practice’s manual screening process was performed. Screening events were new neoplasms, recurrences or progressions, and changes in treatment. Data was collected from July 2022 to December 2023. Demographic information, trials for which each patient was considered, trials patients enrolled on, and trial characteristics were recorded. Results: After adjustment, there were no significant differences between patient factors (age, marital status, race, ethnicity, and number of screening events) and enrollment status. Of trials available, 9% require biopsy and 94.7% are in person. Most studies are interventional (78.9%) and pharmaceutical trials (73.7%). There was a drastic unadjusted estimated increase in enrollment for Medicaid patients compared to other insurances (55.2% vs 32%, p=.031). There was also an increase in enrollment for cervical cancer patients (18.4%, p=.106) compared to other cancer sites (56.5% vs 32.8%, p=.036). There was a positive relationship between distance traveled to the treatment site and clinical trial enrollment. Patients who lived within 20 miles of the clinical trial site had an enrollment percentage of 30.1% (n=93), compared to 46.55% enrollment for those who lived further (n=58). There was a significant increase in enrollment likelihood without an exclusion criterion of previous history of other cancer (50% vs 33.3%, p=.046). Trials that required no history of chemotherapy had an enrollment of 27.3% (compared to 44.1%). Conclusions: Unlike prior reports, we did not demonstrate differences in trial enrollment by race, socioeconomic or insurance status. This suggests that our objective manual screening process enhances equitable clinical trial offerings. It is possible that this objective manual screening process attenuates provider bias and offers clinical trials to historically marginalized and underrepresented groups.
Effects of preoperative breast MRI on surgical management in women aged 40 years and younger with breast cancer.
Authors: Erguven Tugce , Arafat Ayah , Brem Rachel 
Published: May 2024

e13656 Background: Evaluating the use of preoperative breast MRI on the management of breast cancer can help guide surgical decision making for young female patients newly diagnosed with breast cancer. Methods: A retrospective chart review of the George Washington University database between January 2015 and December 2020 identified 125 women age 40 or younger diagnosed with breast cancer. Of the 125 women, 23 were excluded from the study as they were lost to follow up due to care transfers. The management impact of pre-operative MRI was examined and variables collected include age, race, family history, genetic predisposition, breast density, method of detection, initial imaging findings, initial biopsy results, preoperative MRI findings, additional biopsies, additional biopsy results, and surgery type. Results: Patient ages ranged from 22 to 40 (mean 34.5), 69 (67.6%) had a family history of breast cancer, 81 (79.4%) had dense breasts (30 extremely dense and 51 heterogeneously dense), and 82 (80.4%) had the cancer detected due to clinical symptoms with the remaining during surveillance due to high risk. Of all women included, 55 (53.9%) had additional findings on MRI warranting biopsy of which 25 (45.5%) were biopsy proven cancer. 39 (39.4%) patients had lumpectomy and 59 (59.6%) had mastectomy. 1 (0.01%) patient did not undergo surgery due to widespread metastasis. The use of preoperative MRI identified additional carcinoma in 25 (24.5%) out of 102 patients. 24 (96%) in the ipsilateral breast, and 1 (4%) in the contralateral breast. Conclusions: Overall, this data indicates that the use of preoperative MRI for the treatment of breast cancer in young patients increases the detection of occult breast cancer prior to definitive therapy in 24.5% of patients, critically important information for optimal management. We recommend that all young women with newly diagnosed breast cancer undergo pre-operative MRI to further optimize patient care.
Finding errors in the blink of AI: Use of an artificial intelligence tool to identify clinically important corrections to references cited by NCCN Guidelines for gynecologic malignancies.
Authors: Matthews Benjamin J , Ferriss James Stuart 
Published: May 2024

e13658 Background: Scientific articles are being retracted at higher rates for many reasons. Retractions or corrections may be unknown to authors citing these materials as primary sources, unwittingly perpetuating incorrect information. Our goal was to assess the utility of a scholarly artificial intelligence (AI) tool to identify errata to sources cited in NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) for gynecologic cancers and to evaluate the clinical importance of identified errata. Methods: scite (scite Inc., Brooklyn, NY) is an AI tool that analyzes citations to generate a list of linked publications, display the context for each citation, and flag any published errata or retractions for each source. Using scite with references from NCCN Guidelines as of June 2023, we identified corrections to primary sources. Each author categorized errata as having major, minor, or no clinical relevance, with resolution of conflicts through discussion until consensus. To check whether scite missed errata, we assessed a random sample (20%) of unflagged references. Results: The guidelines cited 2,364 primary sources, and scite successfully linked to 1,984 (84%) of them . scite flagged 50 references (2.5%) as having corrections; all 50 did have errata (100% accuracy). scite identified 56 total errata, which were not always apparent on journal websites or PubMed. After review, 31 errors (55%) were classified as clinically relevant; 71% of those errors had major clinical importance and 29% were considered minor. Twenty errors (65%) had relevance to the guideline statements in which the source was cited, but only 1 (3%) error had potential to change current guideline text. Among the sample of sources scite did not flag, we identified none with published errata. Conclusions: We demonstrated use of an AI tool to accurately identify sources in clinical guidelines with published corrections. Sources with errata were uncommon, and our analysis did not reveal perpetuation of major errors in NCCN Guidelines. However, the proportion of corrections with major clinical relevance underscores the importance of awareness of errata when writing clinical guidelines or reviews. Compared to manual review of each source, AI tools can improve speed and accuracy in finding important errors. [Table: see text]
Evaluation of a web-based, interactive guide to enhance patient-clinician electronic communication.
Authors: Alpert Jordan M , Amin Tithi B. , Zhang Zhongyue , Markham Merry Jennifer , Murphy Martina Cathryn , Bylund Carma L 
Published: May 2024

e13659 Background: Secure messaging (SM) enables patients to ask clinicians questions asynchronously. Benefits include enhancing patient engagement, contributing to oral chemotherapy adherence, and lowering symptom distress. However, patients are unsure how to address concerns effectively, and clinicians find it challenging to decipher patients’ needs within messages. Coupled with clinicians not having adequate time to respond to messages, breakdowns in communication are common. We developed an interactive guide to help patients with cancer craft effective secure messages. It featured an oncologist providing strategies to write messages with specificity, expressing concerns, stating needs, being direct, and included a downloadable tip sheet. Methods: Patients 18+ with a current cancer diagnosis who were enrolled in the portal at a large health center were recruited. Between October 2023 and January 2024, participants were sent a link to interact with the guide, followed by a survey. The survey included several measures: 1) Communication and Attitudinal Self-Efficacy scale for cancer (CASE-cancer); 2) Digital Behavior Change Intervention (DBCI) Engagement Scale; 3) Website User Satisfaction questionnaire (WUS); 4) Acceptability of Intervention Measure (AIM); 5) Intervention Appropriateness Measure (IAM). We used R version 4.3.2 for the descriptive analysis. The Wilcoxon rank-sum test compared groups by demographic categories. The local IRB approved this study. Results: One hundred patients participated. Median age was 64 years (range=21-86), 54% male, 83% white, 35% diagnosed with prostate cancer, 35% lung cancer, and 23% hematologic cancer. Respondents reported high levels of self-efficacy based on CASE-cancer (mean 43.3 of 48). Respondents averaged 15 minutes using the guide. 96% thought the guide’s content was presented clearly, 95% found it easy to understand, 90% enjoyed using it, 89% thought the guide was suitable, and 88% could rely on its information. Results by demographic categories found that non-white respondents (M = 43.0) were significantly more engaged than white respondents (M = 28.4), P = 0.038 , and those with a bachelor’s degree+ (M = 96.5) had higher satisfaction than those without (M = 91.6), P = 0.025. Conclusions: Patients with cancer embraced an interactive guide that provided strategies to craft secure messages. They found it engaging, satisfying, acceptable, and appropriate, with racially minoritized patients being most engaged. As SM is a widely used tool for patient-clinician communication, it is crucial to concentrate on the way messages are constructed. We found potential to improve patients’ message-writing techniques, which could result in clinicians writing more complete answers to questions. Additionally, there is the possibility to enhance underserved patients’ communication skills and make them more engaged in their care.
MSK Engage: A comprehensive program for collection of patient reported outcomes at scale.
Authors: Polubriaginof Fernanda C. G. , Lipitz-Snyderman Allison , Chimonas Susan , Moy Morgan , Kuperman Gilad , Stetson Peter D. 
Published: May 2024

e13666 Background: Collecting patient-reported outcomes (PROs) yields high value data for clinical care and research and is associated with improved survival in oncology. However, collection of PROs at scale requires a multifaceted program that in most institutions is not available. This study describes a comprehensive PRO program to support research and standard clinical care at scale. Methods: Memorial Sloan Kettering Cancer Center is a high-volume cancer center in New York. Beginning in 2016, we developed a comprehensive PRO program, MSK Engage, with the following capabilities: (i) A governance process to oversee the creation of questionnaires for both research and clinical care. This process includes alignment with the IRB and standardization of questions presented to patients. (ii) Technology to determine which patients should get an assessment (i.e., instance of a questionnaire) at what point in time. (iii) An interactive module in our patient portal that would present the assessment to the patient. (iv) Various methods to present the PRO data to providers to support clinical decision making. These methods included (a) alerts based on threshold logic that would send secure messages to the clinical team, (b) integration of the PRO data into the EHR to be viewed with other clinical data, and (c) the creation of specialized workflows. The results of assessments are also stored in an analytics database to support operations, research, and the creation of smart reports that compare a patient’s responses with other similar patients. Results: We analyzed the use of the program from 2016 to 2023 (Table). 189 questionnaires have been implemented, generating 3,098,574 completed assessments from 314,685 distinct patients. Completion rates varied. The median completion time for all questionnaires was 2 minutes. Examples of programs enabled by patient-reported outcomes include post-operative symptom monitoring, family history questionnaires to support clinical genetics decision making and operations, intake and follow up questionnaires that support clinical documentation and quality of life questionnaires to support goal concordant care. Conclusions: Adoption of PROs at scale for both research and standard of care in oncology is feasible and innovative programs of care can be supported while maintaining a centralized and acceptable patient experience. This model can be applied across institutions seeking to develop or scale PRO programs to assure patient data are meaningfully incorporated into decision-making. [Table: see text]
Engagement of patients with cancer with digital health platforms: An age-based analysis.
Authors: Kosmidis Paris A. , Baxevanos Panagiotis , Kosmidis Thanos 
Published: May 2024

e13671 Background: Use of digital health platforms has been increasing, including those used by patients directly. In cancer, the nature of the condition, and patient age (higher vs average online users), are among the factors expected to reduce engagement. However, there is limited research to identify and quantify such correlations. We retrospectively analyze them for 4 cancer types. Methods: Patients engage directly with CareAcross platforms to receive information and support based on the data they anonymously provide. These include diagnosis, treatment, optionally year of birth; and side effects (repeatedly, with other inputs in between). On platforms used by breast, lung, prostate and colorectal cancer patients without clinician participation, the number of side effect reports submitted within 6 months of registration was analyzed as a patient engagement metric. This was correlated with patients' ages at registration, also yielding metrics for 10-year age groups around median age. Pearson correlation coefficient (r) was calculated. Results: 1010 breast, lung, prostate and colorectal cancer patients mainly from UK, Germany, France, Spain and Italy have reported birth years. 747 breast cancer patients (28-86yo; median: 53) submitted 2.34 reports on average. Patients above median age tended to submit more reports in general (2.56 vs 2.15 for younger than median); r = 0.055. 10-year age groups analysis: 34-43yo: 2.01 reports (N=109); 44-53yo: 2.19 (287); 54-63yo: 2.61 (236); 64-73yo: 2.45 (94); 74-83yo: 2.92 (12). 135 lung cancer patients (35-84yo; median: 62) submitted 3.27 reports on average. Older patients tended to submit more reports (r = 0.031) except the oldest 10-year age group analyzed (15 patients aged 73-82 reported 1.47 times). 99 prostate cancer patients (48-90yo; median: 68) submitted 2.87 reports on average. Overall correlation was positive: r = 0.039; patients above vs below median age reported 3.47 vs 2.45 times. Beyond these, age groups were not consistent: highest average number of reports was from 69-78yo (average 3.83; N=40) followed by 49-58yo (average 3.77; N=13). 29 colorectal cancer patients (38-81yo; median: 62) submitted 2.34 reports on average, showing negative correlation with age (r = -0.097). Age groups were small and their analysis inconclusive. Conclusions: Although evidence is not clear or uniform, the retrospective analysis of these self-directed digital health platforms shows that older patients appear to engage more than their younger counterparts. More, prospective, research can identify confounders and offer opportunities to improve engagement across all age groups. [Table: see text]
Disparities in cancer care: Is there a benefit for increased use of inpatient chemotherapy (IPC) in safety-net hospitals?
Authors: SUBEDI PAWINA , Gralla Richard J. , Bapatla Anusha , Richy Jessica 
Published: May 2024

e13690 Background: Safety net hospitals provide care to a large number of vulnerable patients (pts) regardless of their ability to pay (1). These pts often have barriers to acceptance at many hospitals, including lack of or poor insurance, social and economic challenges. Additionally, pts frequently present with more advanced disease (2). Systemic treatment is preferentially administered on an ambulatory basis given several factors: clear financial incentives; effective supportive care; the association of good performance status (PS) with better outcomes; and patient preference (3). Only a few malignancies require inpatient systemic treatment. Strong social support is required for safe and effective ambulatory care, thus especially challenging pts at safety net hospitals in which social constraints are common. Methods: We reviewed patterns of IPC to determine efficacy and outcomes over a 3-year period at a safety net hospital, examining: malignancy type, reason for IPC treatment, treatment regimen (typically outpatient or inpatient), length of stay, and survival. Results: 82 pts met criteria. 56% had hematologic or HIV-associated malignancies. Solid tumor types included GI, breast, GYN, GU and small cell malignancies. Reasons for IPC were progression, worsening symptoms, declining PS in 55% and social constraints in 11%. Patient death within 3 months of IPC, or direct discharge to hospice/nursing home occurred in 42%. Further outpatient chemotherapy was never given in 50%. Death occurred in 15% of pts on the same hospitalization as the IPC; several additional pts were readmitted and died within one month. Individuals receiving IPC due to poor social support frequently required rapid rehospitalization, had poor outpatient follow-up, or died shortly thereafter. Three month survival was more favorable in pts with multiple myeloma or those admitted for drug-desensitization. Conclusions: Outcomes, including survival and rapid rehospitalization, were poor for pts receiving typical outpatient regimens on an inpatient basis whether due to social constraints, declining PS or worsening symptoms. A large percentage of pts never received further chemotherapy. IPC is costly, undesired by pts, and is often not a good use of resources. Pts with highly responsive cancers with fair to good PS had acceptable outcomes. IPC should be reserved for such pts if alternative outpatient regimens are not available. Earlier discussion of hospice and goals should not be replaced by chemotherapy. Institutional unavailability of key oral agents and limited outpatient resources often contribute to IPC. Better access to modern agents, enhanced social support, enlarged and better supported outpatient resources, as well as improved decision making improve cancer care. Our results do not support IPC as a frequent approach to cancer care in safety net hospitals. 1. Popescu, JAMA 2019. 2. Farkas, JCO 2012. 3. Mor, J Clin Epid 1988.