Comparing New Tools of Artificial Intelligence to the Authentic Intelligence of our Global Health Students
Authors: Thandla Shilpa R. , Armstrong Grace Q. , Menon Adil , Shah Aashna , Gueye David L. , Harb Clara , Hernandez Estefania , Iyer Yasaswini , Hotchner Abigail R. , Modi Riddhi , Mudigonda Anusha , Prokos Maria A. , Rao Tharun M. , Thomas Olivia R. , Beltran Camilo A. , Guerrieri Taylor , LeBlanc Sydney , Moorthy Skanda , Yacoub Sara G. , Gardner Jacob E. , Greenberg Benjamin M. , Hubal Alyssa , Lapina Yuliana P. , Moran Jacqueline , O'Brien Joseph P. , Winnicki Anna C. , Yoka Christina , Zhang Junwei , Zimmerman Peter A. 
Published: November 2024

Abstract

Introduction - The transformative feature of Artificial Intelligence (AI) is the massive capacity for interpreting and transforming unstructured data into a coherent and meaningful context. The potential that AI will alter traditional approaches to student research and its evaluation appears to be significant.
Methods – Graduate students who successfully completed a required research paper in Fundamentals of Global Health (INTH 401) at Case Western Reserve University (CWRU) were recruited to compare their original papers, focused on a globally significant infectious disease in a geographic region of interest based on three perspectives taught in the course, with a paper generated by ChatGPT4o using the original assignment prompt. Students also completed a Google Forms survey to evaluate different sections of the AI-generated paper (e.g., Adherence to Introduction guidelines, Presentation of three perspectives, Conclusion) and their original papers and their overall satisfaction with the AI work. The original student to ChatGPT4o comparison also enabled evaluation of narrative elements and references.
Results – Of the 54 students who completed the required research paper, 28 (51.8%) agreed to collaborate in the comparison project. A summary of the survey responses suggested that students evaluated the AI-generated paper as inferior or similar to their own paper (overall satisfaction average = 2.39 (1.61-3.17); Likert scale: 1 to 5 with lower scores indicating inferiority). Evaluating the average individual student responses for 5 Likert item queries showed that 17 scores were <2.9; 7 scores were between 3.0 to 3.9; 4 scores were ≥4.0, consistent with inferiority of the AI-generated paper. Evaluation of reference selection by ChatGPT4o (n=729 total references) showed that 54% (n=396) were authentic, 46% (n=333) did not exist. Of the authentic references, 26.5% (105/396) were relevant to the paper narrative; 14.4% of the 729 total references.
Discussion – As indicated by the student assessments, ChatGPT4o produced content very quickly. Some students commented that there were AI-suggested topics that they had not considered in the 3 perspective sections. AI-integration of content into a meaningful summary received the greatest number of significantly inferior scores (7 students). Many stated that the summary simply reworded content from earlier sections. Consistently presenting up-to-date facts and references, as well as further examining the complexities of global health topics, appears to be a current limitation of ChatGPT4o. Our findings reveal strengths and limitations on the potential of AI tools to assist in understanding the complexities of global health and suggest approaches for integrating AI tools in global health education. This exercise has also illustrated the limited current capacity of AI tools to examine complex global health challenges.