GAI vs human expert: A case study of critical reading of primary literature
Creators: name: Bjorn, Genevive
name: Burgasser, Adam
Publisher: name: Zenodo
Year: 2023
Subjects: higher education
generative AI (GAI)
critical reading
primary literature
STEM
qualitative analysis
case study

description: Critical reading primary literature (CRPL) is a crucial skill set in STEM education. The CERIC method offers a structured reading approach for scientific argumentation to address variations in teaching this skill. Despite its potential, the method's instructional value has not been assessed within the context of generative artificial intelligence (GAI) systems, like those now present in STEM courses. This exploratory study compared the CERIC outputs of a human expert (a tenured physics professor at an R1 university) and a GAI (Humata.AI on GBT-3.5). While the GAI provided generalized responses resembling novice readers, the human expert showcased detailed, context-rich analyses indicative of a seasoned professional. Other significant differences included depth and specificity, example selection, adaptability, overall understanding, and critical evaluation. The results suggest that GAI can aid beginners in grasping CRPL to a general level, but human expertise remains unparalleled for nuanced, expert-level comprehension. Merging GAI tools with expert instruction could accelerate CRPL skill development from novice to expert levels. Future studies should explore hybrid educational pathways, considering known limitations of the GAI system, such as overgeneralization and redundancy.
description: {"references": ["Kwan, B. S. C. (2008). The nexus of reading, writing and researching in the doctoral undertaking of humanities and social sciences: Implications for literature reviewing. English for Specific Purposes, 27(1), 42\u201356.", "Margolis, E. (Ed.). (2001). The hidden curriculum in higher education. London, UK: Psychology Press.", "Bandura, A., &amp; Walters, R. H. (1977). Social learning theory (Vol. 1). Prentice Hall: Englewood Cliffs.", "Schunk, D. H. (2012). Information processing theory. In Learning theories: An educational perspective (6th ed., pp. 163\u2013227). United Kingdom: Pearson, Inc.", "Toulmin, S., Rieke, R., &amp; Janik, A. (1984). 27. Argumentation in science. In An introduction to reasoning (pp. 313-348). New York, NY: Macmillan Publishing Co.", "Bjorn, G., Quaynor, L., &amp; Burgasser, A. J. (2022). Reading research for writing: Co-constructing core skills with the CERIC method. Impacting Education: Journal on Transforming Professional Practice, 7(1), 1\u201313. https://doi.org/10.5195/ie.2021.237", "Bjorn, G. (2023, June 5). Exposing the hidden curriculum: Instructional strategies for critical reading of the primary literature that are feasible and effective in graduate education. https://doi.org/10.31235/osf.io/dzufr", "Bjorn, G. (2023). The power of peer engagement: Exploring the effects of social collaborative annotation on reading comprehension of primary literature. AI, Computer Science, and Robotics Technology, 2. https://www.intechopen.com/journals/1/articles/227", "Kong, S. C., Ogata, H., Shih, J. L., &amp; Bissau, G. (2021, November). The role of Artificial Intelligence in STEM education. In Proceedings of 29th International Conference on Computers in Education Conference (pp. 774-776). Asia-Pacific Society for Computers in Education."]}