Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants
Creators: name: Borges, Beatriz
name: Foroutan, Negar
name: Bayazit, Deniz
name: Sotnikova, Anna
name: Montariol, Syrielle
name: Nazaretzky, Tanya
name: Banaei, Mohammadreza
name: Sakhaeirad, Alireza
name: Servant, Philippe
name: Neshaei, Seyed Parsa
name: Frej, Jibril
name: Romanou, Angelika
name: Weiss, Gail
name: Mamooler, Sepideh
name: Chen, Zeming
name: Fan, Simin
name: Gao, Silin
name: Ismayilzada, Mete
name: Paul, Debjit
name: Schöpfer, Alexandre
name: Janchevski, Andrej
name: Tiede, Anja
name: Linden, Clarence
name: Troiani, Emanuele
name: Salvi, Francesco
name: Behrens, Freya
name: Orsi, Giacomo
name: Piccioli, Giovanni
name: Sevel, Hadrien
name: Coulon, Louis
name: Pineros-Rodriguez, Manuela
name: Bonnassies, Marin
name: Hellich, Pierre
name: van Gerwen, Puck
name: Gambhir, Sankalp
name: Pirelli, Solal
name: Blanchard, Thomas
name: Callens, Timothée
name: Aoun, Toni Abi
name: Alonso, Yannick Calvino
name: Cho, Yuri
name: Chiappa, Alberto
name: Sclocchi, Antonio
name: Bruno, Étienne
name: Hofhammer, Florian
name: Pescia, Gabriel
name: Rizk, Geovani
name: Dadi, Leello
name: Stoffl, Lucas
name: Ribeiro, Manoel Horta
name: Bovel, Matthieu
name: Pan, Yueyang
name: Radenovic, Aleksandra
name: Alahi, Alexandre
name: Mathis, Alexander
name: Bitbol, Anne-Florence
name: Faltings, Boi
name: Hébert, Cécile
name: Tuia, Devis
name: Maréchal, François
name: Candea, George
name: Carleo, Giuseppe
name: Chappelier, Jean-Cédric
name: Flammarion, Nicolas
name: Fürbringer, Jean-Marie
name: Pellet, Jean-Philippe
name: Aberer, Karl
name: Zdeborová, Lenka
name: Salathé, Marcel
name: Jaggi, Martin
name: Rajman, Martin
name: Payer, Mathias
name: Wyart, Matthieu
name: Gastpar, Michael
name: Ceriotti, Michele
name: Svensson, Ola
name: Lévêque, Olivier
name: Ienne, Paolo
name: Guerraoui, Rachid
name: West, Robert
name: Kashyap, Sanidhya
name: Piazza, Valerio
name: Simanis, Viesturs
name: Kuncak, Viktor
name: Cevher, Volkan
name: Schwaller, Philippe
name: Friedli, Sacha
name: Jermann, Patrick
name: Käser, Tanja
name: Bosselut, Antoine
Publisher: name: arXiv
Year: 2024
Subjects: subject: Computers and Society (cs.CY)
subject: Artificial Intelligence (cs.AI)
subject: Computation and Language (cs.CL)
subject: FOS: Computer and information sciences
subject: FOS: Computer and information sciences

description: AI assistants are being increasingly used by students enrolled in higher education institutions. While these tools provide opportunities for improved teaching and education, they also pose significant challenges for assessment and learning outcomes. We conceptualize these challenges through the lens of vulnerability, the potential for university assessments and learning outcomes to be impacted by student use of generative AI. We investigate the potential scale of this vulnerability by measuring the degree to which AI assistants can complete assessment questions in standard university-level STEM courses. Specifically, we compile a novel dataset of textual assessment questions from 50 courses at EPFL and evaluate whether two AI assistants, GPT-3.5 and GPT-4 can adequately answer these questions. We use eight prompting strategies to produce responses and find that GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions. When grouping courses in our dataset by degree program, these systems already pass non-project assessments of large numbers of core courses in various degree programs, posing risks to higher education accreditation that will be amplified as these models improve. Our results call for revising program-level assessment design in higher education in light of advances in generative AI.
description: 20 pages, 8 figures