DOI: 10.5281/zenodo.14836899
title: Supplementary Material - Insights from the Use of LLM to Support the Teaching‑Learning Process of Heuristic Evaluation

description: Experimental Setup:

We conducted the experiment over two separate days. On the first day, the participants inspected the object Canva (Web version), while on the second day, they inspected the object Skoob (Web version). Our study had 31 participants (undergraduate students enrolled in the HCI - Human-Computer Interaction Course).

Initially, participants were introduced to the experiment's objective and the procedure for the activity to be performed. On the first day, the class was divided into two groups G1 and G2, each assigned a different usability inspection method. G1 performed the usability inspection manually, while G2 used Solaria-GPT. On the second day, when evaluating a new system, the groups switched methods: G1 used Solaria-GPT, while G2 conducted the manual inspection. The experiment lasted 1 hour and 20 minutes per session. All participants received a usability inspection guide according to the assigned inspection method.

Instrumentation Description:

The instruments were used to conduct a controlled experiment.



Informed Consent Form (ICF) – used to request participants' permission and present the study’s terms.

Participant Characterization Forms – applied to assess participants' experience levels in usability evaluation.

Inspection Guide – detailing the steps for inspecting both objects (Canva e Skoob) using manual and Solaria-GPT approaches. Each object had a dedicated guide according to the approach.

Spreadsheet for recording defects – used to document identified defects, including a brief problem description, location, and violated heuristic for manual inspection. For Solaria-GPT inspection, additional fields are included, such as violated heuristic returned by Solaria-GPT and level of agreement or disagreement with Solaria-GPT response.

Post-evaluation Questionnaire – applied to assess perceived knowledge after the experiment and collect open-ended feedback from participants.


Files Description:

Each instrument is represented by the following files:



Informed Consent Form (ICF)



Instruments_Informed_Consent_Form.docx



Participant Characterization Forms



Instruments_Participant_Characterization_Forms.docx



Inspection Guide:



Instrument_Inspection_Guide_Canva_Manual.docx

Instrument_Inspection_Guide_Canva_Solaria-GPT.docx

Instrument_Inspection_Guide_Skoob_Manual.docx

Instrument_Inspection_Guide_Skoob_Solaria-GPT.docx



Spreadsheet for recording defects



Instruments_Spreadsheet_Recording_Defects_SolariaGPT.xlsx

Instruments_Spreadsheet_Recording_Defects_Manual.xlsx



Post-evaluation Questionnaire 



Instrument_Post_Evaluation_Questionnaire.docx




Steps to Reproduce:

Preconditions (Before the experiment)







All the participants must assign and agree with the Informed Consent Form (ICF)

All the participants must answer the Participant Characterization Forms

Split the participants in two groups (G1 and G2), according to the participant characterization forms. Follow a balanced and randomized experimental design




First Day (Canva Object)







Use the Inspection Guide, according to the group (G1 or G2) - G1 uses Instrument_Inspection_Guide_Canva_Manual.docx and G2 uses Instrument_Inspection_Guide_Canva_Solaria-GPT.docx

Ensure all participants report the defects found in the Spreadsheet for recording defects. 



G1 uses Instruments_Spreadsheet_Recording_Defects_Manual.xlsx

G2 uses Instruments_Spreadsheet_Recording_Defects_SolariaGPT.xlsx



After the experiment, G2 must answer the Post-evaluation Questionnaire



Represented by Instrument_Post_Evaluation_Questionnaire.docx






Second Day (Skoob Object)







Switch the groups' treatment: G1 will perform the Solaria-GPT assisted inspection, while G2 will perform the Manual inspection.

Use the Inspection Guide, according to the group (G1 or G2) - G1 uses Instrument_Inspection_Guide_Skoob_Solaria-GPT.docx and G2 uses Instrument_Inspection_Guide_Skoob_Manual.docx

Ensure all participants report the defects found in the Spreadsheet for recording defects. 



G1 uses Instruments_Spreadsheet_Recording_Defects_SolariaGPT.xlsx

G2 uses Instruments_Spreadsheet_Recording_Defects_Manual.xlsx



After the experiment, G1 must answer the Post-evaluation Questionnaire



Represented by Instrument_Post_Evaluation_Questionnaire.docx