Supplemental for ChatGPT User Study - GenAI in Learning Software Engineering
Creators: name: Choudhuri, Rudrajit
name: Liu, Dylan
name: Steinmacher, Igor
name: Gerosa, Marco
name: Sarma, Anita
Publisher: name: Zenodo
Year: 2023

description: Supplemental Material
Contents:
·       1-Demographic Information.xlsx: contains the demographic information of the participants in the study.
·       2-Forms.zip: contains the forms and questionnaires used to collect data for the experiment: demographic form, pre-study, post-study, and AAR/AI questionnaires.
·       3-GitHub-Repository.zip: a copy of the GitHub repository used in the study.
·       4-Tutorial Scripts.zip: script used in the experiment with the groups to be consistent with all participants.
·       5-Logs-Rubric-Grades.zip: contains the participant data log (commit and PR), rubric for grading submissions, and grades.
·       6-RQ1-Data-and-Analysis.zip: contains the data and analysis with respect to RQ1.
·       7-RQ2-Data-and-Analysis.zip: contains the data and analysis with respect to RQ2.
·       8-Participant Prompts.xlsx: contains the experimental group participant prompts with ChatGPT.
 
2. Forms.zip
The forms zip contains the following files:
·       Demographics.pdf: a form used to collect demographic information from participants before the study.
·       Control Pre-Study Questionnaire.pdf: Pre study questionnaire control group (Self-Efficacy Questionnaire)
·       Control Post-Study Questionnaire.pdf: Post study questionnaire control group (NASA-TLX, Self-Efficacy Questionnaire)
·       Treatment - AAR_AI task.pdf: Pre and Post task AAR/AI questionnaire for experimental group.
·       Experimental Pre-Study Questionnaire.pdf: Pre study questionnaire experimental group (Self-Efficacy Questionnaire, Question for Familiarity with AI)
·       Experimental-Post Study Questionnaire.pdf: Post study questionnaire experimental group (AAR/AI step 7, Continuance Intention, NASA-TLX, HAI Guideline Questions, Self-Efficacy Questionnaire)
3-GitHub-Repository.zip
The GitHub repository used in the study: contains the main.py code file and the Readme.md file (having the written instructions for the participants).
4-Tutorial Scripts.zip
Contains:
·       Control-Script.pdf: Script for the control group.
·       Experimental-Script.pdf: Script for the experimental group.
5-Logs-Rubric-Grades.zip
·       rubric.pdf: Created rubric for grading task performance.
·       GitHub-Task3-Log.xlsx: File containing the data regarding the status of commit made and PR raised for each participant.
·       grades.xlsx: Detailed grades for each participant in experimental (treatment) and control groups.
6-RQ1-Data-and-Analysis.zip
Note: The term 'treatment' has been used in the files of this folder to represent the experimental group: participants using ChatGPT for the tasks.
·       NASA TLX: folder containing the participant data (TLX.xlsx), code for statistical analysis (Stat-TLX.py) and statistical reports (analysis-TLX.csv).
·       Task Performance: folder containing the participant data (grades.xlsx & Scores.xlsx(overall grade)), code for statistical analysis (Stat-Correctness.py) and statistical reports (analysis.csv).
·       Self-Efficacy: folder containing:
              o   Self-Efficacy-detailed.xlsx: participant data
              o   Paired Stats: folder containing data (Total Self Efficacy.csv), code for statistical analysis(paired-stats.py),
                   and statistical reports (analysis.csv).
              o   Box plot: folder containing the code for generating the box plot and its output.
·       Continuance Intention.xlsx: participant data (experimental) for continuance intention of ChatGPT.
·       Stat-Table-H1-2-Paper.xlsx: Statistics table for NASA TLX and task performance as presented in the paper.
7-RQ2-Data-and-Analysis.zip
·       AAR_AI-Responses.xlsx: AAR/AI responses filled by participants in experimental group.
·       Quotation Manager-Faults&Conseq.xlsx: Contains the quotations from AAR/AI responses along with corresponding codes. Also contains the quotes that link faults to consequences in a separate sheet.
·       Codebook.xlsx: The final codebook (faults and consequences).
·       HAI-data.xlsx: Contains the reported guideline violations along with disaggregated analysis (grouped by gender).
·       Likert Plot-HAI: folder contains the code for generating the Likert plot figure presented in the paper.