Supplemental for ChatGPT User Study - GenAI in Learning Software Engineering
Creators: name: Anonymous
Publisher: name: Zenodo
Year: 2023

description: <strong>Supplemental Material</strong> Contents: · <strong>1-Demographic Information.xlsx: </strong>contains the demographic information of the participants in the study. <strong>· </strong><strong>2-Forms.zip: </strong>contains the forms and questionnaires used to collect data for the experiment<strong>: </strong>demographic form, pre-study, post-study, and AAR/AI questionnaires. · <strong>3-GitHub-Repository.zip: </strong>a copy of the GitHub repository used in the study. · <strong>4-Tutorial Scripts.zip:</strong> script used in the experiment with the groups to be consistent with all participants. · <strong>5-Logs-Rubric-Grades.zip: </strong>contains the participant data log (commit and PR), rubric for grading submissions, and grades. · <strong>6-RQ1-Data-and-Analysis.zip: </strong>contains the data and analysis with respect to RQ1. · <strong>7-RQ2-Data-and-Analysis.zip: </strong>contains the data and analysis with respect to RQ2. <strong> </strong> <strong>2. Forms.zip</strong> The forms zip contains the following files: · <strong>Demographics.pdf: </strong>a form used to collect demographic information from participants before the study. · <strong>Control Pre-Study Questionnaire.pdf: </strong>Pre study questionnaire control group (Self-Efficacy Questionnaire) · <strong>Control Post-Study Questionnaire.pdf: </strong>Post study questionnaire control group (NASA-TLX, Self-Efficacy Questionnaire) · <strong>Treatment - AAR_AI task.pdf:</strong> Pre and Post task AAR/AI questionnaire for experimental group. · <strong>Experimental Pre-Study Questionnaire.pdf:</strong> Pre study questionnaire experimental group (Self-Efficacy Questionnaire, Question for Familiarity with AI) · <strong>Experimental-Post Study Questionnaire.pdf:</strong> Post study questionnaire experimental group (AAR/AI step 7, Continuance Intention, NASA-TLX, HAI Guideline Questions, Self-Efficacy Questionnaire) <strong>3-GitHub-Repository.zip</strong> The GitHub repository used in the study: contains the main.py code file and the Readme.md file (having the written instructions for the participants). <strong>4-Tutorial Scripts.zip</strong> Contains: · <strong>Control-Script.pdf: </strong>Script for the control group. · <strong>Experimental-Script.pdf: </strong>Script for the experimental group. <strong>5<strong>-Logs-Rubric-Grades.zip</strong></strong> · <strong>rubric.pdf: </strong>Created rubric for grading task performance. · <strong>GitHub-Task3-Log.xlsx: </strong>File containing the data regarding the status of commit made and PR raised for each participant. · <strong>grades.xlsx: </strong>Detailed grades for each participant in experimental (treatment) and control groups. <strong>6-RQ1-Data-and-Analysis.zip</strong> <strong>Note: </strong>The term ‘treatment’ has been used in the files of this folder to represent the experimental group: participants using ChatGPT for the tasks. · <strong>NASA TLX: </strong>folder containing the participant data (TLX.xlsx), code for statistical analysis (Stat-TLX.py) and statistical reports (analysis-TLX.csv). · <strong>Task Performance: </strong>folder containing the participant data (grades.xlsx &amp; Scores.xlsx(overall grade)), code for statistical analysis (Stat-Correctness.py) and statistical reports (analysis.csv). · <strong>Self-Efficacy: </strong>folder containing: o <strong>Self-Efficacy-detailed.xlsx</strong>:<strong> </strong>participant data o <strong>Paired Stats: </strong>folder containing data (Total Self Efficacy.csv), code for statistical analysis(paired-stats.py), and statistical reports (analysis.csv). o <strong>Box plot: </strong>folder containing the code for generating the box plot and its output. · <strong>Continuance Intention.xlsx: </strong>participant data (experimental) for continuance intention of ChatGPT. · <strong>Stat-Table-H1-2-Paper.xlsx: </strong>Statistics table for NASA TLX and task performance as presented in the paper. <strong>7-RQ2-Data-and-Analysis.zip</strong> · <strong>AAR_AI-Responses.xlsx</strong>: AAR/AI responses filled by participants in experimental group. · <strong>Quotation Manager-Faults&amp;Conseq.xlsx:</strong> Contains the quotations from AAR/AI responses along with corresponding codes. Also contains the quotes that link faults to consequences in a separate sheet. · <strong>Codebook.xlsx: </strong>The final codebook (faults and consequences). · <strong>HAI-data.xlsx: </strong>Contains the reported guideline violations along with disaggregated analysis (grouped by gender). · <strong>Likert Plot-HAI: </strong>folder contains the code for generating the Likert plot figure presented in the paper.