DOI: 10.5281/zenodo.17429846
title: Replication package: ChatGPT: Friend or Foe When Comprehending and Changing Unfamiliar Code

description: README



This is a replication package/supplementary data package for the study ChatGPT: Friend or Foe When Comprehending and Changing Unfamiliar Code.




 




This data package is intended to be used as a companion to the study paper itself.






If you are interested in running a similar study, you will find the files we used to administer this study in 0-study_setup/

If you are interested in the raw data we collected for this study, we have shared anonymized information in 1-collected_data/

If you are interested in how we performed our analysis, you will find the relevant files in 2-data_analysis/



Contents

 

.
├── 0-study_setup
│   ├── demographics_survey-questions.pdf
│   ├── experimenter_handbook.pdf
│   ├── field_notes_template.pdf
│   └── post_task_questionnaire-questions.pdf
├── 1-collected_data
│   ├── chatgpt_logs
│   │   ├── p10_chat.html
│   │   ├── p10_conversations.json
│   │   ├── p11_chat.html
│   │   ├── p11_conversations.json
│   │   ├── p2_chat.html
│   │   ├── p2_conversations.json
│   │   ├── p3_chat.html
│   │   ├── p3_conversations.json
│   │   ├── p7_chat.html
│   │   └── p7_conversations.json
│   ├── post_task_questionnaire-responses.csv
│   ├── sketches
│   │   ├── P1.svg
│   │   ├── P10.svg
│   │   ├── P11.svg
│   │   ├── P2.svg
│   │   ├── P3.svg
│   │   ├── P4.svg
│   │   ├── P5.svg
│   │   ├── P6.svg
│   │   ├── P7.svg
│   │   └── P9.svg
│   ├── task-starter-template-ai.zip
│   └── task-starter-template-nonai.zip
├── 2-data_analysis
│   ├── codebook.csv
│   ├── scripts
│   │   ├── analyze_vscode_logs.py
│   │   └── polya_state_timeline.tgz
│   └── vignettes.md
└── README.md

7 directories, 32 files

 

 

Explanation of files

0-study_setup





demographics_survey-questions.pdf



Reproduced list of questions asked to participants to screen them and collect general information before they booked an in-person study session



experimenter_handbook.pdf



Structured handbook for the scheduled physical study sessions that were arranged with participants after they completed their demographics survey.

During the sessions, participants were required to bring their own device. They were also asked to come prepared with: (1) VS Code and Zoom already installed on their device, and (2) a charger for their laptop

Note that although the participant study sessions had durations of three hours, most participants ran out of time to complete their task, and therefore needed to be stopped by the experimenter so they could move onto the post-task questionnaire.



field_notes_template.pdf:



Template used by the experimenter running the physical study sessions to jot notes while observing each participant complete their task.



post_task_questionnaire-questions



Reproduced list of questions that were administered to participants at the conclusion of their task, or when only





1-collected_data





chatgpt_logs/



For all ChatGPT participants, there are two files:



p#_chat.html - an HTML export of their ChatGPT conversation(s)

p#_conversations.json - a JSON export of their ChatGPT conversation(s)



All participants used the same ChatGPT account. Between each participant session the account data was exported, and all conversations and memories were cleared from the account. If a system prompt was created by the participant, this was also reset to the default.



post_task_questionnaire-responses.csv



The data collected from participants' post-task questionnaires

NOTE: Participants were not allowed access to any resources (or codebase) when completing the post-task questionnaire.



sketches/



Sketches drawn during the post-task questionnaire by participants, prompted by the wording: Sketch a component diagram listing all components you remember that were relevant in completing the task and their relationships (see post_task_questionnaire-questions.pdf)

All sketches have been reproduced electronically to maintain participant confidentiality

NOTE: Participants were not allowed access to any resources (or codebase) when sketching their component diagram.



task-starter-template-ai.zip and task-starter-template-nonai.zip



GitHub repos that was cloned by AI and nonAI participants respectively into GitHub Codespaces, using a general GitHub account to maintain participant confidentality, when they were to begin their task.





2-data_analysis





codebook.csv



The qualitative codebook used for this study. Codes are broken into two sections in the hierarchy:



Polya codes (also known as "PS Codes") - used to identify the problem solving state of participants throughout their task



Note the four codes (EMO, WHA, DOU, and DEL) in the "Uncategorized" section, which we occasionally applied during the coding process in combination with another PS Code. We did not use these in our analysis, but found them similarly useful as to the screen/modes codes to assist in indexing our data.



Screen/modes codes - used to contextualize what participants are currently "doing" in their task



Codes were applied by researchers manually in Excel, by specifying the start and end points of every interval in the screen recording where a qualitative code applied, and noting down the qualitative code for that section



scripts/



analyze_vscode_logs.py - script used to analyze the VS Code logs we collected from participants

polya_state_timeline.tgz - code used to generate the condensed state diagrams of Polya codes - created with Observable



vignettes.md



An extended collection of the vignettes that are included in our paper to narratively explain how participants found themselves stuck while completing their tasks