Designing a Real-Time Intervention to Address Negative Self-Assessments While Programming

Enrollments in university-level introductory computing courses are skyrocketing [3], but many students struggle in these courses [2]. Recent research suggests that student perceptions of the programming process may contribute to this problem. Students often have inaccurate expectations of programming that may lead them to negatively assess their abilities in response to natural programming moments [6]. For example, many students believe they are doing poorly when they use resources to look up syntax, even though this is considered good practice [7]. This is important because negative self-assessments correlate with lower self-efficacy [6], or one’s belief that they can achieve a goal [1], and students with lower self-efficacy tend to exhibit lower persistence in undergraduate computing programs [9]. In this poster, we present an initial design and evaluation of an intervention that aims to reduce overly negative self-assessments and improve self-efficacy by providing real-time feedback as students program. We created an extension to the jGRASP development environment [4] that delivers feedback messages in response to eight self-assessment moments that can be automatically detected by an expert system developed in prior work [5] (see Figure 1). Informed by recommendations from the feedback literature [8, 10], we developed six messages for each moment that aim to help students develop more accurate expectations by normalizing the moment or highlighting how it could support future growth (see Figure 2). By delivering this feedback automatically, in real-time, and in the context of the task, this intervention aims to address negative self-assessments as they occur. This approach has been successful in other domains [8, 10] and allows us to provide individual feedback at scale, which is particularly challenging as course enrollments grow [3, 11]. We conducted a formative user study with 10 CS1 and 11 CS2 students to understand how they perceived the intervention and which feedback messages they preferred, with the goal of informing future design iterations. First, participants completed a modified version of the survey from [6] to measure their self-efficacy and self-assessments; this served as a pretest. Then, they worked on a programming problem with the intervention for twenty minutes. Finally, participants completed the same survey as a posttest and we interviewed students about their reactions to the intervention as they watched a video of their session. The pretest results showed that many participants do not negatively self-assess in response to these eight programming moments, which is surprising since previous research with other populations has found that negative self-assessments are common [6]. Our preliminary analysis indicates that some participants found the messages reassuring and timely while others found them unhelpful. Participants expressed preferences for some message designs over others, and overall the feedback resonated most with participants who had more negative self-assessments or struggled more on the programming problem. Based on this feedback, we are refining the intervention and collecting data with other populations ahead of a summative evaluation to measure the intervention’s impact on self-assessments and self-efficacy.