DOI: 10.5281/zenodo.10912669
title: SelfCode 2.0: Annotated Corpus of Student Self-Explanations to Introductory JAVA Programs in Computer Science

description: Dataset Description: This dataset was collected during a lab study conducted in Spring 2022 for introductory JAVA programming. Students had to provide line by explanations to four JAVA programs in the experimental condition of the study. The JAVA Programs were selected from the examples made available in the PCEX Worked Examples interface. The explanations collected were then split by the number of attempts. Students could attempt twice based on the feedback provided using the the PCEX interface and in their third attempt they filled in the blanks to complete an explanation to the particular line of code. In this dataset, we only have the annotated examples of explanations provided by students. The explanations were annotated on their correctness (binary rating 0 or 1), completeness (binary rating 0 or 1) and similarity (rating scale 1 to 5).

Correctness: Given the line of code and context of the line in the program, if the student explanation covers **only** the topics relevant to the line of code

Completeness: Given the line of code and context of the line in the program, if the student explanation covers **all** the topics relevant to the line of code

Similarity: Given the line of code, the context of the line in the program and an expert explanation to the line of code, the metric compares the similarity on a rating scale from 1 to 5, defined in the following manner:

1 - expert and student explanations are very different,

2 -- expert and student explanations are somewhat alike, but there are major differences in the concepts / topics explained

3 -- expert and student explanations are similar but there are differences in the concepts / topics explained

4 -- expert and student explanations are similar and have few differences in the concepts / topics explained

5 -- expert and student explanations are very similar.

 

Overall 3000 single attempts (corresponding to 40 student explanation submission) were annotated against different various expert explanation pairs.

 

Dataset Summary:

Explanation Type N DefinitionExperts 2 Source Code Line-by-Line Explanations by ExpertsStudents 60 (annotated 40) Source Code Line-by-Line Explanations by Students




COUNT of std_sent_count
 
 
 
 
 
 
 


std_sent_count
1
2
3
4
5
6
Grand Total


1
1854
367
245
107
34
33
2640


2
222
46
40
12
6
6
332


3
21
5
5
5
1
2
39


4
2
1
2
3
 
 
8


Grand Total
2099
419
292
127
41
41
3019




 

Sample Data:

Program: PointTester; Line number: 12; Line code: private int y;Expert1: Every object of the Point class will have its own y-coordinate. Therefore, weneed to declare an instance variable for the class to store the y-coordinate of the point.We declare it as int because we want to have integer coordinates for the point. Notethat an instance variable is a variable defined in a class, for which each instantiatedobject of the class has a separate copy, or instance.Expert2: The instance variables are declared as private to prevent direct access tothem from outside the class. In this way, no unexpected modifications to a Pointobject’s data are possible.Student1: initialize a private value inside the point class with no value yetStudent2: Declares the private int variable y.Student3: Creates a private int that can only be accessed by class Point called int y...Student59: private variable used to store the value entered into the value of the ycoordinate

 

Kappa Scores:








Round



Row Numbers



Correctness Rating Agreement %age



Correctness Rating Kappa



Sufficiency Rating Agreement %age



Sufficiency Rating Kappa





1



1000 - 1432 



92.9



0.365



0.708



-0.0123





2



1432 - 1864



94.2



0.263



77.6



0.329





3



1864 – 1964 



75.3



0



70.3



0.299





4



1964 -- 2064 



86



0.108



74.7



0.275





5



2064 – 2264



95.5



-0.0158



81.5



0.312





6



2264 – 2464



83.5



0.039



86.5



0.648





7



2464 – 2864



92



0.103



74.5



0.188





8



2864 -- 3005



86.5



-0.026



72.3



0.117






 

 

Citation Format:If using this dataset in your project please cite:

Lekshmi-Narayanan, A.-B., Chapagain, J., Brusilovsky, P., & Rus, V. (2023). SelfCode 2.0: Annotated Corpus of Student Self-Explanations to Introductory JAVA Programs in Computer Science [Data set]. Zenodo. https://doi.org/10.5281/zenodo.10912669

Acknowledgements:This project was funded as a part of the NSF AWARD # 1822752