# Mining on Students' Execution Logs and Repairing Compilation Errors Based on Deep Learning
## 3 Mining on Students' Execution Logs
## 6 Experimental Evaluation

In this study, we intend to answer the following four research questions. 1. RQ1: How does our model perform compared to other repair models? 2. RQ2: Can the dataset generated by the RS help the model with program repair? 3. RQ3: Do the different modules of the model all have a positive impact on the repair effectiveness? 4. RQ4: How do models with different architectures perform on different categories of compiler errors? ### Experimental Datasets

During the program repair example generation phase, two types of datasets were used: SUES-COJ and TEGCER (proposed by [PERSON] et al. [16]). The SUES-COJ dataset consists of programs submitted by undergraduate students from Shanghai University of Engineering Science for a C programming course. We adopted the aforementioned strategies to extract reasonable program pairs from execution logs. The dataset comprises a total of 4447 program pairs after the process of mining and filtering. It should be noted that some of the code samples in the dataset contain multiple line modifications. The length of the code ranges from 10 to 30 lines. The TEGCER dataset is composed of programs submitted online by over 400 undergraduate students from the Indian Institute of Technology during the first semester of the 2015-2016 academic year for an introductory C programming course. The dataset consists of a total of 23,275 code pairs. Each code pair includes the original code statement and the modified code statement, with only one change per code pair. The dataset has undergone preprocessing and filtering operations, restricting the code length to 40 lines or less. Additionally, incorrect repair data have been removed. Ultimately, 21,994 code pairs were retained as the training dataset for the model. In addition to the above dataset, we also utilized the program perturbation process described in the reference [13] to generate a large number of repair examples as a training set. This process involved random perturbations, including modules for delimiter, type name, keyword, variable, and variable definition. We randomly applied these modules to disrupt the correct code and paired it with the original correct code, forming perturbation-generated code error-repair sample pairs. By comparing this perturbation dataset with the dataset generated through our simulation, we were able to gather valuable insights and evaluate the effectiveness of the RS approach. During the program repair phase, we also utilized the DeepFix dataset [10]. This dataset was sourced from student-submitted programming task codes and comprises a total of 37,415 compiled programs and 6971 programs that could not be compiled. The dataset contains a significant number of errors spanning multiple lines. Since the DeepFix dataset does not provide paired repair examples, we used the perturbation-generated programs as the training data and the original erroneous programs as the test set. This approach allowed us to train the model on the perturbed programs and evaluate its performance on the original erroneous programs. ### Error Classification

By analyzing the data from the student error programs, we classified common errors based on the error content and compiler feedback information. The specific categories can be found in Figure 3:0