## 4 Discussion
### RQ2: Does WOJR Improve the Number of Correct Answers in Students' Assignments? Our analysis included all the students enrolled in the course, even those who did not use WOJR, to minimize the effect of two biases. The first bias is that WOJR is more valuable \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{3}{*}{**Assignment**} & \multirow{3}{*}{
\begin{tabular}{c} **TF-IDF** \\ **(AOJ)** \\ \end{tabular} } & \multicolumn{3}{c}{**Average**} & \multicolumn{3}{c}{**Max**} \\  & & **TF-IDF** & **JPlag** & **JPlag** & **(All)** & **TF-IDF** & **TF-IDF** & **JPlag** \\  & & **(All)** & **(AOJ)** & **(AOJ)** & **(AJ)** & **(AJ)** & **(AJ)** & **JPlag** \\ \hline General 1 & No & No & Nothing & Nothing & Yes & Yes & Nothing & Nothing \\ General 2 & Yes & Yes & Nothing & No & Nothing & Yes & Yes & No \\ General 3 & Yes & Yes & Nothing & Nothing & Yes & Yes & Yes & Yes \\ General 4 & Yes & Yes & Yes & Yes & Yes & Yes & Yes & Yes \\ General 5 & Yes & Yes & Nothing & Nothing & Yes & Yes & Nothing & Nothing \\ General 6 & Yes & Yes & Nothing & Nothing & Yes & No & No & No \\ General 7 & Yes & Yes & Yes & Yes & Yes & Yes & Yes & Yes \\ General 8 & Nothing & Nothing & Nothing & Nothing & Nothing & No & Nothing & Yes \\ General 9 & Nothing & Nothing & Nothing & Nothing & No & Nothing & Nothing & Nothing \\ Tree 1 & Nothing & Nothing & Nothing & Nothing & No & Nothing & No & Nothing \\ Tree 2 & No & No & No & Nothing & Nothing & Nothing & No & No \\ Tree 3 & Nothing & Nothing & Nothing & Nothing & Yes & Yes & Nothing & Nothing \\ Graph 1 & Nothing & Nothing & Nothing & Nothing & Nothing & Nothing & Nothing & Nothing \\ Graph 2 & Yes & Yes & Yes & Yes & Yes & No & Yes & Yes \\ Graph 3 & Yes & Yes & Yes & Yes & Yes & Yes & Yes & Yes \\ \hline Number of Yes & 8 & 8 & 4 & 4 & 9 & 7 & 6 & 6 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: A summary indicating whether each similar problem with reference code is helpful to solve the assignment problem or not. Yes and No indicate helpful and not helpful, respectively. Nothing indicates that no valid similar problem exists. The highlighted cells indicate that only one of TF-IDF or JPlag found helpful similar problems among the results of the average and maximum operators. for students who cannot solve the problems by themselves. We speculated that students who did not use WOJR would tend to have low skills and low academic performance with respect to the first bias. The second bias is that enthusiastic students are more likely to try using new systems and methods, including WOJR. We hypothesized that such students would have high skills and academic performance due to their high learning motivation. Consequently, we included all enrolled students to eliminate these two biases. Compared to the non-intervention year, the average number of correct answers increased by about 0.4 for the nine assignments in the intervention year, which is about a 4.4% improvement in the correct answer rate. There is a statistically significant difference between the non-intervention year and intervention year, indicating that WOJR improved the number of correct answers (RQ2). The percentage of students who answered all the problems correctly increased by about 11.5%. The value of 11.5% is relatively large. In general, improving the number of correct answers by providing similar problems and answers does not necessarily mean an educational effect because similar problems may work just as hints. However, we consider this improvement an educational effect for the following three reasons. (1) The selected reference code differed from the answer code of the assignment. Therefore, students must solve assignments by understanding similar problems and reference code, and then applying the knowledge to the assignment. (2) Difficult problems still make students think deeply even if similar problems and reference code are provided.