## 3 Evaluation
### Similarity Metrics and Problem Repositories  
TF-IDF considers the similarity of token sequences, but it does not consider structural similarity. The structural similarity may also be effective in finding similar problems and reference code. JPlag is the most famous plagiarism detection tool that considers the structural similarity, and it can show similarity scores [6; 12]. Thus, we compared TF-IDF with JPlag in the second experiment. The size of a set of problem repositories may affect WOJR performance because WOJR cannot recommend similar problems and reference code that do not exist in the problem repositories. We conducted the first WOJR experiment with AOJ, but AtCoder is larger than AOJ, and the submitted code is also available in AtCoder. Thus, we also compared AOJ with AOJ and AtCoder. To evaluate how the similarity metrics and the sizes of problem repositories affect WOJR performance, we implemented the two similarity metrics (TF-IDF and JPlag) for Python language. We also made WOJR support the two sets of problem repositories (AOJ and AOJ + AtCoder). WOJR supports the two aggregation operators (maximum and average) to calculate a similarity score of a similar problem from similarity scores between answer code and reference code candidates. We selected the top five similar problems with reference code with respect to each combination of the two similarity metrics, the two sets of problem repositories, and the two aggregation operators. Note that we excluded too similar (almost same) problems from the top five similar problems because students can change the reference code of the problem to the model answer code easily without thinking. We selected all the nine problems that we used in the intervention experiment and added the six problems that we did not use in the intervention experiment but the professor provided after the interventions in his lecture. Since we expected that TF-IDF would not be suitable to find similar code that had few common words but a similar structure, we selected the six problems that asked about tree and graph algorithms. Contrary to TF-IDF, we expected that JPlag could find structurally similar code for tree and graph algorithms. Table 2 shows the algorithms (or data structures) that selected assignment problems require students to implement. We recruited a programming lecturer who is an expert in algorithms and data structures. The authors and the lecturer judged whether or not each similar problem with reference code recommended by each combination was valid for students with the following standard. For example, General 1 requires students to implement a stack. If problems require other data structures, such as a pure array or a queue, the problems are not similar to General 1. Figure 11: Fragment of reference code of similar problem 2 (selecting the minimum value from the given array) written in C.  
Figure 10: Fragment of model answer code of example assignment 2 (selecting the pair with the smallest difference from the given array) written in C.  
Both an assignment problem and a similar problem require the same algorithm or data structure. * A similar problem is simple and easy to understand. It is not too difficult in comparison to an assignment problem. Table 3 shows the numbers of valid similar problems with reference code. The authors and the lecturer agreed with the results. The maximum value of a cell is five because we selected the top five similar problems. The total number of similar problems with reference code including duplication is 600 (15 \(\times\) 5 \(\times\) 2 \(\times\) 2 \(\times\) 2) since there are combinations of the 15 problems, the top 5 similar problems, the 2 similarity metrics (TF-IDF and JPlag), the 2 sets of problem repositories (AOJ and AOJ + AtCoder), and the 2 aggregation operators (maximum and average). Overall, both TF-IDF with only AOJ using the average operator and TF-IDF with AOJ and AtCode using the maximum operator are the most capable of providing similar problems.