## 3 Experiment
### Experiment Results
#### 3.3.3 Performance Evaluation  
The recommendation performances of the proposed method were evaluated qualitatively and quantitatively. For the qualitative evaluation, we visualized the original and estimated feedback rating matrices and compared them to show the resemblance. For the quantitative evaluation, the RMSE and MAPE of the proposed and compared methods were evaluated to show the superior performances of the proposed method and the effectiveness of the selective utilization of ordinal and binary ratings. Feedback rating matrices were illustrated as heatmaps in Figure 7. Figures 6(a)-c, respectively represent an original feedback rating matrix, a matrix whose elements belong to the test set were changed to null, and an estimated matrix by using the proposed method. In each graph, \(x\) and \(y\)-axis respectively indicate the users and editing experts in the test set, and a box represents the feedback rating constructed by using the user feedback log given by the corresponding user to the corresponding expert. A box filled with white color indicates null, meaning that the corresponding user did not rate the corresponding expert, and the darker the color of a box is, the greater the corresponding ordinal  
Figure 5: Visualization of the ordinal ratings utilized in the experiments. rating is. Overall, the heatmap in Figure 6(c) is similar to that in Figure 6(a), implying that the proposed method was successful at inferring user opinions. Table 3 and 4 respectively show the recommendation performances of the considered nine methods in terms of RMSE and MAPE and the paired sample t-test results comparing the performances of the methods in terms of \(p\)-value. Among the methods using a single type of user feedback, \(O\) outperformed \(B\). The result was foreseen since the number of non-null elements in the feedback rating matrix of \(O\) was greater than that of \(B\) and ordinal ratings usually contain abundant information compared to binary ratings. Between \(O\), \(B\) and the methods using both ratings, TMFs and \(F\)s, the latter performed much better than the former with the minimum of 34% reduction in RMSE. Particularly, RMSE and MAPE of \(F(c_{o})\), which utilizes both ratings and prefers ordinal ratings, were 0.17% and 4.23% while those of \(O\), which only uses ordinal ratings, were 0.26% and 6.49%, where \(p\)-value between the performances of \(F(c_{o})\) and \(O\) was 0.0052. Since the \(p\)-value was smaller than the significance level of 0.05, it is possible to say that the performance gains are statistically significant. It demonstrates that the utilization of binary ratings in addition to ordinal ratings has a positive effect on the recommendation performances. Moreover, \(F(c_{b})\) performed better than \(F(c_{o})\), implying that substituting ordinal ratings with binary ratings was effective as it reduces the effect of bias inherent in ordinal ratings, and this result was considered statistically rigorous with the \(p\)-value of 0.0044. Figure 6: Recommendation performances of the proposed method with diverse \(\lambda\)s, 0.2, 0.02, and 0.002, and \(k\)s ranging from 10 to 90 at intervals of 10 in terms of RMSE according to the iteration number. It was interesting that the methods emphasizing the negative user opinions (dislike) showed better performances than those emphasizing positive ones (like) or both. \(F(c_{d})\) which substituted