# Improving Multi-Instance Multi-Label Learning by Extreme Learning Machine
## 4 Performance Evaluation
### Statistical Significance of the Results
According to the rankings, we computed \(\chi_{F}^{2}=\frac{12\times 4}{10\times 11}\times[(5.5^{2}+4^{2}+3.5^{2}+3.25^{2}+ 3.5^{2}+6.5^{2}+6.875^{2}+8.625^{2}+7^{2}+6.25^{2})-\frac{10\times 11^{2}}{4}] \approx 13.43\) and \(F_{F}=\frac{3\times 13.43}{4\times 9-13.43}\approx 1.79\). With 10 classifiers and four datasets, \(F_{F}\) is distributed according to the \(F\) distribution with \(10-1=9\) and \((10-1)\times(4-1)=27\) degrees of freedom. The critical value of \(F(9,27)\) for \(\alpha=0.05\) is 2.25. Thus, as expected, we could not reject the null-hypothesis. That is, the Friedman test reports that there is not a significant difference among the ten methods on criterion \(C\). This is because what we proposed in this paper is a framework. Equipped with the framework, the effectiveness of MIML can be improved further no matter whether SVM or ELM is explored. Since ELM is comparable to SVM on effectiveness [6; 32; 37], MIMLELM is certainly comparable to MIMLSVM+ on effectiveness. This confirms the general effectiveness of the proposed framework. Similar cases can be found when the tests are conducted on the other three criteria. Limited by space, we do not show them here. In Figures 5-8, we studied the training time and the testing time of MIMLSVM+ and MIMLELM for the efficiency comparison, respectively. In order to further explore if the differences are significant, we performed a Friedman test followed by a Holm _post hoc_ test. In particular, Table 13 shows the rankings of 10 classifiers on each dataset over training time. According to the rankings, we computed \(\chi_{F}^{2}=\frac{12\times 4}{10\times 11}\times[(8^{2}+5.75^{2}+6.5^{2}+7.7 5^{2}+5.5^{2}+7.25^{2}+8.25^{2}+1.5^{2}+2.75^{2}+1.75^{2})-\frac{10\times 11^{2}}{4}] \approx 26.45\) and \(F_{F}=\frac{3\times 26.45}{4\times 9-26.45}\approx 8.31\). With ten classifiers and four datasets, \(F_{F}\) is distributed according to the \(F\) distribution with \(10-1=9\) and \((10-1)\times(4-1)=27\) degrees of freedom. The critical value of \(F(9,27)\) for \(\alpha=0.05\) is 2.25, so we reject the null-hypothesis. That is, the Friedman test reports a significant difference among the ten methods. In what follows, we choose ELM with \(HN=100\) as the control classifier and proceed with a Holm _post hoc_ test. As shown in Table 14, with \(SE=\sqrt{\frac{10\times 11}{6\times 4}}=2.141\), the Holm procedure rejects the hypotheses from the first to the fourth since the corresponding \(p\)-values are smaller than the adjusted \(\alpha\)'s. Thus, it is statically believed that MIMLELM with \(HN=100\) has a significant performance improvement of training over most of the MIMLSVM+ classifiers. Similarly, Table 15 shows the rankings of 10 classifiers on each dataset over testing time. According to the rankings, we computed \(\chi_{F}^{2}=\frac{12\times 4}{10\times 11}\times[(7.5^{2}+6.875^{2}+6.125^{2}+ 6.25^{2}+6.5^{2}+7.5^{2}+8.25^{2}+2.125^{2}+1.75^{2}+2.125^{2})-\frac{10\times 11 ^{2}}{4}]\approx 24.55\) and \(F_{F}=\frac{3\times 24.55}{4\times 9-24.55}\approx 6.43\). With ten classifiers and four datasets, \(F_{F}\) is distributed according to the \(F\) distribution with \(10-1=9\) and \((10-1)\times(4-1)=27\) degrees of freedom. The critical value of \(F(9,27)\) for \(\alpha=0.05\) is 2.25, so we reject the null-hypothesis. That is, the Friedman test reports a significant difference among the ten methods. In what follows, we choose ELM with \(HN=200\) as the control classifier and proceed with a Holm _post hoc_ test. As shown in Table 16, with \(SE=\sqrt{\frac{10\times 11}{6\times 4}}=2.141\), the Holm procedure rejects the hypotheses from the first to the third since the corresponding \(p\)-values are smaller than