# Improving Multi-Instance Multi-Label Learning by Extreme Learning Machine
## 4 Performance Evaluation
### Statistical Significance of the Results  
For the purpose of exploring the statistical significance of the results, we performed a nonparametric [PERSON] test followed by a Holm _post hoc_ test, as advised by [PERSON] [45] to statistically compare algorithms on multiple datasets. Thus, the Friedman and the Holm test results are reported, as well. The Friedman test [51] can be used to compare \(k\) algorithms over \(N\) datasets by ranking each algorithm on each dataset separately. The algorithm obtaining the best performance gets the rank of 1, the second best ranks 2, and so on. In case of ties, average ranks are assigned. Then, the average ranks of all algorithms on all datasets is calculated and compared. If the null hypothesis, which is all algorithms are performing equivalently, is rejected under the Friedman test statistic, _post hoc_ tests, such as the Holm test [52], can be used to determine which algorithms perform statistically different. When all classifiers are compared with a control classifier and \(p_{1}{\leq}p_{2}{\leq}\ldots{\leq}p_{k-1}\), [PERSON]'s step-down procedure starts with the most significant \(p\) value. If \(p_{1}\) is below \(\alpha/(k-1)\), the corresponding hypothesis is rejected, and we are allowed to compare \(p_{2}\) to \(\alpha/(k-2)\). If the second hypothesis is rejected, the test proceeds with the third, and so on. As soon as a certain null hypothesis cannot be rejected, all of the remaining hypotheses are retained, as well. In Figure 4a-d, we have conducted a set of experiments to gradually evaluate the effect of each contribution in MIMLELM. That is, we first modify MIMLSVM+ to include the automatic method for \(k\), then use ELM instead of SVM and then include the genetic algorithm-based weights assignment. The effectiveness of each option is gradually tested on four real datasets using four evaluation criteria. In order to further explore if the improvements are significantly different, we performed a Friedman test followed by a Holm _post hoc_ test. In particular, Table 10 shows the rankings of each contribution on each dataset over criterion \(C\). According to the rankings, we computed \(\chi_{F}^{2}=\frac{12\times 4}{4\times 5}\times[(4^{2}+2.25^{2}+2.75^{2}+1^{2})- \frac{4\times 5^{2}}{4}]=11.1\) and \(F_{F}=\frac{3\times 11}{4\times 3-11\times 1}=37\). With four algorithms and four datasets, \(F_{F}\) is distributed according to the \(F\) distribution with \(4-1=3\) and \((4-1)\times(4-1)=9\) degrees of freedom. The critical value of \(F(3,9)\) for \(\alpha=0.05\) is 3.86, so we reject the null-hypothesis. That is, the Friedman test reports a significant difference among the four methods. In what follows, we choose ELM\(+k{+}w\) as the control classifier and proceed with a Holm _post hoc_ test. As shown in Table 11, with \(SE{=}\sqrt{\frac{4\times 5}{6\times 4}}=0.913\), the Holm procedure rejects the first hypothesis, since the corresponding \(p\) value is smaller than the adjusted \(\alpha\). Thus, it is statically believed that our method, _i.e._, ELM\(+k{+}w\), has a significant performance improvement of criterion \(C\) over SVM. The similar cases can be found when the tests are conducted on the other three criteria. Limited by space, we do not show them here. In Tables 2-5, we compared the effectiveness of MIMLSVM+ and MIMLELM with different condition settings on four criteria, where, for a fair performance comparison, MIMLSVM+ is modified to include the automatic method for \(k\) and the genetic algorithm-based weights assignment as  
Figure 8: The efficiency comparison on the Azotobacter vinelandii dataset. (**a**) The comparison of the training time; (**b**) the comparison of the testing time. MIMLELM does. Table 12 shows the rankings of 10 classifiers on each dataset over criterion \(C\).