# Improving Multi-Instance Multi-Label Learning by Extreme Learning Machine
## 4 Performance Evaluation
### Evaluation Criteria  
In multi-label learning, each object may have several labels simultaneously. The commonly-used evaluation criteria, such as accuracy, precision and recall, are not suitable in this case. In this paper, four popular multi-label learning evaluation criteria, _i.e._, one-error (\(OE\)), coverage (\(Co\)), ranking loss (\(RL\)) and average precision (\(AP\)), are used to measure the performance of the proposed algorithm. Given a test dataset \(S=\{(X_{1},Y_{1}),(X_{2},Y_{2}),\ldots,(X_{p},Y_{p})\}\), the four criteria are defined as below, where  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Data Set**} & \multirow{2}{*}{**\# of Objects**} & \multirow{2}{*}{**\# of Classes**} & **Instances per Bag** & **Labels per Example** \\  & & & **(Mean\(\pm\) std.)** & **(Mean \(\pm\) std.)** \\ \hline Image & 2000 & 5 & \(1.36\pm 0.54\) & \(1.24\pm 0.46\) \\ Text & 2000 & 7 & \(1.64\pm 0.73\) & \(1.15\pm 0.37\) \\ Geobacter sulfurreducens & 397 & 320 & \(3.20\pm 1.21\) & \(3.14\pm 3.33\) \\ Azotobacter vinelandii & 407 & 340 & \(3.07\pm 1.16\) & \(4.00\pm 6.97\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: The information of the datasets. std.: standard deviation.  
\(h(X_{i})\) returns a set of proper labels of \(X_{i}\), \(h(X_{i},y)\) returns a real-value indicating the confidence for \(y\) to be a proper label of \(X_{i}\) and \(rank^{h}(X_{i},y)\) returns the rank of \(y\) derived from \(h(X_{i},y)\).
* \(\textit{one-error}_{S}(h)\ =\ \frac{1}{p}\sum\limits_{i=1}^{p}\left[\left[\arg \max\limits_{y\in Y}h(X_{i},y)\right]\
ot\in Y_{i}\right]\). The one-error evaluates how many times the top-ranked label is not a proper label of the object. The performance is perfect when \(\textit{one-error}_{S}(h)=0\); the smaller the value of \(\textit{one-error}_{S}(h)\), the better the performance of \(h\).
* \(\textit{coverage}_{S}(h)\ =\ \frac{1}{p}\sum\limits_{j\in Y_{i}}^{p}\max \limits_{y\in Y_{i}}rank^{h}(X_{i},y)-1\). The coverage evaluates how far it is needed, on the average, to go down the list of labels in order to cover all of the proper labels of the object. It is loosely related to precision at the level of perfect recall. The smaller the value of \(\textit{coverage}_{S}(h)\), the better the performance of \(h\).
* \(\textit{rloss}_{S}(h)\ =\ \frac{1}{p}\sum\limits_{i=1}^{p}\frac{1}{\|Y_{i}\| \|Y_{i}\|}\left|\{(y_{1},y_{2})|h(X_{i},y_{1})\leq h(X_{i},y_{2}),(y_{1},y_{ 2})\in Y_{i}\times\overline{Y_{i}}\}\right|\), where \(Y_{i}\) denotes the complementary set of \(Y_{i}\) in \(Y\). The ranking loss evaluates the average fraction of label pairs that are misordered for the object. The performance is perfect when \(\textit{rloss}_{S}(h)=0\); the smaller the value of \(\textit{rloss}_{S}(h)\), the better the performance of \(h\).
* \(\textit{avgprec}_{S}(h)\ =\ \frac{1}{p}\sum\limits_{i=1}^{p}\frac{1}{\|Y_{i}\|} \sum\limits_{y\in Y_{i}}\frac{\left|\{y^{\prime}|rank^{h}(X_{i},y^{\prime}) \leq rank^{h}(X_{i},y),y^{\prime}\in Y_{i}\}\right|}{rank^{h}(X_{i},y)}\). The average precision evaluates the average fraction of proper labels ranked above a particular label \(y\)\(\in\)\(Y_{i}\). The performance is perfect when \(\textit{avgprec}_{S}(h)=1\); the larger the value of \(\textit{avgprec}_{S}(h)\), the better the performance of \(h\).