# Which, When, and How: Hierarchical Clustering with Human-Machine Cooperation
## 6 Experiments
### Evaluation Results (LFW Data Set)  
The evaluation results are shown in Tables 3 and 4. All the algorithms stop iterations when there are two clusters remaining (the true number of clusters). It can be seen that all algorithms have relatively low resulting mutual information, since the face images have complex variations in pose and illumination. For TAHC, it even makes a seriously incorrect decision at the last step, leading to a performance degradation. For the \"which\" problem, HMCAHC is still better than HMCAHC-which, since a random question does not bring too much useful information for the machine. For the \"when\" problem, HMCAHC-when1 and HMCAHC-when2 fail to reserve enough questions for the hard decisions. The error propagation is disastrous, and thus shuts down the performance of the HMCAHC-when1 and the HMCAHC-when2 in the later clustering stage (especially for HMCAHC-when2 that makes an incorrect decision at the last step). For the \"how\" problem, the trade-off between the clustering accuracy and the time consumption is still observed by HMCAHC-how1 and HMCAHC-how2. Our back-tracing scheme balances this trade-off. Meanwhile, both PCMK-Means and Active PCK-Means perform more poorly than HMCAMC. This is because they fail to reserve the question operations (i.e., constraints) for really hard decisions. For further analysis, Figure 7(a) shows that the machine's certainty has a good estimation of the true minimum purity during the clustering-building process. Figure 7(b) shows the relationship between the available number of question operations and the resulting mutual information of HMCAHC. Although a few questions can significantly improve the clustering results, too many question operations are unnecessary due to the marginal improvement.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Evaluation** & **Clustering Steps Where** \\
**Results** & **Questions Are Asked** \\ \hline HMCAHC & 177, 185, 186, 187, 188, 198 \\ \hline HMCAHC-which & 177, 185, 186, 187, 188, 198 \\ \hline HMCAHC-when1 & 12, 47, 64, 79, 86, 198 \\ \hline HMCAHC-when2 & 28, 57, 85, 114, 142, 171 \\ \hline HMCAHC-how1 & 177, 185, 186, 187, 188, 198 \\ \hline HMCAHC-how2 & 177, 185, 186, 187, 188, 198 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Questions asked in the LFW Data Set.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**Evaluation** & \multicolumn{6}{c}{**The Resulting Mutual Information for Algorithms at Given Clustering Steps**} \\ \cline{2-11}
**Results** & **1 st** & **50 th** & **100 th** & **150 th** & **170 th** & **180 th** & **190 th** & **192 th** & **194 th** & **196 th** & **198 th** \\ \hline HMCAHC & 0.23 & 0.25 & 0.27 & 0.30 & 0.32 & 0.34 & 0.37 & 0.40 & 0.39 & 0.41 & 0.43 \\ \hline HMCAHC-which & 0.23 & 0.25 & 0.26 & 0.30 & 0.31 & 0.34 & 0.43 & 0.37 & 0.34 & 0.34 & 0.37 \\ \hline HMCAHC-when1 & 0.23 & 0.25 & 0.27 & 0.30 & 0.32 & 0.34 & 0.34 & 0.31 & 0.32 & 0.33 & 0.32 \\ \hline HMCAHC-when2 & 0.23 & 0.25 & 0.27 & 0.30 & 0.31 & 0.34 & 0.34 & 0.31 & 0.34 & 0.33 & 0.16 \\ \hline HMCAHC-how1 & 0.23 & 0.25 & 0.27 & 0.30 & 0.32 & 0.34 & 0.43 & 0.37 & 0.35 & 0.34 & 0.36 \\ \hline HMCAHC-how2 & 0.23 & 0.25 & 0.27 & 0.30 & 0.32 & 0.34 & 0.37 & 0.40 & 0.41 & 0.42 & 0.45 \\ \hline TAHC & 0.23 & 0.25 & 0.27 & 0.30 & 0.32 & 0.34 & 0.34 & 0.31 & 0.32 & 0.33 & 0.16 \\ \hline K-Means & \multicolumn{6}{c}{The mutual information of the final result is 0.30} \\ \hline PCMK-Means & \multicolumn{6}{c}{The mutual information of the final result is 0.32} \\ \hline Active PCK-Means & \multicolumn{6}{c}{The mutual information of the final result is 0.37} \\ \hline Roads & \multicolumn{6}{c}{The mutual information of the final result is 0.41} \\ \hline \hline \end{tabular}
\end{table}
Table 3: Evaluation results for the LFW Data Set.