# An Approach for Selecting the Most Explanatory Features for Facial Expression Recognition
## 5 Results and Discussion
### Statistical Analysis of the Combination Rule  
Tables 3-5 show the accuracy values for the 14 combination rules (columns) and the best combinations of classifiers in different subspaces (rows) for the CK+ data set, BU-3 DFE data set, and JAFFE data set, respectively. Regarding the CK+ data set (Table 3), an accuracy of more than 0.985 was achieved; in the BU-3 DFE data set (Table 4), we achieved an accuracy of more than 0.821; and for JAFFE (Table 5), the accuracy was 0.776 (in all three cases, more than 20 combinations were selected). The best accuracy for each combination is underlined. The last row of tables shows the average accuracy across the 247 combinations. With the large span of classification accuracy values, it is unlikely that these values will be commensurable; however, although the average values across the feature combination do not serve as a valid performance metric, they give a rough indication of the achievements of the combinations. The tables show that the trainable combination rules of MP and ML presented the best results in most cases. The JAFFE data set had the fewest images (213 images), thus limiting the training phase of the trainable combiners. In this case, the best option seemed to be the use of non-trainable combiners. To determine the best choice for CK+ and BU-3 DFE, we calculated the ranks of the combiners. For example, for the combined features R/W/H/P/V/F in the CK+ dataset, the order by rank was as follows: MP (the best), ML, MB, RP, RS, RMD, MV, WMV, LOP, REC, NB, RMX, FI, and RMI (the worst). In case of a tie, the ranks were shared. The average ranks across the combination of features in CK+ (BU-3 DFE) were as follows: RP 6.709 (5.830), RS 6.749 (5.873), RMX 11.699 (12.694), RMI 12.956 (12.484), RMD 6.749 (5.873), MV 7.820 (9.982), WMV 7.524 (7.401), REC 7.512 (6.328), NB 10.757 (9.423), LOP 8.124 (7.294),  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline
**Features** & \multicolumn{4}{c}{**CSC**} & \multicolumn{4}{c}{**CHC**} & \multicolumn{4}{c}{**TC**} \\ \hline  & **RP** & **RS** & **RMX** & **RMI** & **RMD** & **RM** & **WMV** & **REC** & **NB** & **LOP** & **FI** & **MB** & **MP** & **ML** \\ \hline R/B/  
The [PERSON] non-parametric test was executed on the ranks, followed by a multiple comparisons test. The Friedman test score was 2326.60 (2249.70), giving a \(p\)-value of approximately 0 (0), indicating significant differences among the ranks for the CK+ (BU-3 DFE) data set. The Nemenyi post hoc test and Bonferroni-Dunn post hoc test were applied, in order to determine those methods that had significant differences. The result of the Nemenyi post hoc test (two-tailed test) showed that there were significant differences between the MB, MP, and ML methods and all of the other methods, with a significance level of \(\alpha<0.05\). For MB, MP, and ML, we applied the Bonferroni-Dunn post hoc test (one-tailed test) to strengthen the power of the test hypotheses. At a significance level of 0.05, the Bonferroni-Dunn post hoc test did not show significant differences between the MB, MP, and ML methods. Therefore, we can conclude that, in general, these methods had similar behavior in this case. For the case of the BU-3 DFE data set, similar results were obtained. The proposed methods (MB, MP, and ML) were significantly superior to the others for combination rules in FER problems; however, when very little training data sets are available, the use of non-trainable combination rules is suggested. Tables 6-8 show the accuracy, precision, recall, and F1-score measurements of the best schemes for each individual and combination rule in the CK+ data set, BU-3 DFE data set, and JAFFE data set, respectively. In the case of the CK+ data set (Table 6), the R/W/H/P/G/F+MP scheme presented an accuracy of 0.992 and also obtained the best results in terms of precision, recall, and F1-score, with values of 0.991, 0.985, and 0.999, respectively.