# Facial Expression Recognition Using Computer Vision: A Systematic Review
## 8 Insights on Emotion Recognition in the Wild Challenge
This work won the EmotiW 2016 challenge with the best submission achieving 59.02% accuracy. [PERSON] et al. [179] proposed a new learning method named Supervised Scoring Ensemble (SSE) with deep CNNs. They added supervision not only to deep layers but also to intermediate layers and shallow layers to ease the training. They also presented a new fusion structure in which class-wise scoring activation at diverse complementary feature layers are concatenated and further used as the inputs for second-level supervision, acting as a deep feature ensemble within a single CNN architecture. As for the pre-processing step, they applied SDM to track facial features, face frontalization, rescaling, and applied a DCT based method for intensity normalization. As for the feature extraction step, they combined the grayscale face image with its corresponding basic LBP and mean LBP feature maps to form a three-channel input. This work won the Emotiw 2017 challenge with the best submission achieving 60.34% accuracy. [PERSON] et al. [158] proposed a hybrid net containing three main parts for the visual features: Landmark ED, CNN, and LSTM. These parts were then combined with weights for the final classification. For the Landmark ED part, 34 EDs were calculated as well as the mean, maximum, and variance, resulting in 102 total features for each video. This part alone achieved 39.95% accuracy on the AFEW database. For the CNN part, four CNNs were fine-tuned to predict single static images. Features extracted from these four CNNs were used to train a linear SVM. The VGG-Face model [180] was fine-tuned using the FER-2013 database for feature extraction. Those features were then used to train an LSTM, and this part alone was able to achieve 46.21% accuracy. Finally, they combined these three visual parts with the audio part as a final decision step. As for the pre-processing step, DA and face alignment were performed. This work won the EmotiW 2018 challenge with the best submission achieving 61.87% accuracy. Figure 23 illustrates the EmotiW Challenge winners' accuracy over time. It is possible to see that the EmotiW Challenge is stimulating the research done in emotion recognition in the wild, correlating with better results over the years. Recent winners used face frontalization to overcome the pose-variant faces problem. Since the AFEW database is 2D, it is the most reasonable solution. Overall, the winners don't seem to overlook the pre-processing step: most do geometric transformations and illumination corrections in order to normalize the data. Concerning the feature extraction step, at least two winners applied SDM to track facial features. The main facial features that were explored were AUs, HOG, SIFT, and LBP features. As for the classification step, they mainly used SVMs and fine-tuned CNNs in a combined way. Some winners also used LSTMs to make predictions based on the temporal data of the AFEW database. Based on the EmotiW Challenge results, the future direction for FER in uncontrolled environments seems to be converging into:  
* Pre-processing techniques that normalize pose-variant faces as well as the image intensity. * The exploration of AUs, HOG, SIFT, and LBP features. * The use of hybrid classifiers based on SVMs, fine-tuned CNNs, and LSTMs. Figure 23: EmotiW challenge winners’ accuracy over time.