###### Abstract  
In real-world scenarios, the facial expression recognition task faces several challenges, including lighting variations, image noise, face occlusion, and other factors, which limit the performance of existing models in dealing with complex situations. To cope with these problems, we introduce the CoT module between the CNN and ViT frameworks, which improves the ability to perceive subtle differences by learning the correlations between local area features at a fine-grained level, helping to maintain the consistency between the local area features and the global expression, and making the model more adaptable to complex lighting conditions. Meanwhile, we adopt an adaptive learning method to effectively eliminate the interference of noise and occlusion by dynamically adjusting the parameters of the Transformer Encoder's self-attention weight matrix. Experiments demonstrate the accuracy of our CoT_AdaViT model in the Oulu-CASIA dataset as (NIR: 87.94%, VL: strong: 89.47%, weak: 84.76%, dark: 82.28%). As well as, CK+, RAF-DB, and FERPlus datasets achieved 99.20%, 91.07%, and 90.57% recognition results, which achieved excellent performance and verified that the model has strong recognition accuracy and robustness in complex scenes. facial expression recognition; CoT; adaptive method; ViT; complex scenes +
Footnote †: journal: Applied  
0
Footnote †: journal: Applied  
0
Footnote 0: _Applied_  
researchers have actively explored the use of near-infrared (NIR) devices for expression recognition. However, the development of NIR-based facial expression recognition has been limited due to the high cost of NIR devices, the lack of wearability, and the relatively low quality and resolution of NIR images. These challenges make accurate recognition of facial expressions in complex scenes a difficult task. With the continuous development of convolutional networks, the facial expression recognition models proposed based on this technique have achieved excellent performance [7; 8; 9]. However, the spatial localization of convolutional networks makes it difficult for the model to learn the dependencies between different facial regions, thus limiting the understanding of global facial expressions. To overcome these problems, some improved models based on convolutional networks have also been proposed, such as the attention mechanism [7], graph convolutional networks [10; 11], etc., which aim to enhance the model's ability to learn the dependencies between different facial regions, to better capture the global facial expression information and to improve the model's ability to recognize expressions. In recent years, the Transformer architecture [12] has achieved remarkable success in natural language processing tasks, and inspired by its successful application, ViT (Vision Transformer) [13] has been introduced into image classification tasks and achieved remarkable results through the non-local attention mechanism. Researchers have also introduced it into FER tasks, where its global context-awareness capability can help models better understand global facial expression features. However, since ViT lacks spatial information, is more dependent on large-scale data, and requires more computational resources and time for training and inference. In addition, FER datasets are usually small in size, and training FER models using only the Transformer encoder results in models that are difficult to converge and tend to be biased to focus on occluded and noisy regions. Therefore, the direct use of ViT models for expression recognition may be limited. To mitigate the effects of illumination changes on facial expression features and to better capture contextual correlations between local regions, we take note of the CoT module [14] (Contextual Transformer), which is a module based on self-attentional improvements to learn correlations between different regions by combining both static and dynamic contextual representations as the final output, as a way of making full use of the information in neighboring regions.