# ViTFER: Facial Emotion Recognition with Vision Transformers
## 1 Introduction  
Emotion recognition helps humans understand the intentions and psychic conditions of the human mind. Nevertheless, an individual will not be able to recognize the human mentality accurately. Emotions are fundamental features of humans that play an important role in social communication [1, 2]. Automated emotion recognition can help numerous domains such as education, healthcare, cybersecurity, human-computer interaction (HCI), human-robot interaction (HRI), and virtual reality to sense the stimuli of the human brain by applying various modalities such as images, videos, audio, and text. Humans express emotion in various ways, including facial expression [3, 4], speech [5], and body language [6]. Facial expression analysis is the most popular and well-researched element related to emotion recognition. The main objective of this research is to investigate and check the feasibility of the Vision Transformer in the emotion recognition domain by utilizing a combination of well-known datasets for FER. High-quality data are being gathered in this area to meet the system's requirements for sensing, processing, and simulating human emotions. The portrayal of the six fundamental human emotions--happiness, anger, surprise, sadness, fear, and disgust--by humans is a well-established fact [7]. These are the six basic emotions, other than these, several other pieces of research are considered for research according to the respective domain. They include various emotions such as drowsiness, pain, and tiredness. Various areas offer the potential for the application of automated facial expression recognition. Vital security information can be retrieved from live monitoring of people's facial expressions. An alert can be generated from a computer system when a human exhibits fear, disgust, pain, or any such kind of facial expression; this can help medical professionals to keep track of a patient's mental state. When something is found awry, professionals can change their method of treating the patient. Various machine learning and deep learning algorithms are being applied along with some novel approachesto obtain effective results for emotion recognition. In this research, we will focus on the latest deep learning model, i.e., a transformer with an attention mechanism. The pivotal mode of recognizing human emotion is thought to be facial expression. This mode uses facial motion and curvatures of facial characteristics to classify emotions into various categories. Making emotion recognition systems functional for real-life circumstances is one of our priorities. Many researchers in psychology have found that participants correctly identified the emotions represented by the body features when faces and body traits representing various emotions were combined. Faces are one of these features that are frequently used, if not the most frequently used, for various reasons, including their visibility, the wealth of features they contain for emotion recognition, and the ease with which a large dataset of faces can be gathered. Here, we consider facial expressions in the form of images as an input modality for recognizing the emotional state of a human mind. It is interesting that, in the case of facial expressions, the lips and eyes provide most cues, whereas other facial features, such as the ears and hair, play a small role [8]. Considering the facial expression as the relevant cue for the emotion recognition system, we also represented a new dataset named AVFER, which is a combination of FER2013, AffectNet, and CK+48 datasets. We applied fine-tuned ViT with a FER-based base configuration for image recognition and test its robustness on AVFER. Firstly, we analyzed various datasets for facial emotion recognition by identifying the distribution ratio of emotion categories. After rectifying the uneven distribution of data, we decided to merge three datasets, namely FER-2013, AffectNet, and CK+48 in the fraction of proportions. We then split a set of sample data into training, validation, and testing to evaluate our model.