# Nemesis: Neural Mean Teacher Learning-Based Emotion-Centric Speaker
###### Abstract  
Image captioning is the multi-modal task of automatically describing a digital image based on its contents and their semantic relationship. This research area has gained increasing popularity over the past few years; however, most of the previous studies have been focused on purely objective content-based descriptions of the image scenes. In this study, efforts have been made to generate more engaging captions by leveraging human-like emotional responses. To achieve this task, a mean teacher learning-based method has been applied to the recently introduced ArtEmis dataset. ArtEmis is the first large-scale dataset for emotion-centric image captioning, containing 455K emotional descriptions of 80K artworks from WikiArt. This method includes a self-distillation relationship between memory-augmented language models with meshed connectivity. These language models are trained in a cross-entropy phase and then fine-tuned in a self-critical sequence training phase. According to various popular natural language processing metrics, such as BLEU, METEOR, ROUGE-L, and CIDEr, our proposed model has obtained a new state of the art on ArtEmis.  
image captioning; mean teacher learning; self-distillation; self-critical sequence training; natural language processing +
Footnote †: journal: _Accepted: 7 February 2023_  
pairs with two different styles. In stylized image captioning, the core concept is generating objective descriptions, only with different wording according to the target linguistic style.  
Various modalities can be utilized in the task of attention-based image captioning. [PERSON] et al. [17] introduced a novel sound active attention framework to generate captions more specific to the interest of the observer. The process is similar to the active attention in the human mind generated by top-down signals. On the other hand, [PERSON] et al. [18] proposed NUAN, a non-uniform attention network for multi-modal feature fusion. The attention mechanism in NUAN considers three modalities of text, sound, and vision non-uniformly, where the text is utilized as a determinative representation while visual and acoustic representations are leveraged to obtain a solid representation.  
[PERSON] et al. [19] introduced ArtEmis, the first large-scale dataset for affective image captioning accompanied by baseline neural speakers to generate emotion-based descriptions. In addition to being subjective, rich, and diverse, this dataset contains affective language. Figure 1 shows an example from the ArtEmis dataset containing multiple emotional responses for the same artwork. In this paper, we propose Nemesis, a Neural Mean Teacher Learning-based Emotion-centric Speaker trained on ArtEmis. Our model is capable of generating affective utterances describing the triggered human-like emotional responses stemming from visual stimuli. The generated utterances are according to both the visual features and emotion-based supervision signals.  
The proposed pipeline consists of:  
* Auxiliary text-to-emotion and image-to-emotion classification tasks;
* A visual encoder to extract visual features from the input image;  
Figure 1: An example from the ArtEmis dataset containing multiple emotional responses for the same artwork. You can see the different descriptions along with their corresponding emotional class (in bold font).  
* Two interconnected language models following a transformer-based encoder/decoder architecture. In Nemesis, we introduce two main contributions: 1. A novel approach for the image-to-emotion classification task by decreasing the texture bias of the classifier and encouraging the model towards a shape-based classification. This is because of the differing local textures in our input images (artworks) in comparison to the real world; 2. Achieving a state-of-the-art performance using the Nemesis on the ArtEmis dataset. We suggest that a self-critical mean teacher learning-based approach, supervised by extra emotional signals, is a promising path towards generating more human-like, emotionally rich captions.