SlowGuess/NAACL_2022_OCR huggingface dataset:
Model	Text	Image	Desire Detection	Sentiment Analysis	Emotion Recognition
P	R	\(M_a-F1\)	P	R	\(M_a-F1\)	P	R	\(M_a-F1\)
Text	DCNN	-	36.91	31.64	29.55	59.31	53.01	51.19	43.66	41.10	41.60
BiLSTM	-	73.20	67.82	69.14	78.43	78.75	78.58	73.49	72.17	72.73
BERT	-	81.74	80.39	80.88	84.43	84.28	84.35	81.76	80.57	81.10
Image	-	AlexNet	51.47	49.33	50.07	68.76	68.21	68.45	56.42	53.29	54.66
-	ResNet	49.97	49.35	49.20	70.85	70.61	70.64	58.74	54.67	56.40
Text+Image	DCNN	AlexNet	59.42	52.02	52.35	71.02	70.09	70.31	49.56	42.77	43.76
DCNN	ResNet	56.34	50.64	52.89	74.73	74.73	74.64	62.93	59.12	60.48
BiLSTM	AlexNet	67.80	68.00	67.67	78.73	79.22	78.89	71.17	70.70	70.89
BiLSTM	ResNet	54.97	49.94	51.99	75.89	75.27	75.25	63.63	60.80	61.98
BERT	AlexNet	80.84	75.50	77.17	83.22	83.11	83.16	78.06	78.19	78.10
BERT	ResNet	83.42	82.43	82.28	85.83	85.79	85.81	83.54	81.51	82.42
Multimodal Transformer	-	-	81.92	80.20	80.92	83.56	83.45	83.50	81.62	81.61	81.53
Table 5: Comparison of different models.
BERT+ResNet Multi-Modal Fusion	Desire Detection	Sentiment Analysis	Emotion Recognition
Validation	Test	Validation	Test	Validation	Test
Concatenate	83.55	85.21	83.64	85.95	79.63	82.91
Add	85.31	86.48	83.06	85.94	82.08	82.32
Multiply	83.64	83.99	85.21	85.50	78.65	81.59
Maximum	84.62	85.55	83.94	85.11	80.90	81.83
Table 6: Comparison of different multi-modal combinations.
Method	Desire	Sentiment	Emotion
Annotator 1	88.00	90.00	86.00
Annotator 2	84.00	88.00	86.00
Annotator 3	84.00	88.00	82.00
Avg.	85.33	88.67	84.67
BERT+ResNet	82.00	86.00	82.00
Table 7: The human evaluation results against BERT+ResNet for three tasks.
Task Sequence	Desire	Sentiment	Emotion
des ⇒ sen ⇒ emo	84.82	85.46	82.13
des ⇒ emo ⇒ sen	84.82	85.06	82.22
sen ⇒ des ⇒ emo	85.85	82.73	82.62
sen ⇒ emo ⇒ des	85.90	82.73	82.08
emo ⇒ sen ⇒ des	85.60	85.16	80.80
emo ⇒ des ⇒ sen	84.18	84.87	80.80
Table 8: All the possible task inference sequences.
5.4 Human Evaluation Results
Next, we create a new test set including 50 multimodal documents, and recruit three undergraduate volunteers to evaluate the desire, sentiment and emotion labels. We run the inter-annotator agreement study on three volunteers' scores and the average kappa scores are 0.80, 0.82 and 0.78 for our three tasks. We also choose the pre-trained BERT+ResNet (the state-of-art system) to make desire, sentiment and emotion predictions. Table 7 presents the comparative results.
We can see that although BERT+ResNet have attained the best macro-F1 scores before, they still perform worse than human evaluation. One possible reason is that multi-modal representation and fusion may miss some essential contents. This proves that such strong baselines can not guarantee a satisfactory result compared to human judgment. Desire understanding is thus an emerging, yet challenging task, where novel multi-modal desire understanding models are needed. The proposed MSED dataset will provide an available data bed for model evaluation.
5.5 Discussion on Inter-Task Correlation
In order to verify the correlations across multiple tasks, e.g., which task offers the greatest help to other tasks, we improve BERT+ResNet by incorporating the inference sequence knowledge. We choose to merge the former task knowledge (the output of the dense layer) with the features of the latter task to construct a new input for the latter task. This action will naturally leverage the knowledge from other tasks. We have checked all the possible task combinations, e.g.,  $(des \Rightarrow sen \Rightarrow emo)$ ,  $(sen \Rightarrow des \Rightarrow emo)$ , etc. We show the obtained results in Table 8. We see that BERT+ResNet performs the best for the task of desire detection under the task sequence of  $(sen \Rightarrow emo \Rightarrow des)$ . This shows that sentiment and emotion knowledge indeed helps improve desire detection. By comparing the performance of three tasks, we notice that sentiment and emotion tasks gain greater improvement over desire detection under the task sequences of  $(des \Rightarrow sen \Rightarrow emo)$  and  $(sen \Rightarrow des \Rightarrow emo)$ . These results support our argument that desire, sentiment and emotion are
not only inter-entangled, sentiment and emotion are but also actuated by human desire. In addition, the importance of multi-task clues is also investigated.
From Table 5, we perform an ablation study by analyzing the effectiveness of different components of BERT+ResNet. By comparing the classification performance of BERT and ResNet, we see that using textual features is more effective than using visual features, as we expected. The main reasons are: (1) BERT contributes the most to overall framework, as it effectively captures the inter-dependencies between words and extracts refined features; (2) Text cue plays a more important role than visual cue for desire understanding, since visual desire analysis involves a higher level of abstraction. However, ResNet still outperforms DCNN and BiLSTM by a large margin  $(7\%, 5\% \uparrow)$ , which shows the effectiveness of pre-trained visual model.
5.7 Error Analysis
Through presenting the confusion matrices of BERT+ResNet in Fig. 7, we perform an error analysis. We notice that misclassification for BERT+ResNet often happens in four categories of samples, i.e., non-desire, curiosity, social-contact and tranquility. About  $10.6\%$  non-desire samples are mis-classified as various desires.  $29.5\%$  curiosity samples are misdiagnosed. For tranquility detection, BERT+ResNet performs very poorly, which annotates almost half  $(36.8\%)$  tranquility samples as non-desire labels.  $15.2\%$  social-contact desire is misdiagnosed as non-desire. This implicates that BERT+ResNet struggles in differentiating curiosity, social-contact and tranquility from non-desire. Further theoretical and empirical research is needed for better studying human desires. We also show a few misclassification cases for desire detection, as shown in Fig. 8.
Figure 7: The Confusion matrix.
(a) Father with dog and two children relaxing in living room.
(b) Little boy observing insects with a magnifying glass.
(c) Young dominating woman pulling an injured man by tie and giving her thumb up.
(d) Girl closes her eyes tightly as she goes on a zip line.
Figure 8: Wrongly classified multi-modal samples.
Human desire understanding is a relatively unexplored task in NLP. To fill this gap, we expand desire research from psychology to multi-modal language analysis, and thus propose the first multimodal multi-task dataset for desire, sentiment and emotion detection, MSED. Each sample is annotated with six basic desires, three sentiments and six emotions. In addition, qualitative and quantitative studies are performed for analyzing the dataset. We also present a range of baselines to evaluate the potential of MSED. The comparative and human evaluation results demonstrate the need of new desire analysis models and the potential of MSED to facilitate the development of such models.
Our work has also a few limitations. The images available in platforms like Flickr and Getty may not express spontaneous human desire, as many of them are purposefully designed by professional photographers. The current dataset only collects static images and texts, the conversational samples might be considered. Moreover, a larger scale multi-modal dataset with more desire categories is left to our future work. The technique of human desire analysis based on online data also has the potential to be misused, e.g. by integrating them with facial recognition techniques to make interventions or decisions for humans.
In summary, we hope that the creation of MSED will provide a new perspective in NLP for research on human desire analysis. The dataset will be publicly available for research. Given the close relationship between desire, sentiment and emotion, a refined multi-modal multi-task learning framework is left to our future work.
Md Zahangir Alom, Tarek M Taha, Christopher Yakop-cic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian C Van Esesen, Abdul A S Awwal, and Vijayan K Asari. 2018. The history began from alexnet: A comprehensive survey on deep learning approaches. arXiv preprint arXiv:1803.01164.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc."
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):325-335.
Stephanie Cacioppo, Francesco Bianchi-Demicheli, Chris Frum, James G Pfaus, and James W Lewis. 2012. The common neural bases between sexual desire and love: a multilevel kernel density fmri analysis. The journal of sexual medicine, 9(4):1048-1054.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _obviously_ perfect paper). arXiv preprint arXiv:1906.01815.
Dushyant Singh Chauhan, SR Dhanush, Asif Ekbal, and Pushpak Bhattacharyya. 2020a. All-in-one: A deep attentive multi-task learning framework for humour, sarcasm, offensive, motivation, and sentiment on memes. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 281-290.
Dushyant Singh Chauhan, SR Dhanush, Asif Ekbal, and Pushpak Bhattacharyya. 2020b. Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4351-4360.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Jeyoun Dong, Hen-I Yang, and Carl K Chang. 2013. Identifying factors for human desire inference in smart home environments. In International Conference on Smart Homes and Health Telematics, pages 230-237. Springer.
Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613-619.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 214-229. Springer.
Andrew B Goldberg, Nathanael Fillmore, David Andrzejewski, Zhiting Xu, Bryan Gibson, and Xiaojin Zhu. 2009. May all your wishes come true: A study of wishes and how to recognize them. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 263-271.
Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed (Ehsan) Hoque. 2019. UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2046-2056, Hong Kong, China. Association for Computational Linguistics.
Wilhelm Hofmann and Loran F Nordgren. 2015. The psychology of desire. Guilford Publications.
Sabrina Hoppe, Tobias Loetscher, Stephanie Morey, and Andreas Bulling. 2015. Recognition of curiosity using eye movement analysis. In Adjunct proceedings of the 2015 acm international joint conference on pervasive and ubiquitous computing and proceedings of the 2015 acm international symposium on wearable computers, pages 185-188.
Ronald J Hunt. 1986. Percent agreement, pearson's correlation, and kappa as measures of inter-examiner reliability. Journal of Dental Research, 65(2):128-130.
Chammah J Kaunda and Mutale Mulenga Kaunda. 2021. Gender and sexual desire justice in african christianity. *Feminist Theology*, 30(1):21-36.
Xiang Li, Yazhou Zhang, Prayag Tiwari, Dawei Song, Bin Hu, Meihong Yang, Zhigang Zhao, Neeraj Kumar, and Pekka Marttinen. 2022. Eeg based emotion recognition: A tutorial and review. ACM Computing Surveys (CSUR).
Angelica Lim, Tetsuya Ogata, and Hiroshi G Okuno. 2012. The desire model: Cross-modal emotion analysis and expression for robots. Information Processing Society of Japan, 5:4.
Yaochen Liu, Yazhou Zhang, Qiuchi Li, Benyou Wang, and Dawei Song. 2021. What does your smile mean? jointly detecting multi-modal sarcasm and sentiment using quantum probability. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 871-880.
Lisa Mendelman. 2021. Diagnosing desire: Mental health and modern american literature, 1890-1955. American Literary History, 33(3):601-619.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037.
Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, volume 1, pages 973-982.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 527-536.
Paul Portner and Aynat Rubinstein. 2020. Desire, belief, and semantic composition: variation in mood selection with desire predicates. Natural Language Semantics, 28(4):343-393.
Jenefer Robinson. 1983. Emotion, judgment, and desire. The Journal of Philosophy, 80(11):731-741.
Ted Ruffman, Lance Slade, Kate Rowlandson, Charlotte Rumsey, and Alan Garnham. 2003. How language relates to belief, desire, and emotion understanding. Cognitive Development, 18(2):139-158.
Nicola S Schutte and John M Malouff. 2020. A meta-analysis of the relationship between curiosity and creativity. The Journal of Creative Behavior, 54(4):940-947.
Reiss Steven. 2004. Multifaceted nature of intrinsic motivation: The theory of 16 basic desires. Review of General Psychology, 8(3):179-193.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence.
Olga Uryupina, Barbara Plank, Aliaksei Severyn, Agata Rotondi, and Alessandro Moschitti. 2014. Sentube: A corpus for sentiment analysis on youtube social media. In LREC, pages 4244-4249.
Nan Xu, Wenji Mao, and Guandan Chen. 2019. Multi-interactive memory network for aspect based multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 371-378.
Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3718-3727.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmumosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246.
Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, and Hongfei Lin. 2021a. Multimet: A multimodal dataset for metaphor understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3214-3225.
Yazhou Zhang, Yaochen Liu, Qiuchi Li, Prayag Tiwari, Benyou Wang, Yuhua Li, Hari Mohan Pandey, Peng Zhang, and Dawei Song. 2021b. Cfn: A complex-valued fuzzy network for sarcasm detection in conversations. IEEE Transactions on Fuzzy Systems.
Yazhou Zhang, Dawei Song, Xiang Li, Peng Zhang, Pan pan Wang, Lu Rong, Guangliang Yu, and Bo Wang. 2020a. A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis. Information Fusion, 62:14-31.
Yazhou Zhang, Prayag Tiwari, Lu Rong, Rui Chen, Nojoom A AlNajem, and M Shamim Hossain. 2021c. Affective interaction: Attentive representation learning for multi-modal sentiment classification. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM).
Yazhou Zhang, Prayag Tiwari, Dawei Song, Xiaoliu Mao, Panpan Wang, Xiang Li, and Hari Mohan Pandey. 2021d. Learning interaction dynamics with an interactive LSTM for conversational sentiment analysis. Neural Networks, 133:40-56.
Yazhou Zhang, Zhipeng Zhao, Panpan Wang, Xiang Li, Lu Rong, and Dawei Song. 2020b. Scenarios: A dyadic conversational database for interactive sentiment analysis. IEEE Access, 8:90652-90664.
A Model Building
We apply a bi-modal encoder architecture when building models. The bi-modal encoder consists of text (i.e., DCNN, BiLSTM and BERT) and image encoders (i.e., AlesNet and ResNet). The outputs from two encoders are concatenated to form the multi-modal representation, and thus are forwarded to a dense layer to make prediction of three tasks.
A.1 Text Encoder
We use GloVe 6B to initialize the 100 dimensional word embeddings as inputs for DCNN and BiLSTM. As for BERT, the dimension is 768.
DCNN. The first convolutional layer in the DCNN consists of 3 filters of size  $2 \times 2$ . The second convolutional layer consists of 3 filters of size  $3 \times 3$ . The third convolutional layer consists of 3 filters of size  $4 \times 4$ . This network is followed by the fully connected layer (size of 128) and the softmax layer. Finally, the activation values of the fully connected layer are used as the output of the encoder.
BiLSTM. It consists of two LSTM layers that read the input sequence forwardly and backwardly to generate a series of bidirectional hidden states. The  $i^{th}$  hidden representation is obtained by merging the bidirectional hidden states, e.g.,  $h_i = \vec{h}_i \parallel h_i$ , where  $i \in [1,2,\dots,n]$ . In BiLSTM, the dimensions of forward and backward hidden states are set to 50 respectively. Finally, the final hidden sate  $h_n$  is used as the sequence representation.
BERT. We fine-tuned the BERT-base including 12 layers and 110M parameters as the text encoder. Each sequence will be padded or truncated to the size of 50 before it is input. The obtained representation of the first token in the sequence (i.e., the [CLS] token) is used as the output of the encoder, where the dimension is 768.
A.2 Image Encoder
Each image is pre-processed by using mean and standard deviation calculated by ImageNet.
AlexNet. The size of the input images is  $408 \times 612 \times 3$ . The first convolutional layer has 96 kernels of size  $12 \times 40 \times 3$  with a stride of 4 pixels. The second convolutional layer has 256 kernels of size  $5 \times 5 \times 96$  with a stride of 2 pixels. The third convolutional layer has 384 kernels of size  $3 \times 3 \times 256$ . The forth convolutional layer has 384 kernels
of size  $3 \times 3 \times 384$ , and the fifth convolutional layer has 256 kernels of size  $3 \times 3 \times 384$ .
ResNet. The ResNet18 pre-trained model is used in our experiments. All the images are resized to  $612 \times 612 \times 3$  before they are feed into the model.
We use Pytorch (Paszke et al., 2019) to build all models. To avoid overfitting, we choose to perform early stopping during training. During training, the optimal learning rate is set to  $1 \times 10^{-5}$  and the epoch is 40 if the encoder includes pre-trained model, otherwise they are set to  $1 \times 10^{-3}$  and 100 respectively. The dropout rate in the model is 0.5. In our models, cross entropy with  $L2$  regularization is used as the loss function, as shown in Eq. 1:
$$
\zeta_ {s} = - \frac {1}{L} \sum_ {\xi} Y _ {\xi} \log \hat {Y} _ {\xi} + \tau_ {r} \| \phi \| ^ {2} \tag {1}
$$
where  $\zeta_s \in \{\zeta_{sen}, \zeta_{emo}, \zeta_{des}\}$ ,  $Y_\xi$  denotes the ground truth of the  $\xi^{th}$  sample,  $\hat{Y}_\xi$  is the predicted distribution.  $\xi$  is the index of sample, and  $L$  is the total number of samples.  $\tau_r$  is the coefficient for  $L2$  regularization. As for optimizer, we choose Adam to optimize the loss function. We use the back propagation method to compute the gradients and update all the parameters. It takes about 50 minutes for the state-of-the-art system (i.e., BERT+ResNet) to train its best performance over MSED via  $1 \times$  RTX A6000 GPU.