SlowGuess/NAACL_2021_OCR huggingface dataset:
For unimodal experiments, we use the textual encoder or the acoustic encoder independently to compute representations before regression or classification. For the Transformer textual encoder, we use a learning rate of 2e-5; for the MLP acoustic model, we use a learning rate of 5e-4. These learning rates are separately fine-tuned on each unimodal task. Other hyperparameters of the models are kept the same as the multimodal structure.
Table 6 shows the stress detection results with single modalities. Our Transformer encoder outperforms the baseline textual model because of its capability to discover syntactic-level long distance relationships in natural language and the external linguistic knowledge from the advanced BERT pretraining; our acoustic model also improves beyond the baseline results, potentially because we used a more up-to-date version of eGeMaps configuration and a fine-tuned learning rate.
5.3 Multimodal Results
To jointly train with both the textual and acoustic features, we use the multimodal fusion model introduced in Section 4.3 as a basic architecture.
5.3.1 Pre-training
Our MUSER model is trained from scratch to set up a single-task baseline for multimodal stress detection. Besides, a potential alternative to multi-task learning is pre-training on the auxiliary tasks and fine-tuning on the main task. For a complete comparison, we set up several strategies for pretraining. All the pre-training methods use the internal auxiliary task of MuSE. The compared methods are as follows:
Activation-100: pre-train for 100 epochs with the activation annotations, then switch to the main task of stress detection.
Valence-100: pre-train for 100 epochs with the valence annotations, then switch to the main task of stress detection.
Activation-valence-stress: pre-train for 100 epochs on the activation task, then 100 epochs on the valence task, and switch to stress detection.
Valence-activation-stress: pre-train for 100 epochs on the valence task, then 100 epochs on the activation task, and switch to stress detection.
The results are presented in Table 7. Among the pre-training and fine-tuning results, Activation-100 shows the most significant improvement. The second-best score is the valence-activation-stress order. Thus, we can conclude that activation is the better auxiliary task under this paradigm. Additionally, using only one auxiliary task is always
Table 6: Results: stress detection with single modality.
Models	Accuracy	Precision	Recall	F-score
MLP+LIWC (Jaiswal et al., 2020)	0.60	0.74	0.61	0.67
MLP+Opensmile (Jaiswal et al., 2020)	0.67	0.70	0.69	0.69
USER Textual Encoder	0.69	0.77	0.74	0.75
USER Acoustic Encoder	0.79	0.80	0.79	0.80
Table 7: Stress detection results with multiple modalities, with or without pre-training.
Method	Accuracy	Precision	Recall	F-score
LIWC+Audio (Jaiswal et al., 2020)	0.60	0.74	0.61	0.67
A-modal GRU (Bara et al., 2020)	0.573	0.582	0.557	0.569
MUSER from scratch	0.821	0.834	0.828	0.831
Activation-100	0.842	0.866	0.832	0.849
Valence-100	0.823	0.830	0.839	0.834
Activation-valence-stress	0.819	0.841	0.813	0.827
Valence-activation-stress	0.828	0.854	0.817	0.835
better than using two of them; this is because when the model learns from the second auxiliary task, it "forgets" the knowledge from the previous task because it lacks a memory mechanism to look back (Hayes et al., 2020).
Pre-training on the emotion recognition tasks using either activation or valence improves stress detection because the model is equipped with the capabilities to encode the features and predict emotions before the training of stress detection task starts.
5.3.2 Multi-task Learning on MuSE
For multi-task learning, we compare two sampling strategies: uniform sampling and our proposed speed-based dynamic sampling. We also implement and modify the loss-based weight balancing method proposed by (Liu et al., 2019) to adjust the mixing ratios in dynamic sampling instead, and compare it with our methods. The results using the internal MuSE emotion recognition as an auxiliary task are shown in Table 8.
Comparing the uniform sampling results with Table 7, we conclude that using any auxiliary task is better than training from scratch. However, multitask training with the activation and valence tasks together is better than using them separately. This is different from the observations in Table 7 and can be explained by the differences in the training procedure: in multi-task learning, the model looks back-and-forth into each task in each epoch, making it able to memorize the shared knowledge from all the tasks. Additionally, when the model
is optimized for the two emotion tasks at the same time, the lower-level representation becomes more general and informative because it is frequently plugged with different task-specific layers.
Comparing the results of using a single auxiliary task of activation vs. valence, activation leads to better results as compared to valence, which is in agreement with Table 7. This is further supported by the analyses in Tables 2 and 4: given the lower unique indicator count of the activation task, as well as the fact that the pre-training and multi-task learning results are all compatible, we can conclude that for stress detection, the nature of the activation dimension of emotion is closer and more helpful than the valence dimension. This potentially suggests that stress has a major effect on whether people feel excited (activation), but a minor effect on their opinion toward events and objects (valence).
We test our speed-based dynamic sampling algorithm using activation and valence together as auxiliary tasks and it yields promising results with history set to 5 and 10. It significantly outperforms both the uniform sampling and our implementation of the loss-based strategy (Liu et al., 2019) (t-test,  $p < 0.05$ ), achieving state-of-the-art scores on MuSE stress detection task with one single model and only two modalities.
Our model works the best with a history length between 5 and 10. If the history is too short, the model takes longer to converge and has unstable performance, while if the history is too long, it fails to capture the dynamics.
Table 8: Stress detection results with multi-task learning
Strategy	Accuracy	Precision	Recall	F-score
Uniform sampling with activation	0.832	0.833	0.857	0.845
Uniform sampling with valence	0.823	0.837	0.828	0.832
Uniform sampling with activation & valence	0.846	0.862	0.846	0.854
(Liu et al., 2019) - modified	0.842	0.856	0.846	0.851
Speed-based sampling (history=5)	0.854	0.861	0.864	0.863
Speed-based sampling (history=10)	0.856	0.867	0.861	0.864
Table 9: Stress detection results with multi-task learning with OMG-Emotions
Strategy	Accuracy	Precision	Recall	F-score
Uniform sampling with activation & valence	0.844	0.867	0.835	0.850
(Liu et al., 2019) - modified	0.836	0.820	0.886	0.852
Speed-based sampling (history=5)	0.850	0.871	0.842	0.856
5.3.3 Generalization
In real-world applications, stress detection data does not necessarily have emotion labels. However, because of the intrinsic inter-dependence between emotion and stress, any existing dataset with emotion labels can potentially serve as an external auxiliary task for stress detection. However, this requires our model and multi-task training algorithm to generalize beyond the internal MuSE emotion tasks. We test our model on OMG-Emotions as an example of external emotion datasets.
Table 9 shows results on MuSE stress detection using OMG-Emotion as an auxiliary task. Comparing to Table 7, although the source and content of OMG-Emotions are different from MuSE, multitask learning still outperforms single-task learning and pre-training (t-test,  $p < 0.05$ ). This reveals that the connection between stress and emotion widely exists, and our multi-task learning method works in general cases.
Additionally, Table 9 suggests that while using an external emotion dataset, our speed-based sampling method still outperforms uniform sampling, as well as our implementation of loss-based dynamic sampling (Liu et al., 2019). This supports the robustness and effectiveness of our speed-based strategy.
In this work, we uncovered the connections and differences between stress detection and emotion recognition using textual and acoustic features, and proposed to use emotion recognition as an auxiliary task for stress detection. We proposed MUSER: a
Transformer-based model structure, together with a novel speed-based dynamic sampling strategy for multi-task learning. Experimental results support the inter-dependence of stress and emotion (activation/valence), and proves the effectiveness and robustness of our methods. MUSER achieved state-of-the-art results on the MuSE stress detection task both when internal (MuSE) and when external (OMG-Emotions) emotion data and annotations were used.
Our code is publicly available at https://lit.eecs.umich.edu/downloads.html#MUSER
We would like to thank Cristian-Paul Bara and Mimmansa Jaiswal for their helpful discussion on the data processing and features of MuSE dataset. We appreciate the insightful comments from all the reviewers and program committee.
This material is based in part on work supported by the Toyota Research Institute, the Precision Health initiative at the University of Michigan, the National Science Foundation (grant #1815291), and the John Templeton Foundation (grant #61156). Any opinions, findings, conclusions, or recommendations in this material are those of the authors and do not necessarily reflect the views of the Toyota Research Institute, Precision Health initiative, the National Science Foundation, or the John Templeton Foundation.
Ralph Adolphs. 2003. Is the human amygdala specialized for processing social information? Annals of the New York Academy of Sciences, 985(1):326-340.
Jonathan Aigrain, Michel Spodenkiewicz, Severine Dubuiss, Marcin Detyniecki, David Cohen, and Mohamed Chetouani. 2016. Multimodal stress detection from multiple assessments. IEEE Transactions on Affective Computing, 9(4):491-506.
Ane Alberdi, Asier Aztiria, and Adrian Basarab. 2016. Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review. Journal of biomedical informatics, 59:49-75.
Cristian Paul Bara, Michalis Papakostas, and Rada Mihalcea. 2020. A deep learning approach towards multimodal stress detection. In Proceedings of the AAAI-20 Workshop on Affective Content Analysis, New York, USA, AAAI.
Pablo Barros, Nikhil Churamani, Egor Lakomkin, Henrique Siqueira, Alexander Sutherland, and Stefan Wermter. 2018. The OMG-emotion behavior dataset. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE.
M Catarina Botelho, Isabel Trancoso, Alberto Abad, and Teresa Paiva. 2019. Speech as a biomarker for obstructive sleep apnea detection. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5851-5855. IEEE.
Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry, 25(1):49-59.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335.
Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. 2016. Msp-improv: An acted corpus of dyadic interactions to study emotion perception. IEEE Transactions on Affective Computing, 8(1):67-80.
Lan-lan Chen, Yu Zhao, Peng-fei Ye, Jian Zhang, and Jun-zhong Zou. 2017. Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers. *Expert Systems with Applications*, 85:279-291.
Sheng-Yeh Chen, Chao-Chun Hsu, Chuan-Chun Kuo, Lun-Wei Ku, et al. 2018a. Emotionlines: An emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018b. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 794-803. PMLR.
Sheldon Cohen, David AJ Tyrrell, and Andrew P Smith. 1991. Psychological stress and susceptibility to the common cold. New England journal of medicine, 325(9):606-612.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Hilary Dobson and RF Smith. 2000. What is stress, and how does it affect reproduction? Animal reproduction science, 60:743-752.
Florian Eyben, Klaus R Scherer, Björn W Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, et al. 2015. The Geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing. IEEE transactions on affective computing, 7(2):190-202.
Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459-1462.
Hua Gao, Anil Yüce, and Jean-Philippe Thiran. 2014. Detecting emotional stress from facial expressions for driving safety. In 2014 IEEE International Conference on Image Processing (ICIP), pages 5961-5965. IEEE.
Ting Gong, Tyler Lee, Cory Stephenson, Venkata Renduchintala, Suchismita Padhy, Anthony Ndirango, Gokce Keskin, and Oguz H Elibol. 2019. A comparison of loss weighting strategies for multi task learning in deep neural networks. IEEE Access, 7:141627-141632.
Ananth Gottumukkala, Dheeru Dua, Sameer Singh, and Matt Gardner. 2020. Dynamic sampling strategies for multi-task reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 920-924.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018a. Dynamic multi-level multi-task learning for sentence simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 462-476.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. 2018b. Dynamic task prioritization for multitask learning. In Proceedings of the
European Conference on Computer Vision (ECCV), pages 270-287.
Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. 2020. Remind your neural network to prevent catastrophic forgetting. In European Conference on Computer Vision, pages 466-483. Springer.
Frank Horvath. 1982. Detecting deception: the promise and the reality of voice stress analysis. Journal of Forensic Science, 27(2):340-351.
Mimansa Jaiswal, Zakaria Aldeneh, Cristian-Paul Bara, Yuanhang Luo, Mihai Burzo, Rada Mihalcea, and Emily Mower Provost. 2019. Muse-ing on the impact of utterance ordering on crowdsourced emotion annotations. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7415-7419. IEEE.
Mimansa Jaiswal, Cristian-Paul Bara, Yuanhang Luo, Mihai Burzo, Rada Mihalcea, and Emily Mower Provost. 2020. Muse: a multimodal dataset of stressed emotion. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1499-1510.
Natasha Jaques, Sara Taylor, Ehimwenma Nosakhare, Akane Sano, and Rosalind Picard. 2016. Multi-task learning for predicting health, stress, and happiness. In NIPS Workshop on Machine Learning for Healthcare.
Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043.
Nicholas D Lane, Petko Georgiev, and Lorena Qendro. 2015. Deeper: robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 283-294.
Margaret Lech and Ling He. 2014. Stress and emotion recognition using acoustic speech analysis. In *Mental Health Informatics*, pages 163-184. Springer.
Huijie Lin, Jia Jia, Quan Guo, Yuanyuan Xue, Qi Li, Jie Huang, Lianhong Cai, and Ling Feng. 2014. User-level psychological stress detection from social media using deep neural network. In Proceedings of the 22nd ACM international conference on Multimedia, pages 507-516.
Shikun Liu, Edward Johns, and Andrew J Davison. 2019. End-to-end multi-task learning with attention.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1871-1880.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schroder. 2011. The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE transactions on affective computing, 3(1):5-17.
Kannan Muthukumar and Vasanthi Nachiappan. 2013. Phosphatidylethanolamine from phosphatidylserine decarboxylase2 is essential for autophagy under cadmium stress in saccharomyces cerevisiae. Cell biochemistry and biophysics, 67(3):1353-1363.
Ramakanth Pasunuru and Mohit Bansal. 2017. Multi-task video captioning with video and entailment generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1273-1283.
Silke Paulmann, Desire Furnes, Anne Ming Bøkenes, and Philip J Cozzolino. 2016. How psychological stress affects emotional prosody. Plos one, 11(11):e0165022.
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. *Mahway: Lawrence Erlbaum Associates*, 71(2001):2001.
Jens C Pruessner, Katarina Dedovic, Najmeh Khalili-Mahani, Veronika Engert, Marita Pruessner, Claudia Buss, Robert Renwick, Alain Dagher, Michael J Meaney, and Sonia Lupien. 2008. Deactivation of the limbic system during acute psychosocial stress: evidence from positron emission tomography and functional magnetic resonance imaging studies. Biological psychiatry, 63(2):234-240.
Shelley E Taylor and Annette L Stanton. 2007. Coping resources, coping processes, and mental health. Annu. Rev. Clin. Psychol., 3:377-401.
Matthew T Tull, Heidi M Barrett, Elaine S McMillan, and Lizabeth Roemer. 2007. A preliminary investigation of the relationship between emotion regulation difficulties and posttraumatic stress symptoms. Behavior Therapy, 38(3):303-313.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Manjie Wang and Kimberly J Saudino. 2011. Emotion regulation and stress. Journal of Adult Development, 18(2):95-103.
Yiqun Yao, Verónica Pérez-Rosas, Mohamed Abouelenien, and Mihai Burzo. 2020. Morse: Multimodal sentiment analysis for real-life settings. In Proceedings of the 2020 International Conference on Multimodal Interaction, pages 387-396.
Yiqun Yao, Jiaming Xu, and Bo Xu. 2019. The world in my mind: Visual dialog with adversarial multimodal feature encoding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2588-2598.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246.