Reusing Preprocessing Data as Auxiliary Supervision in Conversational Analysis

1 INTRODUCTION . The sharp increase in uses of video-conferencing creates both a need and an opportunity to better understand these conversations ( Kim et al. , 2019a ) . In post-event applications , analyzing conversations can give feedback to improve communication skills ( Hoque et al. , 2013 ; Naim et al. , 2015 ) . In real-time applications , such systems can be useful in legal trials , public speaking , e-health services , and more ( Poria et al. , 2019 ; Tanveer et al. , 2015 ) . Analyzing conversations requires both human expertise and a lot of time , which is what many multimodal conversational analysis systems are trying to solve with automation . However , to build such systems , analysts often require a training set annotated by humans ( Poria et al. , 2019 ) . The annotation process is costly , thereby limiting the amount of labeled data . Moreover , third-party annotations on emotions are often noisy . Noisy data coupled with limited labeled data increases the chance of overfitting ( James et al. , 2013 ) . From the perspective of feature engineering to analyze video-conferences , analysts often employ pre-built libraries ( Baltrušaitis et al. , 2016 ; Vokaturi , 2019 ) to extract multimodal features as inputs to training . This preprocessing phase is often computationally heavy , and the resulting features are only used as inputs . In this paper , we investigate how the preprocessed data can be re-used as auxiliary tasks in Primary Multi-Task Learning ( MTL ) , thereby promoting a more productive use of data , in the form of auxiliary supervised learning . Specifically , our main contributions are ( 1 ) the identification of beneficially auxiliary tasks , ( 2 ) the method of distributing learning capacity between the primary and auxiliary tasks , and ( 3 ) the relative supervision hierarchy between the primary and auxiliary tasks . We demonstrate the value of our approach through predicting emotions on two publicly available datasets , IEMOCAP ( Busso et al. , 2008 ) and SEMAINE ( McKeown et al. , 2011 ) . 2 RELATED WORKS AND HYPOTHESES . Multitask learning has a long history in machine learning ( Caruana , 1997 ) . In this paper , we focus on Primary MTL , a less commonly discussed subfield within MTL ( Mordan et al. , 2018 ) . Primary MTL is concerned with the performance on one ( primary ) task – the sole motivation of adding auxiliary tasks is to improve the primary task performance . In recent years , primary MTL has been gaining attention in computer vision ( Yoo et al. , 2018 ; Fariha , 2016 ; Yang et al. , 2018 ; Mordan et al. , 2018 ; Sadoughi & Busso , 2018 ) , speech recognition ( Krishna et al. , 2018 ; Chen & Mak , 2015 ; Tao & Busso , 2020 ; Bell et al. , 2016 ; Chen et al. , 2014 ) , and natural language processing ( NLP ) ( Arora et al. , 2019 ; Yousif et al. , 2018 ; Zalmout & Habash , 2019 ; Yang et al. , 2019 ; Du et al. , 2017 ) . The benefit of adding multiple tasks is to provide inductive bias through multiple noisy supervision ( Caruana , 1997 ; Lipton et al. , 2015 ; Ghosn & Bengio , 1997 ) . On the other hand , the drawback of adding multiple tasks increases the risk of negative transfer ( Torrey & Shavlik , 2010 ; Lee et al. , 2016 ; 2018 ; Liu et al. , 2019 ; Simm et al. , 2014 ) , which leads to many design considerations . Two of such considerations are , identifying ( a ) what tasks are beneficial and ( b ) how much of the model parameters to share between the primary and auxiliary tasks . In addition , because we are performing Primary MTL , we have the third consideration of ( c ) whether we should prioritize primary supervision by giving it a higher hierarchy than the auxiliary supervision . In contrast with previous MTL works , our approach ( a ) identifies sixteen beneficially auxiliary targets , ( b ) dedicates a primary-specific branch within the network , and ( c ) investigates the efficacy and generalization of prioritizing primary supervision across eight primary tasks . Since our input representation is fully text-based , we dive deeper into primary MTL in the NLP community . Regarding model architecture designs for primary MTL in NLP , Søgaard & Goldberg ( 2016 ) found that lower-level tasks like part-of-speech tagging , are better kept at the lower layers , enabling the higher-level tasks like Combinatory Categorical Grammar tagging to use these lowerlevel representations . In our approach , our model hierarchy is not based on the difficulty of the tasks , but more simply , we prioritize the primary task . Regarding identifying auxiliary supervisors in NLP , existing works have included tagging the input text ( Zalmout & Habash , 2019 ; Yang et al. , 2019 ; Søgaard & Goldberg , 2016 ) . Text classification with auxiliary supervisors have included research article classification ( Du et al. , 2017 ; Yousif et al. , 2018 ) , and tweet classification ( Arora et al. , 2019 ) . There is a large body of work in multimodal sentiment analysis , but not in the use of multimodal auxiliary supervisors , as detailed in the next paragraph . Multimodal analysis of conversations has been gaining attention in deep learning research , particularly for emotion recognition in conversations ( Poria et al. , 2019 ) . The methods in the recent three years have been intelligently fusing numeric vectors from the text , audio , and video modalities before feeding it to downstream layers . This approach is seen in MFN ( Zadeh et al. , 2018a ) , MARN ( Zadeh et al. , 2018b ) , CMN ( Hazarika et al. , 2018b ) , ICON ( Hazarika et al. , 2018a ) , DialogueRNN ( Majumder et al. , 2019 ) , and M3ER ( Mittal et al. , 2020 ) . Our approach is different in two ways . ( 1 ) Our audio and video information is encoded within text before feeding only the text as input . Having only text as input has the benefits of interpretability , and the ability to present the conversational analysis on paper ( Kim et al. , 2019b ) . This is similar to how the linguistics community performs manual conversational analysis using the Jefferson transcription system ( Jefferson , 2004 ) , where the transcripts are marked up with symbols indicating how the speech was articulated . ( 2 ) Instead of using the audio and video information as only inputs to a Single Task Learning ( STL ) model , the contribution of this paper is that we demonstrate how to use multimodal information in both input and as auxiliary supervisors to provide inductive bias that helps the primary task . Hypothesis H1 : The introduced set of auxiliary supervision features improves primary MTL . We introduce and motivate the full set of sixteen auxiliary supervisions , all based on existing literature : these are grouped into four families , each with four auxiliary targets . The four families are ( 1 ) facial action units , ( 2 ) prosody , ( 3 ) historical labels , and ( 4 ) future labels : ( 1 ) Facial action units , from the facial action coding system identifies universal facial expressions of emotions ( Ekman , 1997 ) . Particularly , AU 05 , 17 , 20 , 25 have been shown to be useful in detecting depression ( Yang et al. , 2016a ; Kim et al. , 2019b ) and rapport-building ( Anonymous , 2021 ) . ( 2 ) Prosody , the tone of voice – happiness , sadness , anger , and fear – can project warmth and attitudes ( Hall et al. , 2009 ) , and has been used as inputs in emotions detection ( Garcia-Garcia et al. , 2017 ) . ( 3 and 4 ) Using features at different historical time-points is a common practice in statistical learning , especially in time-series modelling ( Christ et al. , 2018 ) . Lastly , predicting future labels as auxiliary tasks can help in learning ( Caruana et al. , 1996 ; Cooper et al. , 2005 ; Trinh et al. , 2018 ; Zhu et al. , 2020 ; Shen et al. , 2020 ) . Inspired by their work , we propose using historical and future ( up to four talkturns ago or later ) target labels as auxiliary targets . Although historical and future target labels can be used as a pre-training objective and fine-tuned on the current target label , sequential transfer learning is not the focus of this paper . Given that we are extracting actions and prosody families as inputs , we propose to explore whether they can be reused as supervisors ( see Fig . 1 ) . Our hypothesis H1 is that re-using them as auxiliary supervision improves primary MTL . This is related to using hints in the existing MTL literature , where the auxiliary tasks promote the learning of the feature ( Cheng et al. , 2015 ; Yu & Jiang , 2016 ) . Hypothesis H2 : When the primary branch is given maximum learning capacity , it would not be outperformed by models with primary branch having less than the maximum learning capacity . Deeper models with higher learning capacity produce better results ( Huang et al. , 2019 ; Nakkiran et al. , 2019 ; Menegola et al. , 2017 ; Blumberg et al. , 2018 ; Romero et al. , 2014 ) . Also , since the auxiliary branch is shared with the primary supervision , the auxiliary capacity should be limited to enforce information transfer and improve performance ( Wu et al. , 2020 ) . Therefore , given a fixed learning capacity budget , our hypothesis H2 implies that we should allocate the maximum learning capacity to the primary branch because we care only about the primary task performance . Hypothesis H3 : Auxiliary supervision at the lower hierarchy yields better primary MTL as compared to flat-MTL . Having the auxiliary tasks at the same supervisory level as the primary task is inherently sub-optimal because we care only about the performance of the primary task ( Mordan et al. , 2018 ) . To prioritize the primary task , we could change the model architecture such that the auxiliary supervision is at the lower hierarchy than the primary supervision , which will be discussed in the next section . 3 MODEL ARCHITECTURE . 3.1 FLAT-MTL HIERARCHICAL ATTENTION MODEL . We start with an introduction of the Hierarchical Attention Model ( HAN ) ( Yang et al. , 2016b ) . We chose HAN because of its easy interpretability as it only uses single-head attention layers . There are four parts to the HAN model , ( 1 ) text input , ( 2 ) word encoder , ( 3 ) talkturn encoder , and ( 4 ) the predictor . In our application , we perform our predictions at the talkturn-level for both IEMOCAP and SEMAINE . For notation , let si represent the i-th talkturn and wit represent the t-th word in the i-th talkturn . Each single talkturn can contain up to T words , and each input talkturn can contain up to L past talkturns to give content context ( to discuss in section 4.2 ) . Text Input Given a talkturn of words , we first convert the words into vectors through an embedding matrix We , and the word selection one-hot vector , wit . Word encoder The word encoder comprises of bidirectional GRUs ( Bahdanau et al. , 2014 ) and a single head attention to aggregate word embeddings into talkturn embeddings . Given the vectors xit , the bidirectional GRU reads the words from left to right as well as from right to left ( as indicated by the direction of the GRU arrows ) and concatenates the two hidden states together to form hit . We then aggregate the hidden states into one talkturn embedding through the attention mechanism . uit is the hidden state from feeding hit into a one-layer perceptron ( with weights Ww and biases bw ) . The attention weight ( αit ) given to uit is the softmax normalized weight of the similarity between itself ( uit ) and uw , which are all randomly initialized and learnt jointly . xit = Wewit , t ∈ [ 1 , T ] uit = relu ( Wwhit + bw ) −→ h it = −−−→ GRU ( xit ) , t ∈ [ 1 , T ] αit = exp ( u > ituw ) Σtexp ( u > ituw ) ←− h it = ←−−− GRU ( xit ) , t ∈ [ T , 1 ] si = Σtαituit hit = ( −→ h it , ←− h it ) Talkturn encoder With the current and past talkturn embeddings ( content context , to discuss in section 4.2 ) , the talkturn encoder aggregates them into a single talkturn representation ( v ) in a similar fashion , as shown below . −→ h i = −−−→ GRU ( si ) , i ∈ [ 1 , L ] ui = relu ( Wshi + bs ) ←− h i = ←−−− GRU ( si ) , i ∈ [ L , 1 ] αi = exp ( u > i us ) Σiexp ( u > i us ) hi = ( −→ h i , ←− h i ) v = Σiαiui The simplest way of adding the sixteen ( four auxiliary targets from four families of auxiliary supervision ) auxiliary task predictors would be to append them to where the primary task predictor is , as illustrated in Fig . 2 . That way , all predictors use the same representation v. We refer to this architecture as flat-MTL because the auxiliary supervision is at the same level as the primary supervision . We are unable to test H2 and H3 using this architecture .

Summary: This paper addresses challenges faced in the multi-task learning (MTL) models used in analyzing multimodal conversational data. The main challenge paper is trying to solve is on how to select relevant auxiliary tasks that avoid negative transfer. The authors explore how the preprocessed data used for feature engineering can be re-used as auxiliary tasks in the model. The authors identified sixteen relevant auxiliary tasks, identified a method to distribute learning capacity between primary and auxiliary tasks and proposed a relative supervision hierarchy between primary and auxiliary tasks. An extensive set of experiments are conducted to show the effectiveness of the approach.