Reusing Preprocessing Data as Auxiliary Supervision in Conversational Analysis
Authors: Joshua Yee Kim
Kalina Yacef
Venue:  (2021)
Keywords: Multitask Learning
Multimodal Conversational Analysis

Abstract: Conversational analysis systems are trained using noisy human labels and often require heavy preprocessing during multi-modal feature extraction. Using noisy labels in single-task learning increases the risk of over-fitting. However, auxiliary tasks could improve the performance of the primary task learning. This approach is known as Primary Multi-Task Learning (MTL). A challenge of MTL is the selection of beneficial auxiliary tasks that avoid negative transfer. In this paper, we explore how the preprocessed data used for feature engineering can be re-used as auxiliary tasks in Primary MTL, thereby promoting the productive use of data in the form of auxiliary supervision learning. Our main contributions are: (1) the identification of sixteen beneficially auxiliary tasks, (2) the method of distributing learning capacity between the primary and auxiliary tasks, and (3) the relative supervision hierarchy between the primary and auxiliary tasks. Extensive experiments on IEMOCAP and SEMAINE data validate the improvements over single-task approaches, and suggest that it may generalize across multiple primary tasks.

Under review as a conference paper at ICLR 2021
REUSING PREPROCESSING DATA AS AUXILIARY SU- PERVISION IN CONVERSATIONAL ANALYSIS
Anonymous authors
Paper under double-blind review
ABSTRACT
Conversational analysis systems are trained using noisy human labels and often require heavy preprocessing during multi-modal feature extraction. Using noisy labels in single-task learning increases the risk of over-fitting. However, auxiliary tasks could improve the performance of the primary task learning. This approach is known as Primary Multi-Task Learning (MTL). A challenge of MTL is the se- lection of beneficial auxiliary tasks that avoid negative transfer. this paper, we explore how the preprocessed data used for feature engineering as auxiliary tasks in Primary MTL, thereby promoting the productive use of data in the form of auxiliary supervision learning. Our main contributions are: (I) the identification of sixteen beneficially auxiliary tasks, (2) the method of dis- tributing learning capacity between the primary and auxiliary tasks, and (3) the relative supervision hierarchy between the primary and auxiliary tasks. Extensive experiments on IEMOCAP and SEMAINE data validate the improvements over single-task approaches, and suggest that it may generalize across multiple primary tasks.
be re-used
INTRODUCTION
The sharp increase in uses of video-conferencing creates both a need and an opportunity to better understand these conversations (Kim et al. 2019a). In post-event applications, analyzing conversa- tions can give feedback to improve communication skills (Hoque et al. 2013) Naim et al. (015)). In real-time applications, such systems can be useful in legal trials, public speaking, e-health services, and more (Poria et al. 2019 Tanveer et al. 2015).
Analyzing conversations requires both human expertise and a lot of time, which is what many mul- timodal conversational analysis systems are trying to solve with automation. However, to build such systems, analysts often require training set annotated by humans (Poria al.) 2019). The annota- tion process is costly, thereby limiting the amount of labeled data. Moreover, third-party annotations on emotions are often noisy. Noisy data coupled with limited labeled data increases the chance of overfitting (James et al.) 2013).
From the perspective of feature engineering to analyze video-conferences, analysts often employ bre-built libraries (Baltrušaitis et al. 2016 Vokaturi 2019) to extract multimodal features inputs to training. This preprocessing phase is often computationally heavy, and the resulting features only used as imputs. In this paper, investigate how the preprocessed data can be re-used as auxiliary tasks in Primary Multi-Tass Learning (MTL), thereby promoting more productive use of data, in the form of auxiliary supervised learning. Specifically, identification of heneficially auxiliary tasks, (2) the method of distributing learning capacity between the primary and auxiliary tasks, and (3) the relative supervision hierarchy between the primary and auxiliary tasks. We demonstrate the value of our approach through predicting emotions on two publicly available datasets, IEMOCAP (Busso et al. 2008) and SEMAINE (McKeown al. 2011
main contributions are (1) the
2 RELATED WORKS AND HYPOTHESES
Multitask learning has a long history in machine learning (Caruana 1997). In this paper, we focus on Primary MTL. a less commonly discussed subfield within MTL. MModan al. 2018). Primary

---

Under review as a conference paper at ICLR 2021
MTL is concerned with the performance one (primary) task the sole motivation of adding auxiliary tasks is to improve the primary task performance.
In recent years, primary MTL has been gaining attention in computer vision YYo ct al.
2018 Fariha
2016 Yang et al. 2018; Mordan et al.
2018 Sadoughi & Busso 2018), speech recognition (Krishaa
Tao & Busso [2000] BBlll et al. Chen al. 2014), and natural al. 2018 Chen & Mak 2015|
language processing (NLP) (Arora et al. [019 Yousif et al. 2018] Zalmout & Habash (019 Yang
ct al. 2019 Du et al. 2017). The benefit of adding multiple tasks is to provide inductive bias through
multiple noisy supervision (Caruana 1997; Lipton el al. 2013 Ghosn & Bengio, 1997). On the
other hand, the drawback of adding multiple tasks increases the risk of negative transfer Torrry & Shavlik] [2010) Lee et al., [2016] 2018 Liu et al.] 2019; Simm et al.] (014)) which leads to many design considerations. Two of such considerations are, identifying (a) what tasks are beneficial and (b) how much of the model parameters to share between the primary and auxiliary tasks. In addition, because we are performing Primary MTL., we have the third consideration of (c) whether we should prioritize primary supervision by giving it a higher hierarchy than the auxiliary supervision.
In contrast with previous MTL works, our approach (a) identifies sixteen beneficially auxiliary tar- gets, (b) dedicates primary-specific branch within the network, and (c) investigates the efficacy and generalization of prioritizing primary supervision across eight primary tasks.
Since our input representation is fully text-based, we dive deeper into primary MTL in the NLP community. Regarding model architecture designs for primary MTL in NLP, Søgaard & Goldberg (2016) found that lower-level tasks like part-of-speech tagging, are better kept at the lower layers, enabling the higher-level tasks like Combinatory Categorical Grammar tagging to use these lower- level representations. In our approach, our model hierarchy is not based on the difficulty of the tasks, but more simply, we prioritize the primary task. Regarding identifying auxiliary supervisors in NLP, existing works have included tagging the input text (Zalmout & Habash 2019 Yang et al., 2019 Søgaard & Goldberg 2016). Text classification with auxiliary supervisors have included
research article classification (Du et al., 2017] Yousif et al., 2018), and tweet classification (Arora
et al. 2019). There is large body of work in multimodal sentiment analysis, but not in the use of multimodal auxiliary supervisors, as detailed in the next paragraph.
Multimodal analysis of conversations has been gaining attention in deep learning research, particu- larly for emotion recognition in conversations (Poria et al. 2019). The methods in the recent three years have been intelligently fusing numeric vectors from the text, audio, and video modalities be- fore feeding it to downstream layers. This approach is seen in MFN (Zadeh et al. 2018a). MARN (Zadeh al., 20188), CMN (Hazarika et al. 2018b), ICON (Hazarika al. 2018a), DialogueRNN MMjuudder et al. 2019), and M3ER (Mittal et al.) 2020). Our approach is different in two ways. (1) Our audio and video information is encoded within text before feeding only the text as input. Hav- ing only text as input has the benefits of interpretability, and the ability to present the conversational analysis on paper (Kim et al. 2019b). This is similar to how the linguistics community performs manual conversational analysis using the Jefferson transcription system (Jefferson] 2004). where the transcripts are marked up with symbols indicating how the speech was articulated. (2) Instead of using the audio and video information as only inputs to a Single Task Learning (STL) model, the
contribution of this paper that we demonstrate how to use multimodal information in both input and as auxiliary supervisors to provide inductive bias that helps the primary task.
Hypothesis H1:
The introduced set of auxiliary supervision features improves primary MTL. We introduce and motivate the full set of sixteen auxiliary supervisions, all based on existing litera- ture; these are grouped into four families, each with four auxiliary targets. The four families are (1) facial action units, (2) prosody, (3) historical labels, and (4) future labels:
(1) Facial action units, from the facial action coding system identifies universal facial expressions of
emotions (Ekman 1997p. Particularly, AU 05, 17, 20, 25 have been shown to be useful in detecting depression (Yang et al. 20166a Kim et al. 2019b) and rapport-building (Anonymous) 2021).
(2) Prosody, the tone of voice happiness, sadness, anger, and fear can project warmth and at- titudes (Hall et al. 2009), and has been used as inputs in emotions detection (Garcia-Garchh et al.) 2017).
(3 and 4) Using features at different historical time-poine is a common practice in statistical learn- ing, especially in time-series modelling (Christ et al. 2018). Lastly, predicting future labels as auxiliary tasks can help in learning (Caruana et al., 1996; Cooper et al. [2055] TTinh et al. 2018 Zhu et al. 2020. Shen et al. 2020). Inspired by their work, we propose using historical and future
2

---

Under review as a conference paper at ICLR 2021
(up to four talkturns ago later) target labels auxiliary targets. Although historical and future tar- get labels can be used as gre-training objective and fine-tuned on the current target label, sequential transfer learning not the focus of this paper.
MONAH Framework: Feature Engineering
kebbll
prosody
actions
START: Verbatim transcripts Videos
Verbattm
Transcriph
Speech Rate
Happy/SaclAmpe
Tone
Delay
Laughie
Nodding
Fonwar Trunk
Leaning
Smiling
PosiFasd Fabe
A 17 7.20,,25
Compute Percentle Rank
Back pack-propagation
Primary Task
Text
Input
Model
Loss Weighting
Auxiliary Actions
Narrative
Auxiliary Prosody
Auxiliary Historical Future
Predict
Figure 1: Reusing components (dotted lines) the feature engineering process as auxiliary targets (in blue). The MONAH framework (Anonymous 2021 introduced in section 4.2
Given that we are extracting actions and prosody families as inputs, we propose explore whether they can be reused supervisors (see Fig. [ Our hypothesis H1 is that re-using them auxiliary supervision improves primary MTL. This is related using hints the existing MTL literature, where the auxiliary tasks promote the learning of the feature (Cheng al. 2015 Yu & Jiang 2016)-
When the primary branch given maximum learning capacity, it would not be outperformed by models with primary branch having less than the maximum learning capacity.
Hypothesis H2:
Deeper models with higher learning capacity produce better results (Huang et al. 2019 Nakkiran et al. [2019; Menegola et al. 2017, [Blumberg et al. 2018 Romero et al. 2014). Also, since the auxiliary branch shared with the primary supervision, the auxiliary capacity should be limited to enforce information transfer and improve performance (Wu et al. 2020). Therefore, given a fixed learning capacity budget, our hypothesis H2 implies that we should allocate the maximum learning capacity to the primary branch because we care only about the primary task performance.
Hypothesis H3: Auxiliary supervision the lower hierarchy yields better primary MTL as compared to flat-MTL.
Having the auxiliary tasks the same supervisory level as the primary task inherently sub-optimal because care only about the performance of the primary task (Mordan al.) 2018). To prioritize the primary task, we could change the model architecture such that the auxiliary supervision is at the lower hierarchy than the primary supervision, which will be diseussed in the next section.
MODEL ARCHITECTURE
3.1 FLAT-MTL HIERARCHICAL ATTENTION MODEL
We start with an introduction of the Hierarchical Attention Model (HAN) (Yang et al. 2016b). We chose HAN because of its easy interpretability as only uses single-head attention layers. There four parts to the HAN model, (1) text input, (2) word encoder, (3) talkturn encoder, and (4) the predictor. our application, we perform our predictions at the talkturn-leve for both IEMOCAP and SEMAINE. For notation, let s; represent the i-th talkturn and uae represent the 1-th word in the i-th talkturn. Each single talkturn can contain up to T words, and each input talkturn can contain up to L past talkturns to give content context (to discuss in section 4.2)
Text Input Given a talkturn words, we first convert the words into vectors through an embedding matrix W... and the word selection one-hot vector, Wit-

---

Under review as a conference paper at ICLR 2021
Word encoder The word encoder comprises of bidirectional GRUs (Bahdanau et al. 2014) and single head attention to aggregate word embeddings into talkturn embeddings. Given the vectors Tit the bidirectional GRU reads the words from left to right as well as from right to left (as indicated by the direction of the GRU arrows) and concatenates the two hidden states together to form has We then aggregate the hidden states into talkturn embedding through the attention mechanism. uat is the hidden state from feeding hiz into a one-layer perceptron (with weights W., and biases bw). The attention weight (Oar) given uit the softmax normalized weight of the similarity between itself (u,²) and Uws which are all randomly initialized and learnt jointly.
Fu = Wown E [1,T)
Ust = relu (oa whe + bw) exp(uffuu)
h i GRÚ (ou), [1,7]
Oit Ejerp(u,(uw)
To a (RÜ()a)) [T,1]
Eoaulit
h4 = -h it h:))
Talkturn encoder With the current and past talkturn embeddings (content context, to discuss in section(4.2). the talkturn encoder aggregates them into single talkturn representation (v) a similar fashion, shown below.
h. GRÜ (8.), 4]
U1 relu(W, + ba)
h GRÜ (si), i [L.1]
explu, U.)
04 Eerppu Us)
h; (T T
Eçam
The simplest way of adding the sixteen (four auxiliary targets from four families of auxiliary super- vision) auxiliary task predictors would be to append them to where the primary task predictor is, as illustrated in Fig. 2 That way, all predictors use the same representation e. We refer to this architec- ture flat-MTL because the auxiliary supervision is the same level the primary supervision. We are unable to test H2 and H3 using this architecture.
Aux Predictor 16
Aux Predictor
Input
Telkoun
Primary Predictor
Figure 2: Forward pass the Flat-MTL HAN architecture. Auxiliary tasks (yellow) are added the same level of the primary task (orange).
3.2 HAN-ROCK
We adapted the ROCK architecture (Mordan al. 2018) which was built for Convolutional Neural
Networks (LeCun et al. 1995) found in ResNet-SSD (He et al. 2016: Liu et al. 2016) to suit GRUs
(Bahdanau et al. 2014) found in HAN (Yang al. 201 (see Fig. 3).
To study H3, we bring the auxiliary task predictors forward, so that the auxiliary supervision is of
lower hierarchy than primary supervision (see Fig. 3.. It is of lower hierarchy because the back-
propagation from the primary supervision is able temper the back-propagation from the auxiliary
"mplemmnntttin available https://gthhb.com http://gihuh.commaonnyynt com/anonyyoo///lccchooees Please attached supplemen- tary material for implementation details during the review phase.

---

Under review as a conference paper at ICLR 2021
Rond
Flatón Meack
Asa Tredeter 15
Ausraryy Shina tre.
Agn PicckIH
Atemton
Paraay Piedea
Toxt hast
Aay Tekoun Encader
Prinary That wien Choodir
Prisary Teak Talt Chooder
Figure 3: Forward pass of the HAN-ROCK architecture. There is primary branch (orange) where auxiliary supervision (yellow) can not influence. The fusion module (blue) aggregates the talkturn embeddings from all tasks into one.
supervision but not vice-versa. This also sets up to study H2, the impact of distributing different learning capacities to the auxiliary and primary branch. Each the auxiliary tasks has its own talkturn encoder but shares one word encoder in the auxiliary branch (to keep the network small). Subscript a indicates whether the word encoder for the primary or auxiliary branch:
Fa Wawn, [1,
Uait relu (Wa Wawhait + Daw)
ait GRU (zu), [1,T], e {pri, auz)
Qat Eperpl EeerplWastt Haw)
To eit GRU, (au), 11, (pri,auz)
Sei Ei Moit
haa (h ast, h ast)
Each task has its talkturn encoder. Subscript indicates which of the seventeen tasks the primary talkturn task or one of the sixteen auxiliary tasks - is the talkturn encoder is dedicated to:
ho abi GRUas (aa), [L, L] e {pri, aux), b e {pri, aual, aux2, aux 16}
To Jati GRUns(sat)) E [L, 1], E {pri, aux), E {pri, aual, aux2, wu2 16}
exp ub)
habi = abis T abi)
aabi
Ejexp( u ubx
Uahi = relu(Wshaan + bn)
Vah
Ej Unbi
The seventeen talkturn embeddings (Vas) goes through a concatenation, then the single head atten- tion. aggregating talkturn embeddings across seventeen tasks into one talkturn embedding for the primary task predictor. Subscript pertains to the fusion module.
concatenation: U. (Vab), a E {pri, aux), b e {pri, aurl, aux2, ..., our16)
explvt (epluTue) uc)
attention: Ca Exerplu. ue)
overall primary talkturn vector: Eeague
EXPERIMENTS
4.1 DATA AND PRIMARY TASKS
We validate our approach using two datasets with a total eight primary tasks: the IEMOCAP (Busso al. 2008) and the SEMAINE (McKeown al. 2011) datasets. Both datasets used

---

Under review as a conference paper at ICLR 2021
in multimodal emotions detection research (Poria et al., 2019). We divided the datasets into train, development, and test sets in an approximate 60/20/20 ratio such that the sets do not share any speaker (Appendix A.1 details the splits).
The target labels of the eight primary tasks are all at the talkturn-level. The four primary tasks of IEMOCAP consists of the four-class emotions classification (angry, happy, neutral, sad), and three regression problems valence (1-negative, 5-positive), activation (1-calm, 5-excited), and domi- nance (I-weak, 5-strong). The four-class emotions classification target is common for IEMOCAP (Latif et al. 2020 Xia & Liu 2015] Li et al. 2019 Hazarika et al. 2018b; Mittal et al. 2020), albeit not universal. Some researchers have gone up to five (Chang & Scherer 2017) or six (Majumder et al.. 2019 Hazarika et al. (2118) or nine-class emotions classification (Zadeh et al. (0188) target.
For SEMAINE, there are four regression problems- activation, intensity, power, valence. We note that the valence, power, and activation tasks might be related across the two datasets, but cross- domain learning is beyond the scope of this paper. We use two standard evaluation metrics, mean absolute error (MAB), and 4-class weighted mean classification accuracy, MA(4).
4.2 INPUT
Multimodal feature extraction is computed using the MONAH framework (Anonymous) 2021). This framework uses variety of pre-trained models to extract nine multimodal features, associated with the prosody of the speech and the actions of the speaker, and weaves them into a multimodal text narrative. We refer the reader to Anonymous (2021) for the details and efficacy of the MONAH framework. The benefit of the created narrative is that it describes what is said together with how it is said for each talkturn, giving richer nonverbal context to the talkturn (see Fig. 4 for an example). Being fully text- text-based means that the analysis product can be printed out on paper, without the need for speakers nor monitors replay the conversation on a computer.
Verbatim:
"The woman said no."
MONAH:
"The woman raised chin. The woman sadly and slowly said no."
Figure 4: Example of MONAH transcript.
In addition to nonverbal context, we concatenated variable number of preceding talkturns to the current talkturn as content context. Content context has been proven to be useful in CMN (Haz-) arika et al., 2018b) and ICON (Hazarika et al. 8a), DialogueRNN (Majumder et al. 2019). The content-context size is tuned as hyperparameter. The resulting multimodal text narrative, consist- ing of both nonverbal and context context, used as the sole input to the model. The impact of progressively removing nonverbal and content context is not contribution of this paper and hence not analyzed.
4.3 AUXILIARY TARGETS
We first clarify the method of extraction for the auxiliary families. The OpenFace algorithm (Bal- frušaitis et al. (016) is used to extract the four continuous facial action units (AU) - AU 05, 17, 20, 25. The Vokaturi algorithm (Vokaturi) 2019) is used to extract the four continuous dimensions in the tone of voice - happiness, sadness, anger, and fear. As for historical and future features, we simply look up the target label for the past four talkturns, and future four talkturns. If any of the historical or future labels are not available. for example, the target label four talkturns ago is not available for the third talkturn, we substitute with the next nearest non non-missing label.
For all auxiliary targets that reused the input features (actions and prosody), we converted them into a percentile rank that has the range 10.1] using the values from the train partition. This is subtle but note- note-worthy transformation. When reusing an input as an auxiliary target, it would be trivial if the input can easily predict the target. For example, given the following MONAH transcript as input, "The woman sadly and slowly said no." It would be trivial to use a binary (quantized) auxiliary target of "was the tone sad?" because we would only be training the model to look for the word "sadly". However, if the auxiliary target is a percentile rank (less quantized) of the sadness in tone, then the presence of the word "sadly" increases the predicted rank, but the model could still use the rest of the nonverbal cues ("slowly") and what is being said ("no") predict the degree of sadness.

---

Under review as a conference paper at ICLR 2021
That way, representations learnt for the auxiliary tasks uses more of the input. Additionally, this transformation also helped us side-step the decision to use the multimodal streams exclusively in the inputs in the supervision (Caruana & De Sa 1997: Caruana & de Sa 1998) because could use them in both.
Percentile rank also has the convenient property of having the range [0,1]. We scaled the percentile ranks so that they all have the same range as the primary task (see appendix [A.2] for transformation details). This ensures that if we assigned equal loss weights all tasks, the contribution of every task (both auxiliary and primary) is of the same order of magnitude, an important consideration for MTL (Gong et al. 2019 Hassami & Haley 2019 Sener& & Koltun 2018
4.4 MODELS, TRAINING, AND HYPERPARAMETER TUNING
The overall loss is calculated as the weighted average across all seventeen tasks: (I) we picked random weight for the primary task from the range [0.50, 0.99]: this ensures that the primary task has the majority weight. (2) For the remaining weights (1 primary weight), allocated them to the mutual information, or (c) softmax- normalized mutual information. (a) is self-explanatory. As for (b) and (c), mutual information has been shown that it is the best predictor - compared to entropy and conditional entropy - of whether the auxiliary task would be helpful to primary MTL (Bjerval 2017). We computed the mutual information (vector m) of each auxiliary variable with the primary target variable (Kraskov) or sofumax normalized m to sum up to 1. Finally, we multiplied the normalized m with the remaining weights from (2); this ensures that the primary weight and the sixteen auxiliary weight sum up to one. (a), (b), and (c) have ten trials each during hyper-parameters tuning.
sixteen auxiliary tasks by: (a) random, (b)
al., 2004 Ross 2014) using scikit-learn (Pedregosa et al. 201T). Then, we linearly-
Two variants of the HAN architectures are used (Fig 2 and 3). Glove word embeddings (300- dimensions) are used to represent the words (Pennington et al.) 2014). Hyper-parameters tuning is crucial because different combinations of primary and auxiliary tasks require different sets of hyper- parameters. For hyperparameters tuning, we used random search (Bergstra & Bengio 2012) with thirty trials. We tuned the learning rate, batch size, L2 regularization, the number of GRUs assigned to the primary and auxiliary branches, the auxiliary weights assignment, the content-contex size, and lastly the GRU dropout and recurrent dropout (as detailed in Appendix(A.3)
Training is done RTX2070 via the median-stopping rule (Golovin et al. 2017) after the fifth epoch and after every two epochs (i.e., epoch number 5, 7, 9, .... 349). Appendix A.4 details the hyperparameters of models that performed the best on the development set.
V100 GPU, for up 350 epochs. Early stopping is possible
For hypotheses testing, we bootstrapped confidence intervals of the test performance of both baseline and challenger models as well as the confidence intervals of the differences in performances. Please see uppendix A.5 for details on the bootstrap procedure.
RESULTS AND DISCUSSION
The key takeaways are: (H1) The introduced set of auxiliary supervision improves primary MTL significantly in six of the eight primary tasks. (H2) Maximum learning capacity should be given to the primary branch as a default. (H3) HAN-ROCK is unlikely (in one of the eight tasks) to degrade primary MTL significantly, and sometimes significantly improves it (in four of the eight tasks).
(H1): To test H1 (whether the introduced set of auxiliary supervision improves Primary MTL), we first train the model with all sixteen auxiliary targets (from families actions, prosody, historical, and future). Then, to differentiate the effect from the historical and future supervision, we set the loss weights from historical and future targets to be zero; effectively, there is only supervision from eight auxiliary targets (actions and prosody). Lastly, for the baseline model (no auxiliary supervision), we set the loss weights from all sixteen auxiliary targets to zero.
Given auxiliary supervision, the model significantly outperforms the baseline of not having auxiliary supervision in six out of the eight primary tasks (Table D]. Comparing the baseline model with the model with two auxiliary target families, they significantly outperformed the baseline model in five out of eight primary tasks. The addition of two auxiliary target families (historical and
7

---

Under review as a conference paper at ICLR 2021
Table 1: H1 Results. *; the model performance has statistically significant difference with the baseline model. a: action, prosody, h: historical labels. f: future labels.
IEMOCAP
SEMAINE
Aux. Target
Classif.
Val.
Act.
Dom. MAE
Act. MAE
Int.
Pow. MAE
Val. MAE
MA(4)
MAE
MAE
MAE
None 0.625
0.527
0.518
0.667
0.194
0.238
0.170
0.177
(Baseline)
ap 0.715
0.538
0.507
0.600*
0.184
0.218
0.167*
0.178
aphf
0.706"
0.497*
0.504
0.587*
0.187
0.231
0.165*
0.169
future labels) sometimes significantly improved primary MTL (valence in IEMOCAP), but it also sometimes significantly made it worse (activation and intensity in SEMAINE). This shows that the value of auxiliary tasks, and the associated risk of negative transfer, depends on the auxiliary task.
(H2): To test H2 (whether maximum learning capacity should be given to the primary branch), we let P represent the number of GRU assigned to the primary talkturn encoder, and represent the number of GRU assigned to each of the sixteen auxiliary talkturn encoder. We constrained P A be equal to 257. During our experiments,
set P to 64, 128, 192, and 256. We set 256 as the baseline model because is the maximum learning capacity we can give to the primary branch while giving GRU (= 257 256) each of the sixteen auxiliary talkturn encoders.
Table 2: H2 Results. *; the model performance has statistically significant difference with the baseline model (P = 256). " Assigning 1 GRU to the auxiliary task talkturn encoder yields
statistically significant difference with assigning 0 GRU.
IEMOCAP
SEMAINE
Classif.
Val.
Act.
Dom.
Act.
Int.
Pow. MAE
Val. MAE
Pri. GRU
MA(4)
MAE
MAE
MAE
MAE
MAE
256
0.715
0.538
0.507
0.587
0.187
0.231
0.165*
0.169
(Baseline)
192
0.736
0.509
0.518
0.604
0.187
0.228
0.165
0.184
128
0.711
0.537
0.512
0.597
0.189
0.216
0.176*
0. 0.196
0.687
0.540
0.507
0.593
0.191
0.234
0.167
0.192
0.656
0.554
0.509
0.599
0.190
0.229
0.168
0.191
In all primary tasks, the baseline model of assigning 256 GRUs to the primary branch is not sig- nificantly outperformed by models that assigned 1, 64, 128, 192 GRUs (Table 2). Generally, the performance decreased the number of GRUs assigned to the primary talkturn encoder decreased from 256 to 1. We observed significantly worse performance in two out of eight tasks in power and valence in SEMAINE. Also, assigning 256 GRU the primary talkturn encoders and each of the sixteen auxiliary talkturn encoders yields the smallest mode and thus trains the fastest. Therefore, recommend that the maximum capacity be given to the primary branch default.
That said, the presence of an auxiliary branch still important. The baseline of H1 (no auxiliary supervision, Table I can be approximated as P=256 + 16 A=0 because the model architecture is the same, except that the loss weights of all auxiliary tasks are zero. We compared the former to the baseline Table and found that four out of eight primary tasks have significant improvements by changing the number of talkturn encoders assigned to each auxiliary task from zero to one.
(H3): To test H3 (whether auxiliary supervision should be given a lower hierarchy). we compare the results from the flat-MTLL HAN architecture (baseline) against the HAN-ROCK architecture (Table B.. Placing auxiliary supervision at the lower hierarchy significantly improves primary MTL in four out of eight tasks. In only one out of eight tasks (power in SEMAINE), auxiliary supervision significantly degrades primary MTL. We posit that further improvements are possible through the fusion module with future research.
As opposed to, say, assigning GRU to the primary talkturn coder and 256 GRU to each of the sixteen auxiliary talkturn encoder.
8

---

Under review as a conference paper at ICLR 2021
Table 3: H3 Results. *; the model performance has a significant difference with the baseline.
IEMOCAP
SEMAINE
Classif.
Val.
Act. MAE
Dom. MAE
Act.
Int.
Pow. MAE
Val.
Hierarchy
MA(4)
MAE
MAE
MAE
MAE
Flat (Baseline)
0.699
0.520
0.526
0.606
0.183
0.230
0.164
0.185
HAN-
0.715
0.538
0.507
0.600*
0.184
0.218
0.167*
0.178
ROCK
5.1 CLASS-WISH PERFORMANCE AND STATE-OF-THE---
discuss the IEMOCAP classification model in depth by investigating the class-wir wise performance under the three hypotheses (Table 4. Generally, we found that all hypotheses effects are stronger in lower resource labels (sad and anger). We also present the performance 2020). previous state-of-the-an (SoTA) approach. We do not expect the performance of text- only input to match the SoTA approach, which is confirmed in Table By fusing numerical vectors from the three modalities prevalent in SoTA approaches (Zadeh et al.) 2018a b Hazarika et al., 2018b]a Majumddr et al. 2019; Mittal et al.] (2200), the inputs are of a much higher granularity as compared our approach of describing the multimodal cues using discrete words. Although the text-based input is likely to constrain model performance, the multimodal transcription could be helpful for a human to analyze the conversation even before supervised learning- We could also overlay the model perspective on the multimodal transcription to augment human analysis (see Appendix A.6
M3ER (Mittal et al.]
Table 4: Class-wise classification F1 score IEMOCAP. Baseline (challenger) refers HAN-Rock architecture under the three hypotheses. *; the challenger performance has a statistically significant difference with the baseline model. Distribution
H2
H1
SoTA
Challenger
Baseline Aux target:
Challenger Aux target:
Baseline Pri
Baseline
Challenger Pri GRU:
Hier-
Hier-
M3-
Label
Count
archy:
GRU:
archy:
ER
1
HAN
None
aphf
256
Flat
ROCK
Sad
1,084
0.573
0.689*
0.699
0.591
0.674
0.704
0.775
Anger
1,103
0.531
0.683
0.752
0.672*
0.657
0.720*
0.862
Happy
1,636
0.772
0.784
0.776
0.754*
0.804
0.806
0.862
Neutral
1,708
0.664
0.636
0.688
0.627*
0.645
0.631
0.745
6 CONCLUSION, LIMITATIONS, AND FUTURE WORK
We proposed to re-use feature engineering pre-processing data as auxiliary tasks in primary MTL. Three hypotheses were tested for primary MTL. The experimental results confirm H1 Introducing our set of sixteen auxiliary supervisors resulted in better performance in most primary tasks. For H2, maximum learning capacity should be given the primary branch. Lastly, for H3, placing the auxiliary supervision in lower hierarchy is unlikely to hurt performance significantly, and it some- times significantly improves performance. This is encouraging news for multi-men tonti--moall conversational analysis systems we have demonstrated how pre-processed data can be used twice to improve performance, once as inputs, and again as auxiliary tasks. This paper has limitations. The first limi- tation is that the solutions not clear if these would generalize outside this domain. The second limitation is that we have evaluated on HAN, but not other network architectures.
evaluated on eight tasks in the conversational analysis domain, and
challenge to be addressed is the apriori selection of the auxiliary targets. Future research could investigate targets selection, including how to much larger range of auxiliary targets, how to decide the optimum number auxiliary targets, and whether it is possible to perform these automatically.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anonymous. Monah:
Multimodal narratives for humans. Under review at EACL2021, 2021.
Udit Arora, William Scott Paka, and Tanmoy Chakraborty. Multitask learning for blackmarket tweet detection, Proceedings of the 2019 JEÉE/ACM International Conference on Advances in Social Networks Analysis and Mining. pp. 127-130, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv:1409 140.04474 2014.
Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. Openface: an open source fa- cial behavior analysis toolkit. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, pp. 1-10. IEEE, 2016.
Peter Bell, Pawel Swietojanski, and Steve Renals. Multitask learning of context-dependent targets in deep neural network acoustic models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(2):238-244 2016.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1):281-305, 2012.
Johannes Bjerva. Will my auxiliary tagging task help? estimating auxiliary tasks effectivity in multi- task learning. In Proceedings of the 2 Nordic Conference on Computational Linguistics, pp- 216-220, 2017.
Stefano Blumberg, Ryutaro Tanno, lasonas Kokkinos, and Daniel C Alexander. Deeper image quality transfer: Training low-memon, neural networks for 3d images. In International Confer- ence on Medical Image Computing and Computer-Assisted Intervention, pp. 118-125. Springer, 2018.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335, 2008.
Rich Caruana. Multitask learning, Machine learning, 28(1):41-75, 1997.
Rich Caruana and Virginia R De Sa. Promoting poor features to supervisors: Some inputs work better as outputs. In Advances in Neural Information Processing Systems, pp. 389-395, 1997.
Rich Caruana and Virginia R de Sa. Using feature selection to find inputs that work better extra outputs. In International Conference on Artificial Neural Networks, pp. 299-304. Springer, 1998.
Rich Caruana, Shumeet Baluja, and Tom Mitchell. Using the future to "sort out" the present: Rankprop and multitask learning for medical risk evaluation. In Advances in neural informa- lion processing systems, pp. 959-965, 1996.
Jonathan Chang and Stefan Scherer. Learning representations of emotional speech with deep con- volutional generative adversarial networks. In 2017 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 2746-2750. IEEE, 2017.
Dongpeng Chen and Brian Kan-Wing Mak. Multitask learning of deep neural networks for low- resource speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing. 23(7):1172-11883 2015.
Dongpeng Chen, Brian Mak, Cheung- -Chnng-Chiii Leung, and Sunil Sivadas. Joint acoustic modeling of triphones and trigraphemes by multi-task learning deep neural networks for low-resource speech recognition. In 2014 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5592-5596. IEEE, 2014.
Hao Cheng. Hao Fang, and Mari Ostendorf. Open-domain name error detection using multi- task mnn Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 737-746, 2015.
10

---

Under review as a conference paper at ICLR 2021
Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W Kempa-Liehn Time series feature extraction on basis of scalable hypothesis tests (tsfresh-a python package), Neurocomputiigg. 307: 72-77, 2018.
307:
Gregory Cooper, Vijoy Abraham, Constantin Aliferis, John M Aronis, Bruce G Buchanan, Richard Caruana, Michael Fine, Janine E Janosky, Gary Livingston, Tom Mitchell, et al. Pre- dieting dire outcomes of patients with community acquired pneumonia. Journal of biomedical informatics, 3873)6347-36, 2005.
Yongping Du, Yunpeng Pan, and Junzhong Ji. A novel serial deep multi-task learning model for large scale biomedical semantic indexing. In 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 533-537. IEEE, 2017.
Rosenberg Ekman. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS), Oxford University Press, USA, 1997.
Anna Fariha. Automatic image captioning using multitask learning. In In the Proceedings of Neural Information Processing Systems, volume 20, pp. 11 11-20, 2016.
Jose Maria Garcia-Garcia, Victor MR Penichet, and Maria D Lozano. Emotion detection: technol- ogy review. In Proceedings of the XVIII international conference on human computer interaction, pp. 1-8, 2017.
Joumana Ghosn and Yoshua Bengio. Multi-task learning for stock selection. In Advances in neural information processing systems, pp. 946-952, 1997.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. Google vizier: service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1487-1495, 2017.
Ting Gong. Tyler Lee, Cory Stephenson. Venkata Renduchintala, Suchismita Padhy, Anthony Ndi- rango, Gokce Keskin, and Oguz. H Elibol. A comparison loss weighting strategies for multi task learning in deep neural networks. IEEE Access, 7:141627-141632 2019.
Judith Hall, Debra Roter, Danielle C Blanch, and Richard M Frankel. Observer-rated rap- port in interactions between medical students and standardized patients. Patient Education and Counseling, 76(3):323-327, 2009.
Kaveh Hassani and Mike Haley. Unsupervised multi-task feature learning on point clouds. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8160-8171, 2019.
Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. Icon: Interactive conversational memory network for multimodal emotion detection. In Pro- ceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp- 2594 2604,, 2018a.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. Conversational memory network for emotion recognition in dyadic dia- logue videos. In Proceedings of the conference. Association for Computational Linguisties. North American Chapter. Meeting, volume 2018, pp. 2122. NIH Public Access, 2018b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Mohammed Hoque, Matthieu Courgeon, Jean Martin, Bilge Mutlu, and Rosalind W Picard. Mach: My automated conversation coach. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pp. 697-706, 2013.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pp- 103-112, 2019.

---

Under review as a conference paper at ICLR 2021
Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. An introduction statistical learning, volume 112. Springer, 2013.
Gail Jefferson. Glossary of transcript symbols with an introduction. Pragmatics and Beyond New Series, 125:13-38 3-34, 2004.
Joshua Y Kim, Rafael A Calvo, Kalina Yacef, and NJ Enfield. A review dyadic conversation visualizations-purroses,, data, lens of analysis. arXiv preprint arXiv:1905.00653 2019a.
Joshua Y Kim, Greyson Y Kim, and Kalina Yacef. Detecting depression in dyadic conversations with multimodal narratives and visualizations. In Australasian Joint Conference on Artificial Intelligence, pp. 303-314. Springer, 2019b.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E. 69(6):06613)) 138, 2004.
Kalpesh Krishna, Shubham Toshniwal, and Karen Livescu. Hierarchical multitask learning for ctc- based speech recognition. arXiv preprint arXiv:1807.06234. 2018.
Siddique Latif, Rajib Rana, Sara Khalifa, Raja Jurdak, Julien Epps, and Bjórn Wolfgang Schuller. Multi-task semi-supervised adversarial autoencoding for speech emotion recognition. IEEE Transactions on Affective Computing, 2020.
Yann LeCun, Yoshua Bengio, et Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361 1995.
Giwoong Lee, Funho Yang, and Sung Hwang. Asymmetric multi-task learning hased on task relat- edness and loss. In International Conference on Machine Learning, pp. 230-238, 2016.
Hae Beom Lee, Eunho Yang, and Sung Ju Hwang. Deep asymmetric multi-task feature learning. In International Conference Machine Learning, pp. 2956-29666 PMLR, 2018
Yuanchao Li, Tianyu Zhao, and Tatsuya Kawahara. Improved end-to-end speech emotion recog- nition using self attention mechanism and multitask learning. In Interspeech, pp. 2803-2807, 2019.
Zachary C Lipton, David C Kale, Charles Elkan, and Randall Wetzel. Learning to diagnose with Istm recurrent neural networks. arXiv preprint (ooe::511,.3677 2015.
Shengchao Liu, Yingyu Liang, and Anthony Gitter. Loss-balanced task weighting reduce negative transfer in multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9977-9978, 2019.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6818 6818-6825, 2019.
Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Mare Schroder. The semaine database: Annotated multimodal records emotionally colored conversations between a person and a limited agent. IEEE transuctions affective computing, 3(1):5-17, 2011.
Afonso Menegola, Michel Fornaciali, Ramon Pires, Flávia Vasques Bittencourt, Sandra Avila, and Eduardo Valle. Knowledge transfer for melanoma screening with deep learning. In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pp. 297-300. IEEE, 2017.
Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In AAAI, pp. 1359-1367, 2020.
12

---

Under review as a conference paper at ICLR 2021
Taylor Mordan, Nicolas Thome, Gilles Henaff, and Matthieu Cord. Revisiting multi-task learning with rock: a deep residual auxiliary block for visual detection. In Advances Neural Information Processing Systems, pp. 1310-13221 2018.
Iftekhar Naim, M Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque Automated pre- diction and analysis of job interview performance: The role of what you say and how you say it. In 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), volume 1, pp. 1-6. IEEE, 2015.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1542-1543, 2014. ISBN 9781937284961.
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. Emotion recognition con- versation: Research challenges, datasets, and recent advances. IEEE Access, 7:100943-100953, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Brian C Ross. Mutual information between discrete and continuous data sets. PloS one, 9(2):
c87357, 2014.
2014.
Najmeh Sadoughi and Carlos Busso. Expressive speech-driven lip movements with multitask learn- ing. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 409-415. IEEE, 2018.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems, pp. 527-538, 2018
Wei Shen, Xiaonan He, Chuheng Zhang, Qiang Ni, Wanchun Dou, and Yan Wang. Auxiliary-task based deep reinforcement learning for participant selection problem in mobile crowdsourcing. In Proceedings of the 29th ACM International Conference on Information & Knowledge Manage- ment, pp. 1355-13300 1 1364, 2020.
Jaak Simm, Ildefons Magrans de Abril, and Masashi Sugiyama. Tree-based ensemble multi-task learning method for classification and regression. IEICE TRANSACTIONS on Information and Systems, 1776-167771 2014.
Anders Søgaard and Yoav Goldberg. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 231-235, 2016
M Iftekhar Tanveer, Emy Lin, and Mohammed Hoque Rhema: A real-time in-situ intelligent inter- face to help people with public speaking. In Proceedings of the 20th International Conference Intelligent User Interfaces, pp. 286-295, 2015.
Fei Tao and Carlos Busso. End-to-end audiovisual speech recognition system with multitask learn- ing, IEEE Transactions Multimedia, 2020.
Lisa Torryy and Jude Shavlik. Transfer learning. In Handbook of research machine learning applications and trends: algorithms, methods, and techniques, pp. 242-264. IGI global, 2010.
Trieu H Trinh, Andrew M Dai, Minh- Minh-Thang Luong, and Quoc V Le. Learning longer-term depen- dencies in mns with auxiliary losses. arXiv preprint arXiv: 000..0003.00000 2018.
13

---

Under review as a conference paper at ICLR 2021
Vokaturi. Vokaturi Overview, 2019. gett.hng-ttattte/ovvee.o
URL https://dsveeooozzs.ooo vok.uuuurr....oo............. com/
Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information trans- fer in multi-task learning. arXiv preprint arXiv:2005.0009 2020.
Rui Xia and Yang Liu. multi-task learning framework for emotion recognition using 2d continuous space. IEEE Transactions on affective computing, 8(1):3-14, 2015.
Jianliang Yang, Yuenan Liu, Minghui Qian, Chenghua Guan, and Xiangfei Yuan. Information traction from electronic medical records using multitask recurrent neural network with contextual word embedding. Applied Sciences, 9(18)3665, 2019.
Le Yang, Dongmei Jiang. Lang He, Ercheng Pei, Meshia Cédric Oveneke, and Hichem Sahli. Deci- sion tree based depression classification from audio video and language information. In Proceed- ings of the 6th international workshop on audio/visual emotion challenge, pp. 89-96, 2016a.
Min Yang, Wei Zhao, Wei Xu, Yabing Feng. Zhou Zhao, Xiaojun Chen, and Kai Lei. Multitask learning for cross-domain image captioning. IEEE Transactions Multimedia, 21(4):1047- 1061, 2018
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1480-1489, 2016b.
ByungIn Yoo. Youngjun Kwak, Youngsung Kim, Changkyu Choi, and Junmo Kim. Deep facial age estimation using conditional multitask learning with weak label expansion. IEEE Signal Processing Letters, 25(6):808 -12-80888812.0 2018.
Abdallah Yousif, Zhendong Niu, and Ally Nyamawe. Citation classification using multitask con- volutional neural network model. In International Conference Knowledge Science, Engineer- ing and Management, pp. 232-243. Springer, 2018.
Jianfei Yu and Jing Jiang. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. Association for Computational Linguistics, 2016.
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. Memory fusion network for multi-view sequential learning. arXiv preprint 180.0007.. 2018a.
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateck Vij. Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehension. In Pro- ceedings of the... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intel- ligence, volume 2018, pp. 5642. NIH Public Access, 2018b.
Nasser Zalmout and Nizar Habash. Adversarial multitask learning for joint multi-feature and multi- dialect morphological modeling. arXiv preprint arXiv: 19/0. 12702, 2019.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self- supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10012-10022, 2020.
14

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DATASET PARTITIONS
We detail the dataset partition in | for aid reproducibility.
Table
Dataset partitions
IEMOCAP
SEMAINE
Partition
Session
Count
Partition
Session
Count
Train
SesOTF
861
Train
2008. 16.03.15
424
Train
Ses0IM
958
Train
2009.01.30.12.0000
404
Train
Ses02F
889
Train
1009.02.12.10.49..
686
Train
Ses02M
922
Train
2009.05.15.15.04
436
Train
Ses03F
958
Train
760
Train
Ses03M
1178
Train
668
Dev
Ses04F
1105
Train
2009.05.25.11
928
Dev
Ses04M
998
Train
2009.05.22 10.19.53
790
Test
SesÖ5F
1128
Train
2009.06.05. 10. 14.28
732
Test
Ses05M
1042
Train
2009.06. 15.12.1....1.1.
958
Train
2009.06.19.14.01..0
586
Train
440
Dev
2008..2..9..1.03.11
188
Dev
2009.01.06.14.53.
472
Dev
120..05.12.15.02.00
448
Dev
2009.05.05, 11.28.44
752
Dev
3009.06.26.14.33
404
Test
2008. 1008.12.14.1444..0
372
Test
2009.01.066 12.41.42
264
Test
1.09.01.28.15.55.11
364
Test
2009.06.26.14.386
440
Test
2009.06.26.. 14.09.45
448
A.2 SCALING THE AUXILIARY TARGETS
We detail the operations in scaling the percentile scores that range [0,1] to various primary tasks. For IEMOCAP Primary Tasks that are regression problems, we multiply the percentile score by and add 1 to obtain the range [1,5]. For IEMOCAP classification task, we leave the auxiliary targets the range [0,1]. As SEMAINE tasks, which are all regression problems, we multiply the percentile score by 2 and minus I to obtain the range [-1,1].
A.3 RANGE OF HYPER PARAMETERS TUNED
We document the range of hyper parameters tuned in Table 6 to aid reproducibility.
A.4
HYPER PARAMETERS WITH BEST DEVELOPMENT SET PERFORMANCE
We document our best performing hyper parameters Table 7 to aid reproducibility.
A.5 DETAILS OF COMPUTING THE BOOTSTRAP CONFIDENCE INTERVAL
Baseline models for each hypothesis are detailed section All non-baseline models are referred to challenger models. We created 1000 bootstrap samples of the test set performance by (1) resampling the development set performance, then (2) selecting the set hyperparameters that re- sulted in the best development set performance. and (3) looking up the test set performance given the set of best-performing hyperparameters for the development set. To judge whether the chal- lenger outperforms the baseline, computed the 95 percent confidence interval by (1) performing
15

---

Under review as a conference paper at ICLR 2021
Table 6: Range of Hyperparameters tuned. U: Uniform sampling, LU: Uniform sampling on the logarithmic scale.
Name
Min.
Max
Sampling
Learning rate
2 2-10
2 2-5
LU
Batch-Size
32
256
U
Pri GRU
64, 128, 192, 256
Aux GRU
257 (minus) Pri GRU
Loss-weights Assignment
Random, Linear-
U
L2 Regularization
0.0
0.50
LU
Content-Size
30
U
GRU dropout
0.01
0.50
U
Recurrent dropout
0.01
0,50
element-wiss subtraction between the resampled test set performance of the baseline against the challenger, (2) removing the top and bottom 2.5 percent from the differences, and (3) observing whether the remaining 95 percent confidence interval includes zero. it does not include zero, then the difference is statistically significant.
A.6 VISUALIZATION FROM HAN-ROCK
We demonstrate how the HAN-ROCK model could be used to support humans analyze conversa- tions using only text-based inputs. We visualized the attention weights from two models, (1) MTL refers to the classification model with auxiliary supervisors, whilst (2) STL refers to the same model architecture, but its auxiliary supervisors' loss weights are set to zero. In principle, the MTL model should exhibit attention weights that are less likely to overfit because the weights are tempered by auxiliary supervisors. We observe that across the two models, both use the historical talkturns more so than the current talkturn; secondly, both assign high attention to the second word of the talkturns, which is interesting because the second word where the multimodal annotations are inserted.
MTL (HAN-ROCK)
STL (HAN-ROCK)
mon said rually amondo
said really amanda
i-1
H
woman said ah don' be 5O stuoid
H
sciid oh don't stupid
I-2
smiled man soid need nol grand becouse you not happen want any momen
smiled mon sold need bocouse you do not hoppen to want any moment
i-3
H
woman said it: bocome habit with
N
sold its become habil with
mon sadly said you can hardly call three liquor glosses in
man sodly soid you con hardly three liquor g glauses whole evening going on and and
i-4
N
I
whole evening gatng on and on and on
i-5
worren said it seems so afly go on and on and on with wish thing
N womon said ms so sily to go on and and
with
Figure 5: Conversation analysis example. Both models predicted the class label (anger) correctly. The left-most column denotes the talkturn context - refers to the current talkturn where the target emotion class is predicted. The square boxes indicate the level of attention (N: None, L: Low, M: Medium, H: High) assigned to the talkturn. Within the talkturn, we also enlarge and darken the font color ti visualize higher attention weights.
there are three levels of attention over the internal representations, word (osz), talkturm (Qubt), and task (o.)). To compute the overall word and talkturn attention, compute the weighted average of as and Okabi using the on (task attention) as weights. Once we have the overall word and talkturn attention, we standardize the weights by computing the z-score. Depending on the z-score, we bucket the attention in none (z < 0), low (0 < z < 1), medium (1 < 2). or high (2 < z). We plan to validate the efficacy of the attention weights with human users in future research.
As explained in Section 3.2
16

---

Under review as a conference paper at ICLR 2021
Table 7: Best Hyperparameters settings for development set. R: Random, L: Linear-Normalized, S:
Softmax-Normalized.
TEMOCAP
SEMAINE
Classif.
Val.
Act.
Dom.
Act.
Int.
Pow.
Val.
MA(4)
MAE
MAE
MAE
MAE
MAE
MAE
MAE
Dev. set
0.714
0.482
0.492
0.570
0.118
0.164
0.135
0.138
Test
0.747
0.497
0.499
0.587
0.184
0.215
0.164
0.169
6.21
2.21
1.44
3.13
2.43
1,58
3.04
2.94
Learning rate
e-03
e-02
e-02
e-02
e-02
e-02
e-03
e-02
Batch size
43
43
62
77
41
43
33
62
Pri GRU
256
256
256
256
256
256
192
256
Aux GRU
65
1.2 regular-
2.25
1 1.43
2.41
0
0
ization
e-05
e-05
e-04
Content-Size
18
4
3
3
17
21
19
14
GRU
0.27
0.49
0.33
0.30
0.06
0.17
0.05
0.07
dropout
Recurrent
0.04
0.03
0.02
0.32
0.25
0.28
0.08
0.26
dropout
Epoch No.
232
161
74
14
48
116
110
110
Aux. weights
S
R
L
L
L
R
assignment
Main loss
0.840
0.560
0.810
0.610
0.870
0.940
0.950
0.920
weight
AU05 loss
0.009
0.013
0.014
0.000
0.000
0.002
0.002
0.008
weight
AU17 loss
0.009
0.027
0.009
0.003
0.002
0.004
0.002
0.003
weight
AU20 Toss
0.009
0.028
0.0
0.001
0.000
0.007
0.002
0.006
weight
AU25 loss
0.009
0.031
0.006
0.002
0.015
0.008
0.002
0.009
weight
Happy tone
0.010
0.023
0.038
0.006
0.021
0.013
0.002
0.003
loss weight
Sad tone loss
0.010
0.028
0.053
0.008
0.037
0.010
0.002
0.003
weight
Angry tone
0.010
0.016
0.034
0.006
0.055
0.011
0.002
0.000
loss weight
Fear tone
0.010
0.043
0.036
0.004
0.000
0.004
0.002
0.003
loss weight
Y loss
0.010
0.030
0.0
0.040
0.000
0.000
0.005
0.006
weight
Y-2 loss
0.011
0.022
0.0
0.049
0.000
0.000
0.004
0.004
weight
Ye-3 loss
0.010
0.006
0.0
0.046
0.000
0.000
0.003
0.006
weight
Y--4 loss
0.010
0.040
0.0
0.044
0.000
0.000
0.003
0.007
weight
Yo+1 loss
0.010
0.042
0.0
0.042
0.000
0.000
0.005
0.005
weight
Ye42 loss
0.011
0.0
0.0
0.052
0.000
0.000
0.004
0.006
weight
Ye+a loss
0.010
0.050
0.0
0.043
0.000
0.000
0.003
0.008
weight
Ye+4 loss
0.010
0.038
0.0
0.044
0.000
0.000
0.003
0.003
weight
17


Reviews:
[{"review_id": "vT7R7vTRDsn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how the preprocessed data can be reused as auxiliary tasks in primary multi-task learning (MTL) for the multimodal emotion detection task. The authors propose and test three hypotheses for primary MTL. Two different hierarchical-level models, FLAT-MTL hierarchical attention model and HAN-Rock model, are proposed to improve the performance of the primary MTL.     \n\nOverall, the definition of the task and the overall architecture of this paper are both clear and straightforward, making this paper easy to understand. To explore how to use the preprocessed data as auxiliary tasks in primary MTL, the authors present a nicely executed study and test three hypotheses. Besides, the proposed model achieves impressive results on two datasets in terms of the multimodal emotion detection task. \n\nHere are some of my questions and concerns for the paper:\n1) The contribution of this paper seems to be limited since the idea of using visual features, audio features, and context as auxiliary information is similar to lots of emotional classification models such as ICON, CMN. In addition, the proposed model for MTL is also straightforward.\n2) More related works about multi-task learning and multimodal emotion detection tasks should be included.\n3) More in-depth experimental analysis should be carried out. For example, what is the individual performance of models on four different emotions under the three hypotheses?\n4) Some symbols in formulas lack explanations. For instance, X_t,X_(t-1),X_(t-2) on page 2 and b_w,W_w,u_w,u_s on page 3 are all not clearly defined. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cCS6D8vToc6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses multi-task learning for multimodal emotion recognition on two existing datasets (IEMOCAP and SEMAINE). \nStrengths:\n*The issues addressed in this paper are very relevant. The use of a multi-task learning framework to tackle the lack of labeled data and the noisy labels for Affective computing research is a very interesting and still unexplored research line. \n\nWeaknesses:\n*This work is motivated by the analysis of video-conferencing videos which is indeed a crucial and topical issue. However, this motivation is a little bit heavy-handed as the processed data are very different from video-conferencing videos (face-to-face human interactions for IEMOCAP and human-agent/Woz interactions for SEMAINE database). \n*Contrarily to what is claimed in the abstract, the relevance of the auxiliary task is not investigated in the paper. The first type of auxiliary tasks is predicting outputs using external predictors (Open face in order to obtain action units activations from computer vision and Vokaturi in order to extract emotion categories from the prosody). The primary tasks (4-classes emotion classification using the same 4 emotions, and dimensions prediction) are very close to these auxiliary tasks.  The second type of auxiliary tasks is predicting contexts such as done in language models and could be considered unsupervised pre-training objectives used for learning representations (but it is not described like that). Thus, if we push the reflection a little bit further, H1  seems to validate that using external off-the-shelf predictors -that are predicting the same outputs as the ones of the primary tasks- improves the results of the proposed predictor, which is not so interesting.\n*The emotion categories that are considered are four of the Big-six of Ekman and I am wondering how relevant it is. For example, how often fear is observed in the SEMAINE and IEMOCAP datasets? \n*The method used to provide a multimodal representation of the data for the inputs seems interesting (as I understand it: extracting multimodal features and generating an augmented text containing multimodal narratives using MONAH, a previous system proposed by the authors) but it's difficult to understand what this method brings compared to the state of the art of learning multimodal representations. Besides, it will be interesting to discuss shortly the performance of MONAH on the two datasets of human-human interactions (maybe showing some outputs).\n*The proposed MTL framework relies on a combination of existing models (HAN and ROCK). Thus the contribution is rather experimental than methodological.\n*Considering the problem of multi-task learning requires identifying and differentiating tasks and domains and the paper fails in doing this. In Section 5, 8 tasks are identified while 2 datasets and 5 or 6 tasks are actually considered (emotion classif, valence prediction,activation prediction, Dominance/Power prediction) \n\n \nTypos :\n*abstract: it may generalizes\n*Section 3.2: indicateS\n*Section 4.1: valence\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EX1sY6EYN8K", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles conversational analysis problem and more specifically the Primary Multi-Task Learning.\n\nThe three hypotheses the paper tries to address are interesting and important. \n\n- Hypothesis H1: The introduced set of auxiliary supervision features improves primary MTL.\n\nIn this section, I am not quite sure if using future labels would cause information leakage to the primary task. If so, this would make the conclusion questionable.\n\n- Hypothesis H2: When the primary branch is given maximum learning capacity, it would not be outperformed by models with primary branch having less than the maximum learning capacity.\n\nIn this section, I am not sure what is the fairest way to define learning capacity. Is increasing number of GRUs the best way? What if simpler model architecture can be used for auxiliary targets?\n\n- Hypothesis H3: Auxiliary supervision at the lower hierarchy yields better primary MTL as compared to flat-MTL.\nFor this, where is the experimental result for using them as features instead of using as targets?\n\nOverall, the paper adopts an experiments-driven approach to test the three hypotheses, but the main issue is that this approach adopts a specific neural network method. So how can we make sure that the conclusions always hold then?\n\nSome minor observation: it seems the multi-modal approach in the CVPR 2020 paper \"Multimodal Categorization of Crisis Events in Social Media\" can be used here so that in Figure 3, concat can be used with better attention mechanisms.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7HpzSL30Wfs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses challenges faced in the multi-task learning (MTL) models used in analyzing multimodal conversational data. The main challenge paper is trying to solve is on how to select relevant auxiliary tasks that avoid negative transfer. The authors explore how the preprocessed data used for feature engineering can be re-used as auxiliary tasks in the model. The authors identified sixteen relevant auxiliary tasks, identified a method to distribute learning capacity between primary and auxiliary tasks and proposed a relative supervision hierarchy between primary and auxiliary tasks. An extensive set of experiments are conducted to show the effectiveness of the approach. \n\nThe paper is trying to tackle an important problem faced by multi-task learning (MTL) and the way different aspects of the problem are explained in paper is valuable. The paper takes a systematic approach to address various aspects and conduct extensive experiments to show that having auxiliary task in the model significantly improved performance of the primary task, assigning higher learning capacity (number of parameters) to primary tasks helps in the performance and placing auxiliary supervision at the lower hierarchy significantly improves the performance on primary task. The paper is well written and easy to follow.\n\nAlthough the paper is trying to tackle an important challenge, the novelty or contribution of the paper is limited. The useful ideas concluded by the paper has been previously identified in the community, for example, NLP community has been using auxiliary tasks like NER to improve primary task’s performance. Also, hierarchy in which to structure primary and auxiliary tasks in a model is also somewhat discussed previously. Another shortcoming of the paper is that the proposed solutions are evaluated on a specific domain and it is not clear if these findings are general enough to be applied to other domains. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews for this paper were very borderline. The authors provided detailed responses as well as a few additional results and observations. The authors' responses answered the reviewers' questions and addressed their main comments (including in the discussion of related works as well as with more in-depth analysis in a new Section 5.1). Unfortunately, the reviewers did not come to a consensus.

Overall, this paper extends some current methodology for emotional classification, is well-executed, and provides a reasonably thorough study. The results are somewhat in line with previous results from other fields (and notably NLP), but the authors demonstrate the efficacy of using primary multi-task learning for multimodal conversational analysis. 

Unfortunately, this paper also has some flaws as highlighted by the initial reviews. As stated above, the authors did provide a strong rebuttal, but given the different comments raised by the reviewers that spanned many aspects of the paper including motivation, possibly limited contribution and novelty, missing related work, somewhat shallow analysis of the results, I find that another full round of reviewing would be useful to assess the paper.

As a result, this remains a very borderline paper, and given the strong competition at this year's conference, I cannot recommend acceptance at this stage.

I suggest that the authors incorporate some of the discussions from this forum (and especially with respect to related work, new findings, and clearly defining the motivation and contribution of this work) into the next version of their paper.
Decision: Reject