Unifying Categorical Models by Explicit Disentanglement of the Labels' Generative Factors

In most machine learning tasks , the datasets are mainly annotated by categorical labels . For example , in emotion recognition , most datasets rely only on categorical labels , such as “ happy ” and “ sad ” . Usually , different datasets use different labelling systems ( e.g. , different number of categories and different names ) , even when describing the same data attributes . As a consequence , only a small subset of all the available datasets can be used for any supervised learning task , since the labelling systems used in the training data are not compatible with each other . In this paper , we propose a multi-type continuous disentanglement variational autoencoder to address this problem by identifying and disentangling the true dimensional generative factors that determine each categorical label . By doing so , it is possible to merge multiple datasets based on different categorical models by projecting the data points into a unified latent space . The experiments performed on synthetic datasets show a perfect correlation between the disentangled latent values and the true generative factors . Also , by observing the displacement of each label ’ s explicit distributions , we noticed that the encoded space is a simple affine transformation of the generative factors ’ space . As the latent structure can be autonomously learnt by the model , and each label can be explicitly decomposed into its generative factors , this framework is very promising for further exploring explainability in new and existing neural networks architectures . 1 INTRODUCTION . Supervised machine learning is the most widely used form of machine learning . The possibility to exploit expert knowledge through human annotations allows for simple and direct learning of the data features . As a consequence , the performance and quality of the results obtained by a model are directly linked to the amount of high-quality data available . A clear example of this phenomenon can be seen in machine translation : state-of-the-art models are considered almost indistinguishable from human-performed translation ( Popel et al. , 2020 ) . However , these results only apply in scenarios in which both source and target languages have a large text corpus available ; with low-resource languages , neural methods are far from reaching the same level of performance ( Artetxe et al. , 2018 ) . In particular , it has been observed that neural machine translation methods have a particularly steep learning curve with respect to the amount of data , obtaining very high-quality results with abundant data , but performing more poorly than traditional statistical methods in low-resource settings ( Koehn & Knowles , 2017 ) . Another field , Emotion Recognition in Conversation ( ERC ) , also suffers from the lack of sufficient availability of high-quality data . In Poria et al . ( 2019 ) the main challenges of producing ERC datasets are reported : as emotions are subjective topics , it is hard to define an agreed-upon standard on how to generate broadly useful labelled data . The existing models , used to annotate datasets , can be classified according to the following dichotomy : • Categorical models : Each data point is given one or more labels , chosen from a fixed discrete set of possibilities . Therefore , each label defines a cluster of items that share one ( or more ) common features . As the annotation process is generally performed by humans , the boundaries of each cluster are defined qualitatively and not quantitatively . Thus , dimensional datasets are more versatile , but more difficult to produce . In fact , it is challenging to guarantee a high inter-annotator agreement during the labelling phase as the model complexity increases ( e.g. , it is easier to label a color as “ orange ” , compared to assigning to it the three RGB values ( 255 , 165 , 0 ) ) , especially in subjective tasks . For this reason , categorical models are generally preferred over dimensional ones . However , categorical datasets are difficult to work with , as comparison between models is not immediate , since there is not an explicit relation between multiple labelling systems . Therefore , given a specific task , it is not trivial to merge different data sources to obtain a larger shared training corpus . Although the majority of ERC datasets ( e.g. , IEMOCAP ( Busso et al. , 2008 ) , Emotionlines ( Chen et al. , 2019 ) , DailyDialog ( Li et al. , 2017 ) , and EmoContext ( Chatterjee et al. , 2019 ) ) consist of utterances and categorical emotional labels associated with them , they are based on different taxonomies . EmoContext , for example , subdivides emotions into four categories , and IEMOCAP uses six of them , while Emotionlines and DailyDialog rely on the scheme proposed by Ekman ( 1993 ) ( Table 1 ) . On the other hand , there exist few datasets based on the dimensional model proposed by Russell & Mehrabian ( 1977 ) , according to which each emotion can be described by three independent real values : Valence , Arousal , and Dominance ( VAD ) . An example is the EmoBank dataset ( Buechel & Hahn , 2017 ) . The method proposed in this paper is able to relate categorical models across datasets . As done by Sha & Lukasiewicz ( 2021 ) , when talking about categorical models , we will refer to each label class ( e.g. , object , color , . . . ) as style type , and to each possible value within a label class ( e.g. , dog , cat , red , yellow , . . . ) as style value . We assume that each data point x is generated from the true world simulator Sim given the vector v = ( v1 , v2 , . . . , vd ) ∈ V ⊆ Rd of true continuous generative factors , that is , x = Sim ( v ) . Therefore , each style type s is determined by a subset Vt = { vi1 , vi2 , . . . , vik } of the generative factors . In addition , we assume that the distributions of each style value representation can be approximated by Gaussian distributions . The architecture performs explicit multi-type disentanglement , preserving the original latent structure of the generative factors . That is , given a labelled sample ( x , ts ) , where ts is the style value assigned to the style type s , we are able to correctly approximate p ( v̂ts |x , ts ) , where v̂ts is the style representation disentangled from x , which lies in a space that is an affine transformation of the original V. Thus , the representation v̂st , which is obtained by sampling from the obtained distributions or by using a maximum likelihood approach , can be used to infer the true values that determined the feature of x labelled with st . Being based on a multi-type disentanglement architecture , the method is simultaneously applicable to multiple style types . Therefore , as we are able to compute the true generative factors determining each style value l , it is possible to project each data point in a shared dimensional continuous space representing each style type , independently of the used categorical model , thus merging different datasets . The model is based on an extended version of the architecture proposed by Sha & Lukasiewicz ( 2021 ) in which an explicit sampling layer is inserted between the encoder and the decoder of a standard variational autoencoder model ( Kingma & Welling , 2014 ) . The sampling layer learns and stores information related to the ground-truth distributions of the style values for each style type . The encoder generates a content vector and multiple style vectors ( one for each style type ) that are forced to be close to the correct explicit distributions . Then , for each style type , the sampling layer produces a new style vector that is used by the decoder to reconstruct the original input sample . Compared to the original architecture , instead of relying on a single distribution , we sample a new style vector from each style value ’ s distribution and combine them through a weighted sum , based on a probability score . By doing so , we are able to enforce a smooth transition among similar style values , as each style vector is generated not only by its own style value ’ s ground-truth distribution , but also from neighbour ones . As a consequence , the underlying latent space is continuous . We conducted experiments on two datasets , dSprites ( Matthey et al. , 2017 ) and Base Face Model 2019 ( used to generate a collection of almost one hundred thousands sample images ) ( Gerig et al. , 2018 ) , by labelling the data points with different categorical models and trying to reconstruct , through disentanglement , the original real-value generative factors . The results show perfect correlation between the encoded and original values . Furthermore , as each style value ’ s distribution is explicitly stored within the model , we observe that the latent space produced by the model is an affine transformation of the original one . The contributions of this paper are briefly as follows : • We propose a model that is able to perform an explicit disentanglement , relying exclusively on categorical annotated data . The disentangled style representations lie in a space that is an affine transformation of the true generative factors ’ one . • As a consequence , we propose a framework that is able to discover the true generative factors influencing each style type and determining each style value , making it possible to merge multiple datasets that are annotated with different categorical models . The rest of this paper is organised as follows . Section 2 discusses similar methods in the literature that led to the development of the novel architecture in this paper , highlighting the limitations that it is able to overcome . Section 3 describes in detail the new model . We then present the experiments performed and the obtained results in Section 4 and our conclusive thoughts in Section 5 . 2 RELATED WORK . Disentanglement is a wide-spread technique , used to isolate the true factors generating the input data , as described in the original β-VAE paper ( Higgins et al. , 2017 ) . However , most techniques rely on implicit methods that do not enable for an easy interpretation of the disentangled values . Some components may be pruned ( Stühmer et al. , 2020 ) and quantitative measures of disentanglement are still in their infancy ( Chen et al. , 2018 ) . Latent traversals can provide a qualitative summary of each latent unit ’ s encoded features , but they rely on human expertise and subjectivity and , therefore , may be difficult to apply in real-world scenarios to establish the quality of the obtained disentanglement . In comparison , explicit disentanglement ( i.e. , the model targets specific factors to disentangle , guided by annotated data ) produces easily interpretable components , as it only requires to check that the latent units define the expected features . Since the annotated features are always categorical labels , previous explicit disentanglement works ( John et al. , 2019 ; Romanov et al. , 2019 ; Sha & Lukasiewicz , 2021 ) usually represent the expected feature with a vector , and encouraged it to belong to either of the style values . However , these methods are “ uncontinuous ” , namely , intermediate style values are not encouraged in previous works due to the limitation of categorical labels . In the emotion recognition task in the conversation domain , Wu et al . ( 2019 ) introduced a semisupervised model ( SRV-SLTSM ) in order to obtain explicitly disentangled values for the VAD components of each utterance ’ s emotion . However , the architecture requires a percentage of dimensional annotated data to force each latent unit to encode the desired factor , and it relies on the a-priori assumptions of the VAD model . Furthermore , it can not incorporate the information existing in categorical datasets , as the training process is unsupervised ( i.e. , the model is an autoencoder and relies only on the input samples x and the model ’ s output to compute its loss ) . Park et al . ( 2019 ) proposed a method to extract the continuous values among the VAD dimensions using exclusively categorical annotations . Their solution outperforms SRV-SLTSM in several metrics , showing the importance of the extra data incorporated into the model , as theorized . Being based on the Earth Mover ’ s Distance ( EMD ) loss ( Hou et al. , 2016 ) , the architecture relies even more on the assumptions made on the latent space and the displacement of the categorical values ’ distributions in it . In fact , it requires the labels to be ordered and displaced correctly along each axis , according to their true dimensional value ( e.g. , joy , sad , happy , anger would be sorted as ( anger , sad , joy , happy ) on the valence axis ) . The values used in their experiments are estimated by using the NRC-VAD-Lexicons ( Mohammad , 2018 ) and are fixed throughout the learning process . Our proposed method , instead , is able to obtain an explicit and continuous disentanglement , by relying exclusively on categorical data .

Summary: This work proposes a continuous disentanglement variational autoencoder. The approach is an extension of the disentangled variational autoencoder proposed by Sha & Lukasiewicz in which they modify sampling layer to use weighted sum of multiple distributions instead of single style vector distribution. This ensure continuity in the latent space.