Unifying Categorical Models by Explicit Disentanglement of the Labels' Generative Factors

In most machine learning tasks , the datasets are mainly annotated by categorical labels . For example , in emotion recognition , most datasets rely only on categorical labels , such as “ happy ” and “ sad ” . Usually , different datasets use different labelling systems ( e.g. , different number of categories and different names ) , even when describing the same data attributes . As a consequence , only a small subset of all the available datasets can be used for any supervised learning task , since the labelling systems used in the training data are not compatible with each other . In this paper , we propose a multi-type continuous disentanglement variational autoencoder to address this problem by identifying and disentangling the true dimensional generative factors that determine each categorical label . By doing so , it is possible to merge multiple datasets based on different categorical models by projecting the data points into a unified latent space . The experiments performed on synthetic datasets show a perfect correlation between the disentangled latent values and the true generative factors . Also , by observing the displacement of each label ’ s explicit distributions , we noticed that the encoded space is a simple affine transformation of the generative factors ’ space . As the latent structure can be autonomously learnt by the model , and each label can be explicitly decomposed into its generative factors , this framework is very promising for further exploring explainability in new and existing neural networks architectures . 1 INTRODUCTION . Supervised machine learning is the most widely used form of machine learning . The possibility to exploit expert knowledge through human annotations allows for simple and direct learning of the data features . As a consequence , the performance and quality of the results obtained by a model are directly linked to the amount of high-quality data available . A clear example of this phenomenon can be seen in machine translation : state-of-the-art models are considered almost indistinguishable from human-performed translation ( Popel et al. , 2020 ) . However , these results only apply in scenarios in which both source and target languages have a large text corpus available ; with low-resource languages , neural methods are far from reaching the same level of performance ( Artetxe et al. , 2018 ) . In particular , it has been observed that neural machine translation methods have a particularly steep learning curve with respect to the amount of data , obtaining very high-quality results with abundant data , but performing more poorly than traditional statistical methods in low-resource settings ( Koehn & Knowles , 2017 ) . Another field , Emotion Recognition in Conversation ( ERC ) , also suffers from the lack of sufficient availability of high-quality data . In Poria et al . ( 2019 ) the main challenges of producing ERC datasets are reported : as emotions are subjective topics , it is hard to define an agreed-upon standard on how to generate broadly useful labelled data . The existing models , used to annotate datasets , can be classified according to the following dichotomy : • Categorical models : Each data point is given one or more labels , chosen from a fixed discrete set of possibilities . Therefore , each label defines a cluster of items that share one ( or more ) common features . As the annotation process is generally performed by humans , the boundaries of each cluster are defined qualitatively and not quantitatively . Thus , dimensional datasets are more versatile , but more difficult to produce . In fact , it is challenging to guarantee a high inter-annotator agreement during the labelling phase as the model complexity increases ( e.g. , it is easier to label a color as “ orange ” , compared to assigning to it the three RGB values ( 255 , 165 , 0 ) ) , especially in subjective tasks . For this reason , categorical models are generally preferred over dimensional ones . However , categorical datasets are difficult to work with , as comparison between models is not immediate , since there is not an explicit relation between multiple labelling systems . Therefore , given a specific task , it is not trivial to merge different data sources to obtain a larger shared training corpus . Although the majority of ERC datasets ( e.g. , IEMOCAP ( Busso et al. , 2008 ) , Emotionlines ( Chen et al. , 2019 ) , DailyDialog ( Li et al. , 2017 ) , and EmoContext ( Chatterjee et al. , 2019 ) ) consist of utterances and categorical emotional labels associated with them , they are based on different taxonomies . EmoContext , for example , subdivides emotions into four categories , and IEMOCAP uses six of them , while Emotionlines and DailyDialog rely on the scheme proposed by Ekman ( 1993 ) ( Table 1 ) . On the other hand , there exist few datasets based on the dimensional model proposed by Russell & Mehrabian ( 1977 ) , according to which each emotion can be described by three independent real values : Valence , Arousal , and Dominance ( VAD ) . An example is the EmoBank dataset ( Buechel & Hahn , 2017 ) . The method proposed in this paper is able to relate categorical models across datasets . As done by Sha & Lukasiewicz ( 2021 ) , when talking about categorical models , we will refer to each label class ( e.g. , object , color , . . . ) as style type , and to each possible value within a label class ( e.g. , dog , cat , red , yellow , . . . ) as style value . We assume that each data point x is generated from the true world simulator Sim given the vector v = ( v1 , v2 , . . . , vd ) ∈ V ⊆ Rd of true continuous generative factors , that is , x = Sim ( v ) . Therefore , each style type s is determined by a subset Vt = { vi1 , vi2 , . . . , vik } of the generative factors . In addition , we assume that the distributions of each style value representation can be approximated by Gaussian distributions . The architecture performs explicit multi-type disentanglement , preserving the original latent structure of the generative factors . That is , given a labelled sample ( x , ts ) , where ts is the style value assigned to the style type s , we are able to correctly approximate p ( v̂ts |x , ts ) , where v̂ts is the style representation disentangled from x , which lies in a space that is an affine transformation of the original V. Thus , the representation v̂st , which is obtained by sampling from the obtained distributions or by using a maximum likelihood approach , can be used to infer the true values that determined the feature of x labelled with st . Being based on a multi-type disentanglement architecture , the method is simultaneously applicable to multiple style types . Therefore , as we are able to compute the true generative factors determining each style value l , it is possible to project each data point in a shared dimensional continuous space representing each style type , independently of the used categorical model , thus merging different datasets . The model is based on an extended version of the architecture proposed by Sha & Lukasiewicz ( 2021 ) in which an explicit sampling layer is inserted between the encoder and the decoder of a standard variational autoencoder model ( Kingma & Welling , 2014 ) . The sampling layer learns and stores information related to the ground-truth distributions of the style values for each style type . The encoder generates a content vector and multiple style vectors ( one for each style type ) that are forced to be close to the correct explicit distributions . Then , for each style type , the sampling layer produces a new style vector that is used by the decoder to reconstruct the original input sample . Compared to the original architecture , instead of relying on a single distribution , we sample a new style vector from each style value ’ s distribution and combine them through a weighted sum , based on a probability score . By doing so , we are able to enforce a smooth transition among similar style values , as each style vector is generated not only by its own style value ’ s ground-truth distribution , but also from neighbour ones . As a consequence , the underlying latent space is continuous . We conducted experiments on two datasets , dSprites ( Matthey et al. , 2017 ) and Base Face Model 2019 ( used to generate a collection of almost one hundred thousands sample images ) ( Gerig et al. , 2018 ) , by labelling the data points with different categorical models and trying to reconstruct , through disentanglement , the original real-value generative factors . The results show perfect correlation between the encoded and original values . Furthermore , as each style value ’ s distribution is explicitly stored within the model , we observe that the latent space produced by the model is an affine transformation of the original one . The contributions of this paper are briefly as follows : • We propose a model that is able to perform an explicit disentanglement , relying exclusively on categorical annotated data . The disentangled style representations lie in a space that is an affine transformation of the true generative factors ’ one . • As a consequence , we propose a framework that is able to discover the true generative factors influencing each style type and determining each style value , making it possible to merge multiple datasets that are annotated with different categorical models . The rest of this paper is organised as follows . Section 2 discusses similar methods in the literature that led to the development of the novel architecture in this paper , highlighting the limitations that it is able to overcome . Section 3 describes in detail the new model . We then present the experiments performed and the obtained results in Section 4 and our conclusive thoughts in Section 5 . 2 RELATED WORK . Disentanglement is a wide-spread technique , used to isolate the true factors generating the input data , as described in the original β-VAE paper ( Higgins et al. , 2017 ) . However , most techniques rely on implicit methods that do not enable for an easy interpretation of the disentangled values . Some components may be pruned ( Stühmer et al. , 2020 ) and quantitative measures of disentanglement are still in their infancy ( Chen et al. , 2018 ) . Latent traversals can provide a qualitative summary of each latent unit ’ s encoded features , but they rely on human expertise and subjectivity and , therefore , may be difficult to apply in real-world scenarios to establish the quality of the obtained disentanglement . In comparison , explicit disentanglement ( i.e. , the model targets specific factors to disentangle , guided by annotated data ) produces easily interpretable components , as it only requires to check that the latent units define the expected features . Since the annotated features are always categorical labels , previous explicit disentanglement works ( John et al. , 2019 ; Romanov et al. , 2019 ; Sha & Lukasiewicz , 2021 ) usually represent the expected feature with a vector , and encouraged it to belong to either of the style values . However , these methods are “ uncontinuous ” , namely , intermediate style values are not encouraged in previous works due to the limitation of categorical labels . In the emotion recognition task in the conversation domain , Wu et al . ( 2019 ) introduced a semisupervised model ( SRV-SLTSM ) in order to obtain explicitly disentangled values for the VAD components of each utterance ’ s emotion . However , the architecture requires a percentage of dimensional annotated data to force each latent unit to encode the desired factor , and it relies on the a-priori assumptions of the VAD model . Furthermore , it can not incorporate the information existing in categorical datasets , as the training process is unsupervised ( i.e. , the model is an autoencoder and relies only on the input samples x and the model ’ s output to compute its loss ) . Park et al . ( 2019 ) proposed a method to extract the continuous values among the VAD dimensions using exclusively categorical annotations . Their solution outperforms SRV-SLTSM in several metrics , showing the importance of the extra data incorporated into the model , as theorized . Being based on the Earth Mover ’ s Distance ( EMD ) loss ( Hou et al. , 2016 ) , the architecture relies even more on the assumptions made on the latent space and the displacement of the categorical values ’ distributions in it . In fact , it requires the labels to be ordered and displaced correctly along each axis , according to their true dimensional value ( e.g. , joy , sad , happy , anger would be sorted as ( anger , sad , joy , happy ) on the valence axis ) . The values used in their experiments are estimated by using the NRC-VAD-Lexicons ( Mohammad , 2018 ) and are fixed throughout the learning process . Our proposed method , instead , is able to obtain an explicit and continuous disentanglement , by relying exclusively on categorical data .

Summary: The authors consider multiple disentangled factors in a variational autoencoder. The proposed method extends the structure of Sha&Lukasiewcz(2021) by modifying the sampling layer. In the new sampling layer, multiple factor representation is generated as well as a probabilistic weight for each factor. Then, the final representation is a weighted sum of all the factors.  The proposed method is evaluated on toy data, i.e., modified dSprites and Base Face Model 2019. The results show that the method is able to encode multiple factors.