LABELS’ GENERATIVE FACTORS
Year: 2021

Abstract:
In most machine learning tasks, the datasets are mainly annotated by categorical labels. For example, in emotion recognition, most datasets rely only on categorical labels, such as “happy” and “sad”. Usually, different datasets use different labelling systems (e.g., different number of categories and different names), even when describing the same data attributes. As a consequence, only a small subset of all the available datasets can be used for any supervised learning task, since the labelling systems used in the training data are not compatible with each other. In this paper, we propose a multi-type continuous disentanglement variational autoencoder to address this problem by identifying and disentangling the true dimensional generative factors that determine each categorical label. By doing so, it is possible to merge multiple datasets based on different categorical models by projecting the data points into a unified latent space. The experiments performed on synthetic datasets show a perfect correlation between the disentangled latent values and the true generative factors. Also, by observing the displacement of each label’s explicit distributions, we noticed that the encoded space is a simple affine transformation of the generative factors’ space. As the latent structure can be autonomously learnt by the model, and each label can be explicitly decomposed into its generative factors, this framework is very promising for further exploring explainability in new and existing neural networks architectures.

text: In most machine learning tasks, the datasets are mainly annotated by categorical labels. For example, in emotion recognition, most datasets rely only on categorical labels, such as “happy” and “sad”. Usually, different datasets use different labelling systems (e.g., different number of categories and different names), even when describing the same data attributes. As a consequence, only a small subset of all the available datasets can be used for any supervised learning task, since the labelling systems used in the training data are not compatible with each other. In this paper, we propose a multi-type continuous disentanglement variational autoencoder to address this problem by identifying and disentangling the true dimensional generative factors that determine each categorical label. By doing so, it is possible to merge multiple datasets based on different categorical models by projecting the data points into a unified latent space. The experiments performed on synthetic datasets show a perfect correlation between the disentangled latent values and the true generative factors. Also, by observing the displacement of each label’s explicit distributions, we noticed that the encoded space is a simple affine transformation of the generative factors’ space. As the latent structure can be autonomously learnt by the model, and each label can be explicitly decomposed into its generative factors, this framework is very promising for further exploring explainability in new and existing neural networks architectures.

heading: 1 INTRODUCTION
text: Supervised machine learning is the most widely used form of machine learning. The possibility to exploit expert knowledge through human annotations allows for simple and direct learning of the data features. As a consequence, the performance and quality of the results obtained by a model are directly linked to the amount of high-quality data available. A clear example of this phenomenon can be seen in machine translation: state-of-the-art models are considered almost indistinguishable from human-performed translation (Popel et al., 2020). However, these results only apply in scenarios in which both source and target languages have a large text corpus available; with low-resource languages, neural methods are far from reaching the same level of performance (Artetxe et al., 2018). In particular, it has been observed that neural machine translation methods have a particularly steep learning curve with respect to the amount of data, obtaining very high-quality results with abundant data, but performing more poorly than traditional statistical methods in low-resource settings (Koehn & Knowles, 2017). Another field, Emotion Recognition in Conversation (ERC), also suffers from the lack of sufficient availability of high-quality data. In Poria et al. (2019) the main challenges of producing ERC datasets are reported: as emotions are subjective topics, it is hard to define an agreed-upon standard on how to generate broadly useful labelled data. The existing models, used to annotate datasets, can be classified according to the following dichotomy:
• Categorical models: Each data point is given one or more labels, chosen from a fixed discrete set of possibilities. Therefore, each label defines a cluster of items that share one (or more) common features. As the annotation process is generally performed by humans, the boundaries of each cluster are defined qualitatively and not quantitatively.
Thus, dimensional datasets are more versatile, but more difficult to produce. In fact, it is challenging to guarantee a high inter-annotator agreement during the labelling phase as the model complexity increases (e.g., it is easier to label a color as “orange”, compared to assigning to it the three RGB values (255, 165, 0)), especially in subjective tasks. For this reason, categorical models are generally preferred over dimensional ones. However, categorical datasets are difficult to work with, as comparison between models is not immediate, since there is not an explicit relation between multiple labelling systems. Therefore, given a specific task, it is not trivial to merge different data sources to obtain a larger shared training corpus. Although the majority of ERC datasets (e.g., IEMOCAP (Busso et al., 2008), Emotionlines (Chen et al., 2019), DailyDialog (Li et al., 2017), and EmoContext (Chatterjee et al., 2019)) consist of utterances and categorical emotional labels associated with them, they are based on different taxonomies. EmoContext, for example, subdivides emotions into four categories, and IEMOCAP uses six of them, while Emotionlines and DailyDialog rely on the scheme proposed by Ekman (1993) (Table 1). On the other hand, there exist few datasets based on the dimensional model proposed by Russell & Mehrabian (1977), according to which each emotion can be described by three independent real values: Valence, Arousal, and Dominance (VAD). An example is the EmoBank dataset (Buechel & Hahn, 2017).
The method proposed in this paper is able to relate categorical models across datasets. As done by Sha & Lukasiewicz (2021), when talking about categorical models, we will refer to each label class (e.g., object, color, . . .) as style type, and to each possible value within a label class (e.g., dog, cat, red, yellow, . . .) as style value. We assume that each data point x is generated from the true world simulator Sim given the vector v = (v1, v2, . . . , vd) ∈ V ⊆ Rd of true continuous generative factors, that is, x = Sim(v). Therefore, each style type s is determined by a subset Vt = {vi1 , vi2 , . . . , vik} of the generative factors. In addition, we assume that the distributions of each style value representation can be approximated by Gaussian distributions. The architecture performs explicit multi-type disentanglement, preserving the original latent structure of the generative factors. That is, given a labelled sample (x, ts), where ts is the style value assigned to the style type s, we are able to correctly approximate p(v̂ts |x, ts), where v̂ts is the style representation disentangled from x, which lies in a space that is an affine transformation of the original V. Thus, the representation v̂st , which is obtained by sampling from the obtained distributions or by using a maximum likelihood approach, can be used to infer the true values that determined the feature of x labelled with st. Being based on a multi-type disentanglement architecture, the method is simultaneously applicable to multiple style types. Therefore, as we are able to compute the true generative factors determining each style value l, it is possible to project each data point in a shared dimensional continuous space representing each style type, independently of the used categorical model, thus merging different datasets.
The model is based on an extended version of the architecture proposed by Sha & Lukasiewicz (2021) in which an explicit sampling layer is inserted between the encoder and the decoder of a
standard variational autoencoder model (Kingma & Welling, 2014). The sampling layer learns and stores information related to the ground-truth distributions of the style values for each style type. The encoder generates a content vector and multiple style vectors (one for each style type) that are forced to be close to the correct explicit distributions. Then, for each style type, the sampling layer produces a new style vector that is used by the decoder to reconstruct the original input sample. Compared to the original architecture, instead of relying on a single distribution, we sample a new style vector from each style value’s distribution and combine them through a weighted sum, based on a probability score. By doing so, we are able to enforce a smooth transition among similar style values, as each style vector is generated not only by its own style value’s ground-truth distribution, but also from neighbour ones. As a consequence, the underlying latent space is continuous.
We conducted experiments on two datasets, dSprites (Matthey et al., 2017) and Base Face Model 2019 (used to generate a collection of almost one hundred thousands sample images) (Gerig et al., 2018), by labelling the data points with different categorical models and trying to reconstruct, through disentanglement, the original real-value generative factors. The results show perfect correlation between the encoded and original values. Furthermore, as each style value’s distribution is explicitly stored within the model, we observe that the latent space produced by the model is an affine transformation of the original one. The contributions of this paper are briefly as follows:
• We propose a model that is able to perform an explicit disentanglement, relying exclusively on categorical annotated data. The disentangled style representations lie in a space that is an affine transformation of the true generative factors’ one.
• As a consequence, we propose a framework that is able to discover the true generative factors influencing each style type and determining each style value, making it possible to merge multiple datasets that are annotated with different categorical models.
The rest of this paper is organised as follows. Section 2 discusses similar methods in the literature that led to the development of the novel architecture in this paper, highlighting the limitations that it is able to overcome. Section 3 describes in detail the new model. We then present the experiments performed and the obtained results in Section 4 and our conclusive thoughts in Section 5.

heading: 2 RELATED WORK
text: Disentanglement is a wide-spread technique, used to isolate the true factors generating the input data, as described in the original β-VAE paper (Higgins et al., 2017). However, most techniques rely on implicit methods that do not enable for an easy interpretation of the disentangled values. Some components may be pruned (Stühmer et al., 2020) and quantitative measures of disentanglement are still in their infancy (Chen et al., 2018). Latent traversals can provide a qualitative summary of each latent unit’s encoded features, but they rely on human expertise and subjectivity and, therefore, may be difficult to apply in real-world scenarios to establish the quality of the obtained disentanglement. In comparison, explicit disentanglement (i.e., the model targets specific factors to disentangle, guided by annotated data) produces easily interpretable components, as it only requires to check that the latent units define the expected features. Since the annotated features are always categorical labels, previous explicit disentanglement works (John et al., 2019; Romanov et al., 2019; Sha & Lukasiewicz, 2021) usually represent the expected feature with a vector, and encouraged it to belong to either of the style values. However, these methods are “uncontinuous”, namely, intermediate style values are not encouraged in previous works due to the limitation of categorical labels.
In the emotion recognition task in the conversation domain, Wu et al. (2019) introduced a semisupervised model (SRV-SLTSM) in order to obtain explicitly disentangled values for the VAD components of each utterance’s emotion. However, the architecture requires a percentage of dimensional annotated data to force each latent unit to encode the desired factor, and it relies on the a-priori assumptions of the VAD model. Furthermore, it cannot incorporate the information existing in categorical datasets, as the training process is unsupervised (i.e., the model is an autoencoder and relies only on the input samples x and the model’s output to compute its loss). Park et al. (2019) proposed a method to extract the continuous values among the VAD dimensions using exclusively categorical annotations. Their solution outperforms SRV-SLTSM in several metrics, showing the importance of the extra data incorporated into the model, as theorized. Being based on the Earth Mover’s Distance
(EMD) loss (Hou et al., 2016), the architecture relies even more on the assumptions made on the latent space and the displacement of the categorical values’ distributions in it. In fact, it requires the labels to be ordered and displaced correctly along each axis, according to their true dimensional value (e.g., joy, sad, happy, anger would be sorted as (anger, sad, joy, happy) on the valence axis). The values used in their experiments are estimated by using the NRC-VAD-Lexicons (Mohammad, 2018) and are fixed throughout the learning process. Our proposed method, instead, is able to obtain an explicit and continuous disentanglement, by relying exclusively on categorical data.

heading: 3 APPROACH
text: We will now analyze a single-style type scenario (e.g., where the input data is annotated with a single label); however, the described methods can be applied independently to every style vector. As mentioned above, our approach consists of a modification of the original architecture proposed by Sha & Lukasiewicz (2021), which predefined a Gaussian distribution for each style type value, and, given a sample (x, t), required every disentangled d-dimensional style representation s generated by the encoder to belong to the style distribution that represents the annotated style type value t. This is achieved by maximizing the probability
p(t|s) = pNor(s|t)p(t) p(s) , (1)
where p(t) is the prior distribution of the style values, and pNor is as follows:
pNor(s|t) = exp(− 12 (s− µt) TΣ−1t (s− µt))√ (2π)d det(Σt) . (2)
Then, a new style vector s′ is sampled from its corresponding correct distribution, according to the input label, and forwarded to the decoder. To make sure that each style distribution actually encodes the correct style representation, they sample from their corresponding distributions a vector s′j for each j ∈ 1, . . . ,m, being T = {t1, t2, . . . , tm} the set of possible style values. Then, each s′j , and thus its distribution, is forced to be similar to the correct style value representation through the following classification loss (to be computed for each style type):
LCL = − 1
|T | |T |∑ j=1 log(pc(tj |s′j)), (3)
where pc is defined as:
pc(tj |s′j) = pNor(s
′ j |tj)p(tj)∑
t′∈T pNor(s ′ j |t′)p(t′)
= pNor(s ′ j |tj)∑
t′∈T pNor(s ′ j |t′)
. (4)
since we assume that all labels share the same prior probability. The vector s′, however, cannot contain any of the unique nuances of the input data point that could be represented in s: as it is obtained from a single style value’s distribution, it can only encode the average features shared by all the samples with the same style value.

heading: 3.1 ARCHITECTURE
text: The general structure of our architecture is unchanged compared to the one employed by Sha & Lukasiewicz (2021): it consists of an encoder, a decoder, and a bottleneck layer, in which a resampling operation happens for each of the generated style vectors. In order to obtain a continuous disentangled representation, we modified the sampling layer. Consider a sample (x, t) and its generated embedding h, split into content vector c and a single style vector s, whose possible m style values are the elements of T . As described above, a probability distribution Ni is associated with each of them. Sampling a new s′ from a single distribution discards any extra information (i.e., detailed features other than the discrete style category) stored in s. Our method preserves it by sampling a different style vector ŝi from each distributionNi and summing them to s′, weighted by each probability distribution p(ti|s). Formally, the new resampled vector ŝ′ is obtained as follows:
ŝ′ = s′ + j∑ i=1 ŝi p(ti|s). (5)
The vector ŝ′ is then fed to the decoder. By doing so, we obtain a continuous latent space, as each distribution contributes to the final representation, and therefore similar style value encodings are forced to be close also in the latent space. In addition, we can encode intermediate style features by interpolating the representations obtained from different distributions,since ŝ′ is a weighted sum of them. In order to disentangle the true independent generative factors instead of an alternative representation of the style values, we used one-dimensional style vectors. This also defines a clear notion of order, induced by the real line. As it is possible for a style value to be multidimensional (i.e., being determined by the combination of multiple generative factors), we allow for multiple style vectors to represent the same style value, each learning to encode a different generative factor. As a consequence, we have to modify the classification loss proposed by Sha & Lukasiewicz (2021) as follows (remembering that we are considering a single style type):
L̂CL = − 1
|T | T∑ j=1 log(p̂c(tj |s′j1, . . . , s′jk)), (6)
where k is the numbered of style vectors reserved to encode a style value, and p̂c is as follows: p̂c(tj |s′j1, . . . , s′jk) = ∏k i=1 pNor(s ′ ji|tj)∑
t′∈T ( ∏k i=1 pNor(s ′ ji|t′)) . (7)
Furthermore, assuming a d-dimensional style type, with d < k, our method is able to find d, reporting k − d style vectors as unused. By measuring the KL divergence between a style type’s distributions, it is possible to quantify the amount of information stored in each latent unit. When the KL divergence is close to 0, there is no difference between the style representations among that dimension, and the resampled value encodes only random noise and remains unused. This allows us to better understand the structure of the generative factors’ hidden space. The complete diagram of the architecture is given in Figure 1.

heading: 3.2 DATASETS
text: As we focused on defining a general-purpose framework for disentanglement and explainability, we tested our method on synthetic datasets used in disentanglement-related tasks. In order to obtain experimental results, we created a modified version of the dSprites (Matthey et al., 2017) and Base Face Model 20191 (Gerig et al., 2018) datasets. In both cases, we replaced some of the continuous
1To construct the dataset, a collection of 96100 data points was generated, using 100 different face IDs and 31 evenly distributed values for both the face orientation (“yaw-range”) and illumination angle (“illu-range”).
values used to generate each data point with specific categorical labels, based on predefined ranges among each dimension. The model is then given access only on the sample image and its categorical labels, while the true generative values are used as ground truth to be compared with the obtained results in order to evaluate the model’s performance. dSprites is a dataset composed by 737280 square images of 64× 64 pixels. Each image is determined by five factors that can assume different values in a specific range: shape, scale, rotation, x, and y. We defined two categorical models. Each uses two one-dimensional style types (i.e., that rely on a single generative factor) in order to label the x and y values separately. A third one, instead, defines a single two-dimensional style type to describe the combination of the x and y coordinates. The ranges and labels used for each categorical model are as follows:
• dSprites AP (Average Precision): The x value (in pixels) is labelled according to the ranges RAP = {[0, 23], [24, 39], [40, 63]} corresponding to the three style value names left-ap, center-ap, and right-ap. Analogously, y is labelled according to RAP , using the style value names top-ap, middle-ap, and bottom-ap.
• dSprites HP (High Precision): The x value is labelled according to the ranges RHP = {[0, 10], [11, 25], [26, 37], [38, 52], [53, 63]} and the style value names far-left-hp, left-hp, center-hp, right-hp, and far-right-hp. Analogously, y is labelled according to RHP , using the style value names far-top-hp, top-hp, middle-hp, bottom-hp, and far-bottom-hp.
• dSprites Co (Compound): The x and y values are labelled as a single feature according to the ranges RCo = RAP ×RAP . This defines nine style value names, defining a 3× 3 grid over the input images. The used names are ij-co, each identifying the grid cell at row i and column j.
A single shape type was used to generate the dataset associated with each categorical model (e.g., dSprites AP is built using the elliptic shapes). A visualization of the dSprites AP and dSprites HP categorical models is given in Figure 2. The dataset generated using the Base Face Model 2019 was processed in a similar way, defining five evenly sized ranges for the face orientation (i.e., the “yaw-range” value, ranging from −90 to +90 degrees) and three for the illumination angle (i.e., the “illu-range” value, ranging as well from −90 to +90 degrees). A total of 96100 images were generated and labelled, defining the BFM 2019 dataset.

heading: 4 EXPERIMENTS AND RESULTS
text: To our knowledge, there are no previous works that analyze the relation between explicit disentanglement and categorical annotations. Therefore, we describe the metrics used to evaluate the model, and we briefly introduce the results obtained by using the original architecture by Sha & Lukasiewicz (2021), in order to establish a baseline. As we are interested in the structure of the latent space in which the disentangled style representations are embedded, we focused on the displacement of the distributions and the performance in encoding and reconstructing intermediate style values. As our method relies on one-dimensional style vectors (i.e., scalar variables), we used the same one-dimensional style encoding also for the architecture by Sha & Lukasiewicz (2021) to
get a fair comparison. It is true that, by using multidimensional style vectors, better results could be achieved. The encoding latent space, however, would be completely unrelated to the original generative factors, and a comparison between different categorical models unfeasible.
We trained the architecture by Sha & Lukasiewicz (2021) on the dSprites AP and dSprites HP datasets and analyzed the style representations in the latent space. By measuring the correlation between each style value distribution’s mean and each label’s range average value (e.g., far-left-hp represents x values in the range [0, 10], so its average value is 5), we observed that the distribution displacements do not reflect the order induced by their true generative factors, resulting in a low correlation. For example, considering dSprites HP and the x coordinate, we require the ordered sequence (far-left-hp, left-hp, center-hp, right-hp, far-right-hp)2 in order to obtain a continuous encoding space. Our method, instead, achieves an almost perfect correlation for every style value on both datasets (Figure 3), forcing the distributions to reflect the order of the ground-truth labelling sequences. This behaviour is consistent across several runs of the algorithm. In contrast, the results using the architecture by Sha & Lukasiewicz (2021) suffer from a high variance, as any random initialization of the model’s distribution parameters leads to a different displacement, and thus correlation. Appendix A shows several examples of the distributions’ displacement for both datasets. This is possible, as the parameters defining each distribution are explicitly stored inside the model.
Having a continuous encoding space is necessary to represent intermediate style values. To show this, we performed latent traversals on the embedded vectors produced by the architectures trained on the dSprites HP dataset. The model by Sha & Lukasiewicz (2021) encodes each style value as an independent group of data points. There is no smooth transition for intermediate representations and the discontinuities in the latent space can be clearly observed (Figure 4a). In comparison, our method shows an ordered sequence of data points on both the x and y traversals (Figure 4b).
2As we are dealing with linear transformations, also the reversed sequence would be acceptable
In addition, we compared the relative size of the ranges used to define each style value with the ranges defined by the model, obtained by considering the maximum likelihood style distribution for each possible value of the latent space. Table 2 reports the results obtained by our method on the x coordinate in the dSprites HP dataset. There is a high correlation between the two spaces, showing that the encoded latent space is an approximation of an affine transformation of the original one. As a consequence, it is possible to train a single model on both the dSprites AP and dSprites HP datasets simultaneously, by using two sampling layers (one for each dataset) that share the same encoder and decoder. The result is that the distributions in both layers belong to the same space. The representations of similar style values in both categorical models are, therefore, similar even in the latent space (Figure 5a). This would not happen if each distribution was learnt independently, since it would be displaced randomly and not uniformly across datasets. If we compare the normalized latent values for the x generative factor, we see that our method produces similar encoding for both dSprites datasets, highly correlated with the ground-truth x value. This does not happen using the architecture by Sha & Lukasiewicz (2021), as each style value representation is independent and does not follow a precise structure (Figure 5b).
We used the dSprites Co dataset to study the disentanglement of multi-dimensional style types. Since the labels used in the dataset are determined by the combination of the two independent factors x and y, we defined the style type “position” with nine possible style values (i.e., the labels ij-co, i ∈ {1, 2, 3}, and j ∈ {1, 2, 3}). We set k = 2 and expected the model to disentangle the two generative factors on the two available dimensions. By normalizing each dimension and comparing the displacement of the distribution with the grid induced byRCo, we observe a high correlation with the ground-truth values (Figure 6a), confirming the capability of the model to disentangle them. In addition, we analyzed the scenario in which k > d to show how our method uses only the strictly necessary number of dimensions to represent a style value, corresponding to the underlying number of generative factors. By setting k = 2 and training on dSprites HP, we observe that the encoding for the style type factor x (and analogously for y) only uses one of the two available dimensions, as expected. In fact, the distributions in the second dimension associated with the style type completely overlap with each other, having a pairwise KL divergence approximately equal to 0 (Figure 6b). As the model has to sample a new style vector from its corresponding style distribution, and given that all the probability densities are equal, we conclude that the latent unit can only encode random noise, not being able to distinguish different style values.

heading: 5 CONCLUSION
text: We have shown that current methods performing explicit disentanglement are not suitable to identify the true generative factors corresponding to each style type. Instead, an independent representation in a different latent space is learnt. Since there is no enforced structure on the new space, the style representations depend exclusively on the random initialization of the model parameters. As a consequence, it is not possible to project different categorically annotated datasets into the same dimensional space. Furthermore, as the focus of the architecture by Sha & Lukasiewicz (2021) is on each single style value, seen as a category of objects that share the same feature, the model fails to represent the unique characteristics of each input sample, and therefore to provide a smooth transition between similar style values. To fix these disadvantages, we obtain a unified style representation across different categorical models, and making it consistent with the generating space by forcing an ordering among the distribution on each encoding dimension. In addition, since similar style values are now also close in the latent space, our model is able to encode individual nuances of each data point. By doing so, it is possible to label each data point with its true dimensional attributes, allowing for multiple categorical datasets to be merged together.

heading: A APPENDIX
text: We report the displacement of the distribution in the latent space for both architectures and the different datasets (Figure 7 and 8).

Review summaries:
This paper proposes a method for projecting datasets with different categorical labels into a unified latent space. It is based on an architecture proposed by Sha and Lukasiewicz (2021) which is a VAE with a structured latent space. The latent space is split into a content and style part, where the style part models the classification of labels, and the content the rest of the features to reconstruct the input. The goal of the proposed modification is to have similar label values close together in latent space while disentangling across labels. In experiments on synthetic datasets, the model is shown to provide the specified properties.

This work proposes a continuous disentanglement variational autoencoder. The approach is an extension of the disentangled variational autoencoder proposed by Sha & Lukasiewicz in which they modify sampling layer to use weighted sum of multiple distributions instead of single style vector distribution. This ensure continuity in the latent space.

The authors consider multiple disentangled factors in a variational autoencoder. The proposed method extends the structure of Sha&Lukasiewcz(2021) by modifying the sampling layer. In the new sampling layer, multiple factor representation is generated as well as a probabilistic weight for each factor. Then, the final representation is a weighted sum of all the factors.  The proposed method is evaluated on toy data, i.e., modified dSprites and Base Face Model 2019. The results show that the method is able to encode multiple factors.

The paper is a follow-up to Sha and Lukasiewicz 2021 "Multi-type Disentanglement without Adversarial Training" (SL21 from now on) which uses a slight modification of the technique from SL21 to the new task of training a single model on multiple labeled datasets with different label sets that encode the same information at different levels of granularity (so in a way it's a variant of multi-task learning). The model effectively learns a single continuous embedding for examples from which all label sets are easily predictable, effectively unifying the different label sets. The experiments are performed on modified versions of the dsprites and base face datasets. On dsprits it shows a more smooth representation than SL21.