Unifying Categorical Models by Explicit Disentanglement of the Labels' Generative Factors
Authors: Luca Pinchetti
Lei Sha
Thomas Lukasiewicz
Venue: ICLR 2022 Submitted (2022)
Keywords: Disentanglement
explainability
latent representation

Abstract: In most machine learning tasks, the datasets are mainly annotated by categorical labels. For example, in emotion recognition, most datasets rely only on categorical labels, such as ``happy'' and ``sad''. Usually, different datasets use different labelling systems (e.g., different number of categories and different names), even when describing the same data attributes. As a consequence, only a small subset of all the available datasets can be used for any supervised learning task, since the labelling systems used in the training data are not compatible with each other.
In this paper, we propose a \emph{multi-type continuous disentanglement variational autoencoder} to address this problem by identifying and disentangling the true dimensional generative factors that determine each categorical label. By doing so, it is possible to merge multiple datasets based on different categorical models by projecting the data points into a unified latent space.
The experiments performed on synthetic datasets show a perfect correlation between the disentangled latent values and the true generative factors. Also, by observing the displacement of each label's explicit distributions, we noticed that the encoded space is a simple affine transformation of the generative factors' space. As the latent structure can be autonomously learnt by the model, and each label can be explicitly decomposed in its generative factors, this framework is very promising for further exploring explainability in new and existing neural networks architectures.

Under review as a conference paper at ICLR 2022
UNIFYING CATEGORICAL MODELS BY EXPLICIT DISENTANGLEMENT OF THE LABELS' GENERATIVE FACTORS
Anonymous authors Paper under double-bling review
ABSTRACT
In most machine learning tasks, the datasets mainly annotated by categorical labels. For example, in emotion recognition, most datasets rely only on categor- ical labels, such "happy" and "sad". Usually, different datasets use different labelling systems (e.g.. different number of categories and different names), even when describing the same data attributes. As a consequence, only a small subset of all the available datasets can be used for any supervised learning task, since the labelling systems used in the training data are not compatible with each other. In this paper, we propose multi-type continuous disentanglement variational autoencoder to address this problem by identifying and disentangling the true di- mensional generative factors that determine each categorical label. By doing so, it is possible to merge multiple datasets based on different categorical models by projecting the data points into unified latent space. The experiments performed on synthetic datasets show a perfect correlation between the disentangled latent values and the true generative factors. Also, by observing the displacement of each label's explicit distributions, we noticed that the encoded space is simple affine transformation of the generative factors' space. As the latent structure can be autonomously learnt by the model, and each label can be explicitly decomposed into generative factors, this framework is very promising for further exploring explainability in new and existing neural networks architectures.
INTRODUCTION
Supervised machine learning is the most widely used form of machine learning. The possibility to exploit expert knowledge through human annotations allows for simple and direct learning of the data features. As consequence, the performance and quality of the results obtained by model are directly linked to the amount of high-quality data available. A clear example of this phenomenon can be seen in machine translation: state-of-the-arr models are considered almost indistinguishable from human-performed translation (Popel et al., 2020). However, these results only apply in scenarios in which both source and target languages have a large text corpus available; with low-resource languages, neural methods are far from reaching the same level of performance (Artetxe et al. 2018). In particular, it has been observed that neural machine translation methods have particularly steep learning curve with respect to the amount of data, obtaining very high-quality results with abundant data, but performing more poorly than traditional statistical methods in low-resource settings (Koehn) & Knowles] (017). Another field, Emotion Recognition in Conversation (ERC), also suffers from the lack of sufficient availability of high-quality data. Poria et al.] (2019) the main challenges of producing ERC datasets are reported: emotions subjective topics, it is hard to define an agreed-upon standard on how to generate broadly useful labelled data. The existing models, used to annotate datasets, can be classified according the following dichotomy:
Categorical models: Each data point is given one or more labels, chosen from a fixed discrete set of possibilities. Therefore, each label defines cluster of items that share one (or more) common features. As the annotation process is generally performed by humans, the boundaries of each cluster are defined qualitatively and not quantitatively.

---

Under review as a conference paper at ICLR 2022
Table Emotion taxonomies used by different datasets.
EmoContext
IEMOCAP
Emotionlines
DailyDialog
Neutral
X
X
Happiness
X
Sadness
X
Anger
X
Frustrated
Excited
X
Disgust
Fear
X
X
Surprise
Other
X
Each data point is labelled by projecting it into multi-dimennional continuous space. a consequence, the annotation more fine-grained, and it conveys information about the structure of the space itself. Therefore, it is possible to rely the properties and metrics of a vector space to gain more insights on the data. This is not directly feasible with categorical labels.
Dimensional models:
Thus, dimensional datasets are more versatile, but more difficult to produce. In fact, it is challenging to guarantee high inter-annotator agreement during the labelling phase the model complexity increases (e.g.. it is easier to label color as "orange", compared to assigning to it the three RGB values (255, 165,00 0)), especially subjective tasks. For this reason, categorical models are gen- erally preferred over dimensional ones. However, categorical datasets are difficult to work with, as comparison between models is not immediate, since there is not an explicit relation between multiple labelling systems. Therefore, given specific task, it is not trivial to merge different data sources to obtain a larger shared training corpus. Although the majority of ERC datasets (e.g., IEMOCAP (Busso al. 2008), Emotionlines (Chen al.) 2019), DailyDialog (Li et al.) (017). and EmoContext (Chatterjee et al. (019)) consist of utterances and categorical emotional labels as- sociated with them, they are based on different taxonomies. EmoContext, for example, subdivides emotions into four categories, and IEMOCAP uses six of them, while Emotionlines and Daily DialyDialog rely on the scheme proposed by Ekman (1993) (Table T). On the other hand, there exist few datasets based on the dimensional model proposed by Russell & Mehrabian (1977), according to which each emotion can described by three independent real values: Valence, Arousal, and Dominance (VAD). An example is the EmoBank dataset (Buechel & Hahn (017).
The method proposed this paper is able to relate categorical models across datasets. As done by Sha & Lukasiewicz (2021). when talking about categorical models, we will refer to each label class (e.g., object, color, ...) as style type, and to each possible value within a label class (e.g., dog, cat, red, yellow, ... .) style value. We assume that each data point is generated from the true world simulator Sim given the vector v = (V1, U2,.... Va) E V C Rd of true continuous Sim(v). Therefore, each style type is determined by subset V, = {vi,, ", "., of the generative factors. In addition, we assume that the distributions of each style value representation can be approximated by Gaussian distributions. The architecture performs explicit multi-type disentanglement, preserving the original latent structure of the gener- ative factors. That is, given a labelled sample (ax, t.), where i, the style value assigned to the style type s. we are able to correctly approximate p(0e, |x, t,), where òt, is the style representation disentangled from æ, which lies in a space that an affine transformation of the original V. Thus, the representation U.,, which is obtained by sampling from the obtained distributions or by using maximum likelihood approach, can used to infer the true values that determined the feature of labelled with s;. Being based neously applicable to multiple style types. Therefore, as we are able to compute the true generative factors determining each style value I, it is possible project each data point in shared dimen- sional continuous space representing each style type, independently of the used categorical model, thus merging different datasets.
generative factors, that is,
multi-type disentangiement architecture, the method is simulta-
The model is based on an extended version of the architecture proposed by Sha & Lukasiewicz 2021) in which explicit sampling layer is inserted between the encoder and the decoder of
2

---

Under review as a conference paper at ICLR 2022
standard variational autoencoder model (Kingma & Welling 2014). The sampling layer learns and stores information related to the ground-truth distributions of the style values for each style type. The encoder generates content vector and multiple style vectors (one for each style type) that are forced to be close to the correct explicit distributions. Then, for each style type, the sampling layer produces a new style vector that is used by the decoder to reconstruct the original input sample. Compared to the original architecture, instead of relying on a single distribution, we sample new style vector from each style value's distribution and combine them through weighted sum, based on probability score. By doing so, we are able to enforce a smooth transition among similar style values, as each style vector generated not only by its own style value's 's ground -truth distribution, but also from neighbour ones. As a consequence, the underlying latent space is continuous.
We conducted experiments on two datasets, dSprites (Matthey et al., 2017) and Base Face Model 2019 (used to generate a collection of almost one hundred thousands sample images) (Gerig et al.) 2018). by labelling the data points with different categorical models and trying to reconstruct, through disentanglement, the original real-value generative factors. The results show perfect cor- relation between the encoded and original values. Furthermore, as each style value's distribution is explicitly stored within the model, we observe that the latent space produced by the model is an affine transformation of the original one. The contributions of this paper are briefly as follows:
We propose model that is able to perform an explicit disentanglement, relying exclusively categorical annotated data. The disentangled style representations lie in a space that is an affine transformation of the true generative factors' one.
As a consequence, we propose a framework that is able to discover the true generative factors influencing each style type and determining each style value, making it possible to merge multiple datasets that are annotated with different categorical models.
The rest of this paper is organised as follows. Section 2 discusses similar methods in the literature that led to the development of the novel architecture in this paper, highlighting the limitations that it able to overcome. Section 3 describes in detail the new model. We then present the experiments performed and the obtained results in Section 4 and our conclusive thoughts in Section 5.
RELATED WORK
Disentanglement wide-spread technique, used to isolate the true factors generating the input data, as described in the original B-VAE paper (Higgins et al. 2017). However, most techniques rely on implicit methods that do not enable for an easy interpretation of the disentangled values. Some components may be pruned (Stühmer et al. 2020) and quantitative measures of disentangle- ment are still in their infancy (Chen al.] 2018). Latent traversals can provide a qualitative sum- mary of each latent unit's 's encoded features, but they rely on human expertise and subjectivity and, therefore, may be difficult to apply in real-world scenarios to establish the quality of the obtained disentanglement. In comparison. explicit disentanglement (i.e., the model targets specific factors to disentangle, guided by annotated data) produces easily interpretable components, as it only requires always
to check that the latent units define the expected features. Since the annotated features categorical labels, previous explicit disentanglement works (John et al. 2019: Romanov et al. 2019)] Sha & Lukasiewicz] (2021) usually represent the expected feature with a vector, and encouraged to belong to either of the style values. However, these methods are "uncontinuous", namely, in- termediate style values are not encouraged in previous works due the limitation of categorical labels.
In the emotion recognition task in the conversation domain, Wu al. (0019) introduced semi- supervised model (SRV-SLTSM) in order to obtain explicitly disentangled values for the VAD com- ponents of each utterance's 's emotion. However, the architecture requires a percentage of dimensional annotated data to force each latent unit to encode the desired factor, and it relies on the a-priori assumptions the VAD model. Furthermore, it cannot incorporate the information existing in cate- gorical datasets, as the training process is unsupervised (i.e., the model is an autoencoder and relies only on the input samples E and the model's output to compute its loss). Park et al. (2019) proposed method to extract the continuous values among the VAD dimensions using exclusiie exclusively categorical annotations. Their solution outperforms SRV-SLTSM in several metrics, showing the importance of the extra data incorporated into the model, as theorized. Being based on the Earth Mover's Distance
3

---

Under review as a conference paper at ICLR 2022
(EMD) loss (Hou et al. 2016), the architecture relies even more the assumptions made on the latent space and the displacement of the categorical values' distributions in it. In fact, it requires the labels to be ordered and displaced correctly along each axis, according to their true dimensional value joy, sad, happy, anger would be sorted as (anger, sad, joy, happy) on the valence axis). The values used in their experiments estimated by using the RRC-VAD-Lexicons (Mohammad, 2018) and are fixed throughout the learning process, Our proposed method, instead, is able to obtain an explicit and continuous disentanglement, by relying exclusively on categorical data.
APPROACH
We will now analyze single-style type scenario (e.g., where the input data is annotated with single label); however, the described methods can be applied independently to every style vector. As mentioned above, our approach consists of a modification of the original architecture proposed by Sha & Lukasiewicz (202I), which predefined a Gaussian distribution for each style type value, and, given sample (x: t), required every disentangled d-dimensional style representation s generated by the encoder to belong to the style distribution that represents the annotated style type value t.
This is achieved by maximizing the probability
p(t[s) PNor (8 (stp)(()
p(s)
where p(t) is the prior distribution of the style values, and pNor is follows:
oNor (8)() exp(- 2s me) (s-M)))
(2)
V )d det(E,)
Then, new style vector s' sampled from its corresponding correct distribution, according to the input label, and forwarded to the decoder. To make sure that each style distribution actually encodes the correct style representation, they sample from their corresponding distributions a vector s', for each j 1,...,m, m, being T = {41, 1,, I the set of possible style values. Then, each s/.. and thus its distribution, is forced to be similar to the correct style value representation through the following classification loss (to be computed for each style type):
12
1
LeL
Σ [T]
log(pe (tyls))),
(3)
where Pe is defined as:
PNor (3(()))((t)) pe(()1s)) = Ever oN (8 (s(()")((())
PNor (Nor(s/(lt))
(4)
Ever (5,
since we assume that all labels share the same prior probability. The vector s', however, cannot contain any of the unique nuances of the input data point that could be represented in 8: as it is obtained from single style value's distribution, it can only encode the average features shared by all the samples with the same style value.
3.1 ARCHITECTRRE
The general structure of our architecture is unchanged compared to the one employed by Shha & Lukasiewicz (2021): it consists of an encoder, a decoder, and bottleneck layer, in which a resam- pling operation happens for each of the generated style vectors. order to obtain a continuous disentangled representation, we modified the sampling layer. Consider sample (ær, () and its gen- crated embedding h, split into content vector c and a single style vector s, whose possible m style values the elements of T. As described above, a probability distribution Nr is associated with each of them. Sampling a new s' from a single distribution discards any extra information (i.e., detailed features other than the discrete style category) stored in Our method preserves it by sampling a different style vector 8; from each distribution N₁ and summing them to s', weighted by each probability distribution p(tz[s). Formally, the new resampled vector s' is obtained as follows:
s' 3'=' +
8; p(t s).
(5)
4

---

Under review as a conference paper at ICLR 2022
Decuder
Selection
Weighted
Analogous process for each style vector 8ij
sampling through reparametrization trick
A1x
Agl Far
computing probability
Paus Pogs
Poentend Pipeident
forwarding probability
Figure I: Architecture of our method: the ellipses represent the μ and E of the style value distribu- tions used sample the new style vectors.
The vector s' is then fed the decoder. By doing so, obtain continuous latent space, each distribution contributes to the final representation, and therefore similar style value encodings are forced to be close also in the latent space. In addition, by interpolating the representations obtained from different distributions,sines š' a weighted sum of them. In order disentangle the true independent generative factors instead of alternative representation of the style values, we used one-dimensional style vectors. This also defines clear notion of order, induced by the real line. As it is possible for a style value be multidimensional (i.e.. being determined by the combination of multiple generative factors), we allow for multiple style vectors represent the same style value, each learning to encode a different generative factor. As consequence, we have to modify the classification loss proposed by Sha & Lukasiewicz, (201)) as follows (remembering that we are considering a single style type):
encode intermediate style features
La
Σ
log(le(tyls)) s',)),
(6)
iT'
where & is the numbered style vectors reserved to encode a style value, and P. is as follows:
Il PNar (8/
pe /
2/1x)
Ever (Nor(s',,(tt)
Furthermore, assuming d-dimensional style type, with d < k. our method is able to find d, re- porting &o d style vectors as unused. By measuring the KL divergence between style type's distributions. it is possible to quantify the amount of information stored in each latent unit. When the KL divergence is close to 0, there no difference between the style representations among that dimension, and the resampled value encodes only random noise and remains unused. This allows us to better understand the structure of the generative factors' hidden space. The complete diagram of the architecture is given in Figure
3.2 DATASETS
As we focused on defining a general-purpose framework for disentanglement and explainability, we tested our method on synthetic datasets used in disentanglementt-riklen experimental results, we created a modified version of the dSprites (Matthey et al., 2017) and Base Face Model 2019 Gerig al. 2018) datasets. both cases, we replaced some of the continuous
tasks. In order to obtain
'To construct the dataset, collection of 96100 data points generated, using 100 different face IDs and 31 evenly distributed values for both the face orientation ("yaw-range") and illumination angle ("illu-range").

---

Under review as a conference paper at ICLR 2022
far-top-hp
top-ap
top-hp
middle-ap
middle-hp
bottom-hp
bottom-ap
far-bottom-hp
Visualization the dSprites AP (left) and dSprites_HD HP (right) categorical systems on the (solid color) and y (dotted line) coordinates. The labels refer to the y value. Given the image data img, the model receives as input the tuple (ing, right-ap, top-ap) or the tuple (ing, far-right-ap, top-hp), depending the categorical model chosen.
Figure 2:
values used to generate each data point with specific categorical labels, based on predefined ranges among each dimension. The model is then given access only on the sample image and its categorical labels, while the true generative values are used as ground truth to be compared with the obtained results in order to evaluate the model's performance. dSprites a dataset composed by 737280 square images of 64 x 64 pixels. Each image is determined by five factors that can assume different values a specific range: shape, scale, rotation, x, and y. We defined two categorical models. Each uses one-dimensional style types (i.e., that rely on single generative factor) in order to label the x and values separately. A third one, instead, defines a single two-dimensional style type to describe the combination of the and y coordinates. The ranges and labels used for each categorical model are as follows:
dSprites AP (Average Precision): The x value (in pixels) is labelled according to the ranges RAP = {00, 23). [24, 39], [40, 63]} corresponding the three style value names left-ap, center-ap. and right-ap. Analogously, y is labelled according to RAP, using the style value names top-ap, middle-ap, and bottom-ap.
dSprites HP (High Precision): The value is labelled according to the ranges RuP [26, 37]. [38.52] 52), [53, 63]} and the style value names far-left-hp left-hp, center-hp, right-hp. and far-right-hp. Analogously, y is labelled according RuP, using the style value names far-top-hp, top-hp, middle-hp. bottom-hp. and far-bottom-hp.
dSprites Co (Compound): The x and values are labelled as a single feature according to the ranges Rc. = RAP RAP. This defines nine style value names, defining a 3 x grid over the input images. The used names are ij-co, each identifying the grid cell at row and column j.
A single shape type used to generate the dataset associated with each categorical model (c.g.. dSprites AP is built using the elliptic shapes). visualization of the dSprites AP and dSprites HP categorical models is given in Figure The dataset generated using the Base Face Model 2019 was processed in a similar way, defining five evenly sized ranges for the face orientation (i.e., the "yaw-range" value, ranging from -90 +90 degrees) and three for the illumination angle (i.e.. the "illu-range" value, ranging as well from -90 to +90 degrees). A total of 96100 images were generated and labelled, defining the BFM_2019 dataset.
EXPERIMENTS AND RESULTS
To our knowledge, there are no previous works that analyze the relation between explicit disen- tanglement and categorical annotations. Therefore, we describe the metrics used to evaluate the model, and we briefly introduce the results obtained by using the original architecture by Sha & Lukasiewicz (2021). in order to establish baseline. As we are interested in the structure of the latent space in which the disentangled style representations are embedded, we focused on the dis- placement of the distributions and the performance in encoding and reconstructing intermediate style values. As our method relies on one-dimensional style vectors (i.c., scalar variables), we used the same one-dimensional style encoding also for the architecture by Sha & Lukasiewicz (021 to

---

Under review as a conference paper at ICLR 2022
014
tpoch
tpoch
a Lukusiewizz, 2021]=--w (Shu Lukasiewizz, 2021)
yaw wikasiewizz 2021) (Shu Lukasiewic, 2021]
(ours)
(ours)
yaw (ours)
illu (ours)
Figure 3: Correlation between the style distribution means and true label range average value for each disentangled style type in the dSprites _HP (a) and BFM 2012 (b) datasets. The "illu" style type has only three possible style values, and therefore it is straightforward for both architectures to achieve optimal results.
Ranges encoded by each style value's distribution compared to the true label values (the latent space is clipped according to the distributions' standard deviation, considering one standard deviation away from the mean.)
Table 2:
Style value far-left-hp left-hp
Distribution's range
Size %
Label's range
Size %
2.32, 1.45
19.8%
0, 10
17 17.2%
(-4.5,, -0.72
16.6%
[11, 25
23.4%
center-hp
(-0.72, 0.11
18.9%
26,33 37
18.8%
right-hp
(0.11, 0.98
19.8%
138, 52
23.4%
fur-right-hp
(0.98,
21.8%
[53, 63]
17.2%
get a fair comparison. It is true that, by using multidimensional style vectors, better results could be achieved. The encoding latent space, however, would be completely unrelated to the original generative factors, and a comparison between different categorical models unfeasible.
We trained the architecture by Sha & Lukasiewicz (2021) the dSprites_AP and dSprites datasets and analyzed the style representations in the latent space. By measuring the correlation between each style value distribution's 's mean and each label's range average value (e.g.. far-lfar-efttip represents æ values in the range [0, 10]. so its average value is 5), we observed that the distribution displacements do not reflect the order induced by their true generative factors, resulting a low correlation. For example, considering "Sprites_HP and the coordinate, we require the ordered sequence (far-left-hp, left-hp, center-hp, right-hp, far-right-hp coding space. Our method, instead, achieves an almost perfect correlation for every style value both datasets (Figure 3). forcing the distributions to reflect the order of the ground-truth labelling sequences. This behaviour is consistent across several runs of the algorithm. In contrast, the results using the architecture by Sha & Lukasiewicz (202T) suffer from high variance, as any random initialization of the model's distribution parameters leads different displacement, and thus relation. Appendix shows several examples of the distributions' displacement for both datasets. This is possible, the parameters defining each distribution are explicitly stored inside the model.
order to obtain a continuous en-
Having continuous encoding space is necessary to represent intermediate style values. To show this, we performed latent traversals the embedded vectors produced by the architectures trained on the dSprites HP dataset. The model by Sha & Lukasiewicz (2021) encodes each style value as an independent group of data points. There is no smooth transition for intermediate representations and the discontinuities in the latent space can be clearly observed (Figure 4ap. In comparison, our method shows an ordered sequence of data points on both the x and y traversals (Figure 4b).
As dealing with linear transformations, also the reversed sequence would be acceptable

---

Under review as a conference paper at ICLR 2022
(a) Sha & Lukasiewicz 2021)
X
(h) Ours
Figure 4: Latent traversals performed on dSprites HP by fixing all the latent units except the ones encoding the x and values, whose value is increased from left to right. In (a), the discontinuity points can be clearly seen in the reconstructed samples, which fail to encode intermediate style values, and instead present two shapes. There is no smooth transition between different style values. Instead, possible to identify the groups defined by the five style values as each one produces almost identical images (b) shows smooth transition across both dimensions.
0.90
0.50
0.40
0.00
000
12
150
EFO
A9
Generative value
dSprites_ HP (Sha Lukasiewicz, 2021)
dSprites_ (ours)
(a)
dSprittss (ours)
Ground
(b)
Figure 5: (a) Style value representation density functions of the x coordinate for both dSprites HP and dSprites_AP. The distributions are displaced in the correct order (i.e., far-lftih left-hp, center- hp, right-hp. far-right-hp and left-ap, center-ap, right-ap) and overlap nicely to represent interme- diate values. (b) Values encoded in the latent unit corresponding to the style value. Our method shows a high correlation with the ground truth, across both datasets.
In addition, we compared the relative size of the ranges used to define each style value with the ranges defined by the model, obtained by considering the maximum likelihood style distribution for each possible value of the latent space. Table reports the results obtained by our method on the r coordinate in the dSprites_HP dataset. There high correlation between the two spaces, showing that the encoded latent space is an approximation of an affine transformation of the original one. As a consequence.
possible train single model on both the dSprites_AP and dSprites_HD datasets simultancously. by using two sampling layers (one for each dataset) that share the same encoder and decoder. The result is that the distributions in both layers belong the same space. The representations of similar style values in both categorical models are, therefore, similar even in the latent space (Figure 5a). This would not happen if each distribution was learnt independently, since it would be displaced randomly and not uniformly across datasets. If we compare the normalized latent values for the r generative factor, we see that our method produces similar encoding for both dSprites datasets, highly correlated with the ground-truth value. This does not happen using the architecture by [Sha & Lukasiewiczl (202I), as each style value representation is independent and does not follow precise structure (Figure 5b
8

---

Under review as a conference paper at ICLR 2022
T-00
0.66
00
21-00
1.00
0.00
0.33.
22
13:00
1.40
24-c0
11
1-00
32-co
0.66
530
0.33 R, grid (normalized)
1.00
44
far-lefl-hp
left-hp
center-hp
right-hp
far-right-hp
(b)
Figure 6: Displacement of the style distribution in the latent space. (a) shows the disentanglement of the generative factors among each dimension of a two-dimensional style type. (b), instead, demon- strates how, when & > d, the distributions in the unused dimensions do not encode any information, as they are completely overlapping, and therefore every style representation appears as the same.
We used the dSprites_Co dataset to study the disentanglement of cilli-dimensional style types. Since the labels used in the dataset are determined by the combination of the two independent factors x and we defined the style type "position" with nine possible style values (i.e., the labels ii-co, i E (,,, and j E {1, (.) We set & = 2 and expected the model to disentangle the two generative factors on the two available dimensions. By normalizing each dimension and comparing the displacement of the distribution with the grid induced by Rc.. We observe high correlation with the ground -ruthh values (Figure ba). confirming the capability of the model to disentangle them. In addition, we analyzed the scenario in which k > show how our method uses only the strictly necessary number of dimensions to represent a style value. corresponding to the underlying number of generative factors. By setting for the style type factor x (and analogously for y) only uses one of the two available dimensions, as expected. In fact, the distributions in the second dimension associated with the style type completely overlap with each other, having pairwise KL divergence approximately equal to 0 (Figure (6)). As the model has to sample a new style vector from its corresponding style distribution, and given that all the probability densities equal, conclude that the latent unit can only encode random noise, not being able to distinguish different style values.
2 and training on dSprites HP,, we observe that the encoding
CONCLUSION
We have shown that current methods performing explicit disentanglement are not suitable identify the true generative factors corresponding to each style type. Instead, an independent representation in a different latent space is learnt. Since there is no enforced structure the new space, the style representations depend exclusively on the random initialization of the model parameters. As a consequence, it is not possible to project different categorically annotated datasets into the same dimensional space. Furthermore, as the focus of the architecture by Shha & Lukasiewicz (202T) is on each single style value, seen as category of objects that share the same feature, the model fails to represent the unique characteristics of each input sample, and therefore to provide smooth transi- tion between similar style values. To fix these disadvantages, we obtain unified style representation across different categorical models, and making it consistent with the generating space by forcing addition, since similar style
an ordering among the distribution on each encoding dimension. model is able to encode individual nuances of each
values now also close in the latent space, data point. By doing so, it is possible to label each data point with its true dimensional attributes, allowing for multiple categorical datasets be merged together.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, 2018.
Sven Buechel and Udo Hahn. EmoBank: Studying the impact of annotation perspective and repre- sentation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp.
578-585. Association for Computational Linguistics, 2017.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N. Chang. Sungbok Lee, and Shrikanth S. Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(())335- -599,,3599--0 2008
Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. SemEval-200 task 3: EmoContext contextual emotion detection in text. Proceedings of the I 13th International Workshop on Semantic Evaluation, pp. 39-48. Association for Computational Linguistics, 2019.
Sheng- Ych Chen, Chao-Chun Hsu, Chuan Chhan-Chunn Kuo, Ting-Har K. Huang, and Lun-Way Wei Ku. Emo- tionlines: An emotion corpus of multi-party conversations. In Proceedings of the 11th Interna- tional Conference on Language Resources and Evaluation, LREC 2018, pp. 1597-1601, 2019.
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 2615-2625, 2018.
Paul Ekman. Facial expression and emotion. The American Psychologist, 48(4):384-392, 1993.
Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schoenborn, and Thomas Vetter. Morphable face models - An open framework. In Proceedings of the 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pp. 75- 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, 2017.
Le Hou, Chen-Ping Yu, and Dimitris Samaras. Squared earth mover's distance-based loss for train- ing deep neural networks. CoRR, abs/1611.0591 1.05916, 2016.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 424 434. Association for Computational Linguis- tics, 2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes CoRR, abs/1312.6114, 2014.
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. CoRR, abs/1706.03872, 2017.
Yanran Li, Hui Su. Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 986 995. Asian Federation of Natural Language Processing, 2017.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Disentanglement 2017.
testing sprites dataset.
Saif Mohammad. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 174-184. Association for Computational Linguistics, 2018.
10

---

Under review as a conference paper at ICLR 2022
Sungjoon Park, Jiseon Kim, Jaeyeol Jeon, Heeyoung Park, and Alice Oh. Toward dimensional emotion detection from categorical emotion annotations. CoRR, abs/1911.02499, 2019.
Martin Popel, Marketa Tomkova, Jakub Tomek, Lukasz Kaiser, Jakob Uszkoreit, Ondiej Bojar, and Zdenčk Zabokrtský. Transforming machine translation: A deep learning system reaches news translation quality comparable to human professionals. Nature Communications, 11(1). 2020.
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard H. Hovy. Emotion recognition in conversation: Research challenges, datasets, and recent advances. CoRR, Abss/105.00997, 2019.
Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. Adversarial decomposition of text representation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 815-825, Association for Computational Linguistics, 2019.
James A. Russell and Albert Mehrabian. Evidence for a three-factor theory of emotions. Journal of Research in Personality, 1 1(33)2773-944 1977.
Lei Sha and Thomas Lukasiewiez. Multi-type disentanglement without adversarial training. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021.
Jan Stühmer, Richard E. Turner, and Sebastian Nowozin. Independent subspace analysis for unsu- pervised learning of disentangled representations. Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 1200-1210. PMLR, 2020.
Chuhan Wu, Fangzhao Wu, Sixing Wu, Zhigang Yuan, Junxin Liu, and Yongfeng Huang. Semi- supervised dimensional sentiment analysis with variational autoencoder. Knowledge- Sys- fems, 165:30-39, 2019.
11

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
We report the displacement of the distribution the latent space for both architectures and the different datasets (Figure 7 and 8).
1.204
5.466
far-iett-hp
center-ap right-ap
left-hp
center-hu
0.602
2753
right-hp
far-right--tp
0.000
0.000
-444
-2.4
-0.5
1.5
-4.
-3.1
-1.5
0.2
(a)
(b)
0.589
0.904
left-ap center-ap
far-left-hp
left-hp
0.284
right-ap
0.452
right-tp
far-righthhp
0.000
0.000
-0.8
2.4
Figure 7: Explicit distributions of the style representations along the coordinate in the (a) dSPrites AP and (b) dSprites HP datasets, using the architecture developed by [Sha & Lukasiewicz (2021). Analogously, (c) and (d) report the distributions' displacement achieved by our new method on the same datasets. The charts are constructed from the μ and o² values stored in sampling layers of the models. Similar results are obtained along the coordinate.
0.180
8.113
0.888
0.000
yau-1
JaW-2
Jaw-3
Jpw-4
yaw
0.481
0.303
0.277
0.234
0.138
0.002
0.048
0.000
-48
center
Blu
right
Figure 8: Explicit distributions of the style representations for the yaw and illu style types in the BFM 2019 dataset. The displayed order correctly reflects the ground generative values (up to an affine transformation).
12


Reviews:
[{"review_id": "uRi_hRgu0Zh", "reviewer": "Reviewer_w1er", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is as a minor variation of Sha and Lukasiewicz (2021). Specifically, the main modification is that instead of sampling from only the ground-truth latent distribution for the decoder, a weighted sum is used over all style values. Hence, the technical contribution of this paper is not as high as other papers.\n\nThe explanation of the method in Section 3 is not very clear, and it took a couple of reads to understand the method. For instance, what is the overall loss that the model is trained on? If it is a VAE objective, then it can be stated in the first part of Section 3, and all elements of the losses can be explained one by one. In the current form, the text misses a combination of all mentioned loss factors.\nAdditionally, it is unclear why the selection sample $s'$ and the weighted sum are summed together to $\\hat{s}'$. What does $\\hat{s}'$ intuitively represent? For a model with perfect accuracy, $\\hat{s}'$ is twice $s'$. Is it just a simple way of combining the selection and weighted sum? Can there be any issues with this setup?\nNext, in the introduction, it is multiple times mentioned that the model is able to 'correctly approximate' certain distributions. These claims initially suggest that it can be proven that the model always converges to the correct distributions, but the rest of the paper suggests that this is empirical. Further, the latent distribution $p(v|x,t)$ can probably not be found up to isomorphism classes like translation or scaling, but the paper misses a discussion of such.\n\nFurthermore, the presented experiments do not sufficiently evaluate the proposed method. Firstly, the method is only compared to Sha and Lukasiewicz (2021) on the aspects that the proposed modifications were intended to improve on. Since the method of Sha and Lukasiewicz (2021) was not explicitly proposed for this, it is not unsurprising that the proposed method outperforms the baseline. However, no other was compared to it. While there might be not many works on explicit disentanglement and categorical annotation, the experiments need to convey the reader of the usefulness of the proposed method. This can be done on downstream tasks, where it can be compared to many other works. For instance, the motivation given in the abstract was that different labelling systems across datasets do not allow the joint supervised training on all of those. Nonetheless, one straight-forward way of doing it is to train a model with multiple classification heads, one per dataset. The proposed method can then be compared to this setting to show its benefit of transforming the classification into a continuous space.\nThe second issue is that the experiments rely on datasets that were created for this paper specifically and not standard benchmarks. While the fundamental data comes from benchmarks, the task setup is not. Similarly, as in the first point, this does not convey any usefulness of the method.\nFinally, the introduction's motivation is heavily based on emotion recognition, even presenting a table of such in Table 1. Still, none of the experiments in the paper touch upon this area, which convincingly is an area where categorical annotations are difficult. This leaves the reader confused in terms of why the method has been proposed with this motivation if it might not be able to help in emotion recognition.\n\n__Minor comments__\n- Eqn 6 is likely missing a log\n- Figures look in several parts blurry, it is better to use vector graphics where possible\n- Page 3, second paragraph: typo in 'Basel Face Model' \n- A visualization of the second dataset (BFM) is missing, which would help in getting an intuition for the dataset\n\n__Post-rebuttal update__\nI appreciate the authors' response and changes to the submitted paper. While I agree that the paper should not just focus on a single downstream task, I believe it is important to have an experiment at least one of these domains so that the method can be compared to more methods (e.g. the simple multi-head training) to clearly show its benefits. The current experimental evaluations are not sufficient to fully convince the reader. Hence, I strongly recommend including such an experiment in a future version, and keeping my initial score.", "summary_of_the_paper": "This paper proposes a method for projecting datasets with different categorical labels into a unified latent space. It is based on an architecture proposed by Sha and Lukasiewicz (2021) which is a VAE with a structured latent space. The latent space is split into a content and style part, where the style part models the classification of labels, and the content the rest of the features to reconstruct the input. The goal of the proposed modification is to have similar label values close together in latent space while disentangling across labels. In experiments on synthetic datasets, the model is shown to provide the specified properties.", "summary_of_the_review": "In summary, the minor modification of the architecture proposed in previous work and the limited experimental comparison are not sufficient to pass the acceptance level for the paper. Thus, my current recommendation is 'Reject'. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V4Ke1Potr-5", "reviewer": "Reviewer_ugco", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper seems like a very incremental improvement over SL21. It's a small modification on the overall architecture, and the evaluation is done only on a single synthetic dataset. I was a little confused also about the distinction between a single multidimensional style vector and many scalar style values; the paper seems to argue that multiple scalars provide better disentanglement than a single vector but I couldn't follow what this means in practice in terms of the model equations.\n\nThe evaluation is very sparse; it's unclear to me how to interpret these results on small synthetic datasets. The results don't seem substantially better than the SL21 baseline, and it's also not clear how to apply these models to other problems meaningfully.\n", "summary_of_the_paper": "The paper is a follow-up to Sha and Lukasiewicz 2021 \"Multi-type Disentanglement without Adversarial Training\" (SL21 from now on) which uses a slight modification of the technique from SL21 to the new task of training a single model on multiple labeled datasets with different label sets that encode the same information at different levels of granularity (so in a way it's a variant of multi-task learning). The model effectively learns a single continuous embedding for examples from which all label sets are easily predictable, effectively unifying the different label sets. The experiments are performed on modified versions of the dsprites and base face datasets. On dsprits it shows a more smooth representation than SL21.", "summary_of_the_review": "Overall the paper seems very incremental on top of a single baseline, and the evaluation is very sparse. I don't see the novelty or significance here.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MrIRLZobY98", "reviewer": "Reviewer_4rCs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n1. The model shows perfect correlation between latent values and true generative factors on the synthetic data.\n2. The approach is simple and mathematically sound.\n\n[Weaknesses]\n1. The novelty of the approach is very thin. Model is a simple extension of disentangled variational autoencoder using a weighted sum. The idea of using weighted sum instead of single distribution is common in many models including Transformer and GAN. The application to variational autoencoder is straightforward.\n2. The motivation in abstract and title states the need to merge multiple datasets. However, the experiments are only on individual synthetic datasets and are limited to disentanglement use case. Showing examples of using the model to merge datasets and getting improved state-of-the-art would strengthen the claims and support motivation.\n3. The writing of the paper is unclear in several sections. Understanding the approach requires multiple readings as the intuition is not provided well.", "summary_of_the_paper": "This work proposes a continuous disentanglement variational autoencoder. The approach is an extension of the disentangled variational autoencoder proposed by Sha & Lukasiewicz in which they modify sampling layer to use weighted sum of multiple distributions instead of single style vector distribution. This ensure continuity in the latent space.", "summary_of_the_review": "The paper proposes a simple extension of disentangled variational encoder improving the continuity in latent space. The approach is simple and logical but has little novelty and experiments do not showcase the utility in merging datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C_HI8Rj8TZS", "reviewer": "Reviewer_RNFG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n It is interesting to split the multiple factors in variational autoencoder. The proposed method is effective in learning disentangled factors.\n\nWeakness:\nIt would be better if the authors provide  intuitive explanation of some key notations when presenting details of the methods. For example, what is the intuitive meaning of P_Nor (Eq.2). What are the differences between s, \\hat{s}, and s^\\prime? Are all of them the factor representations?\n\nEqu 4 is similar to bayes but missing a prior probability p(tj). Please explain why p(tj) is omitted.\n\nIt would be better if the authors evaluate the methods on emotional data, as they mentioned in the abstract and introduction.", "summary_of_the_paper": "The authors consider multiple disentangled factors in a variational autoencoder. The proposed method extends the structure of Sha&Lukasiewcz(2021) by modifying the sampling layer. In the new sampling layer, multiple factor representation is generated as well as a probabilistic weight for each factor. Then, the final representation is a weighted sum of all the factors.\n\nThe proposed method is evaluated on toy data, i.e., modified dSprites and Base Face Model 2019. The results show that the method is able to encode multiple factors.", "summary_of_the_review": "The idea is interesting and reasonable. \nHowever, the experiments on real data is suggested.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a continuous disentanglement variational autoencoder. The approach is a direct extension of Sha & Lukasiewcz (2021). The proposed method appears effective in learning disentangled factors on synthetic data. However, the approach is a minor change to Sha & Lukasiewcz (2021) that samples a weighted sum over all style values. This limits the novelty of the paper. Additionally, evaluation is only on small synthetic datasets that was created for this paper. The lack of evaluation on standard datasets such as an emotion dataset as motivated in the paper, means the results may be due to data selection rather than a superior method. This raises doubts as to whether the approach would generalize to other datasets. In the rebuttal the authors state they wanted to focus on a synthetic dataset since various metrics are easily method but additional real-world/standard dataset results can be added while keeping the synthetic results.
Decision: Reject