Hierarchical Disentangle Network for Object Representation Learning
Authors: Shishi Qiao
Ruiping Wang
Shiguang Shan
Xilin Chen
Venue:  (2020)

Abstract: An object can be described as the combination of primary visual attributes. Disentangling such underlying primitives is the long objective of representation learning. It is observed that categories have the natural multi-granularity or hierarchical characteristics, i.e. any two objects can share some common primitives in a particular category granularity while they may possess their unique ones in another granularity. However, previous works usually operate in a flat manner (i.e. in a particular granularity) to disentangle the representations of objects. Though they may obtain the primitives to constitute objects as the categories in that granularity, their results are obviously not efficient and complete. In this paper, we propose the hierarchical disentangle network (HDN) to exploit the rich hierarchical characteristics among categories to divide the disentangling process in a coarse-to-fine manner, such that each level only focuses on learning the specific representations in its granularity and finally the common and unique representations in all granularities jointly constitute the raw object. Specifically, HDN is designed based on an encoder-decoder architecture. To simultaneously ensure the disentanglement and interpretability of the encoded representations, a novel hierarchical generative adversarial network (GAN) is elaborately designed. Quantitative and qualitative evaluations on four object datasets validate the effectiveness of our method.

Under review as a conference paper at ICLR 2020
HIERARCHICAL DISENTANGLE NETWORK FOR OB- JECT REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
An object can be described as the combination of primary visual attributes. Dis- entangling such underlying primitives the long-term objective representation learning. It is observed that categories have the natural hierarchical characteristic- s, i.e. any two objects can share some common primitives in a particular category level while they may possess their unique ones in another. However, previous works usually operate in a flat manner (i.e. in a particular level) to disentangle the representations of objects. Though they may obtain the primitives to constitute objects as the categories that level, their results are obviously not efficient and complete. In this paper, we propose the hierarchical disentangle network (HDN) to exploit the rich hierarchical characteristics among categories to divide the dis- entangling process in coarse-to-fine manner, such that each level only focuses on learning the specific representations and finally the common and unique represen- tations all levels jointly constitute the raw object. Specifically, HDN designed architecture. To simultaneously ensure the disentan-
based on an encoder-decome glement and interpretability of the encoded representations, a novel hierarchical generative adversarial network (GAN) is elaborately designed. Quantitative and qualitative evaluations on four object datasets validate the effectiveness of our method.
INTRODUCTION
Representation learning, as one basic and hot topic in machine learning and computer vision com- munity, has achieved significant progress in recent years on different tasks such as recognition (Rus-) sakovsky et al., 2015), detection (Ren et al.) 2015 Redmon et al. 2016 Liu et al.) (016b) and gen- cration (Goodfellow et al., 2014), benefiting from the rapid development of representation learned by deep neural networks. Considering the strong capacity of deep representation, in this paper, we mainly focus on the deep representation learning framework.
Despite great success the deep representations have achieved mentioned above, two important problems are still unresolved or less considered, i.c. the interpretability and the disentanglement of the learned representations. In the past decades, various works have been developed reveal the black box of deep learning (Zeiler & Fergus 2014 Dosovitskiy & BBrx 2016b) Bau et al.) 2017] Simonyan al.) [2013 Stock & Cissé] [017] Zhang al.] (017) and move us closer to the goal of disentangling the variations within data (Reed et al.) 20144] Mathieu et al., 2016 Rifai et al. 2012
Tran al. 2017 Gonzalez-Garcia et al., 2018 |Hang al.) 2018] Chen et 2016). Even though they have brought great insights to us, they still have some limitations. For instance, (Chen et al,, 2016 Xie et al., 2017 Zhao et al., (017) learn disentangle variation factors within each category using generative models, instead of investigating the similarities and differences among categories, leading to poor discriminability. Therefore, the learned representations would not well conform to human perception. Though Gaorzalez-Gariia et al. 2018 Huang et al., 2018) try to obtain the domain-invarien and domain-specific knowledge, they can only handle two categories one time, which is not that efficient. In this paper, we attempt to learn disentangled representations in more natural and efficient manner.
Let us first discuss how humans understand an object. Generally speaking, object can be regarded as the combination of many semantic attributes. Hundreds of thousands of objects in the world can be clustered and recognized by humans just because can figure out the common and unique

---

Under review as a conference paper at ICLR 2020
Purt ture
Common
Fumiture
Unique teatures
Agggegatton
Table
Clue ctai
Loveseer
L-couch
balc
Work
Bilaad
table
Tuerris table
Urique features
AApresçation
Billand table
(a) Hierarchical structure
(b) Hierarchical features extraction
Figure 1: An illustration of a hierarchical structure (a) and extracting the hierarchical features that constitute leaf-level image (b). In (b), the common features that only contain the information of its being the root category are first extracted. By tracing from the root to leaf, the unique features that contain additional information of its being the finer-grained category are further extracted.
attributes of object compared to others. Besides, man who never play the billiards only recognize a table in an image, while a sports fan may regard it as a billiard table. Both of them are right since categories have natural hierarchical structure. As shown in Fig. [[a]] given six leaf-level categories, they can be organized in a three-level hierarchy considering the common and different features they have. Each child category in the hierarchy special case of its parent category since it inherits all features from its parent category and has extra features that are not present in its parent category. From another perspective, each parent category the abstraction of all its child categories considering it contains the attributes that are present in all its child categories. Then we come back to the task of disentangling representation learning. It aims to learn the representation encoding useful information that can be applied in other tasks (e.g. building classifiers and predictors) (Bengio et al.) 2013). Taking the hierarchical nature of categories into account, we only learn the representations of object in a flat manner for a specific category level as previous works do, it will not be scalable and comprehensive for the machine to be qualified for various tasks the real world.
Our work aims to exploit the natural hierarchical characteristics among categories to divide the rep- resentation learning in a coarse-to-fie to-fine manner, such that each level only focuses on learning the specific representations. For instance, given billiard table image in Fig. [[b] it tangles the in- formation of being a furniture, a table and a billiard table. We first extract the features that only contain the information of furniture from the image. By tracing from the root leaf level, more and more information is extracted until we can recognize its belonging categories in all hierarchi- cal levels. By doing so, the disentangled representations are expected to find wide and promising applications. For example, one can transfer the semantics in a specific category level from one ob- ject another while keep information of other levels unchanged. Besides. it would help for the hierarchical image compression task using different levels of the disentangled representations. To achieve the objective of hierarchical disentangling and simultaneously interpreting the results so that humans can understand, we propose the hierarchical disentangle network (HDN), which draws lessons from hicrarchical classification and the recent proposed generative adversarial nets (Good-) fellow al. 2014). Extensive experiments are conducted on four popular object datasets to validate the effectiveness of our method.
2 RELATED WORKS
Disentangling Deep Representations. The goal of disentangling representation learning is to dis- cover factors of variation within data (Bengio et al. (013)). Recent years have witnessed substantial interest on such research area (Tenenbaum & Freeman, 1996). including works based on deep learn-
ing (Reed et al., 2014 Mathieu et al. [016]] Rifai al. [0012] Wang et al. 2017 Tran et al. 2017 Gonzalez- et al. 2018 Huang et al. 2018 Chen et al. 2016). (Rifai et al. 2012) is proba- bly the earliest to learn disentangled representations using deep networks for the task of emotion recognition. (Reed et al., 2014) is based on a higher-order Boltzmann machine and regards each factor variation of the manifold as its sub-mare mobboodnifodd. (Mathieu et al.) 2016) and (Chen et al. 2016) leverage the generative adversarial nets (GAN) to learn factors of variation. Recently, cross-domain translation methods (Gonzalez-Ganee and domain- -emaine representations. These works ignore the existing natural and inherent hierarchy relationships among categories, with which we can conduct the disentangling in a coarse-to tor--to-fiii ---fine manner such that each level only focuses learning the specific representations.
et al. 2018 Huang et al., 2018) learn the domain-invariant invariant
2

---

Under review as a conference paper at ICLR 2020
Reconstructed images
Resifial Bldda
Samplt
Blocks
Rasic Prshare
(ampling
Down Samsingg
1 ecna
MLP
corry
sofe talle Ballrth tenris;
enrrrchccal unique features
Images
Encoder
Decoder
Adversorial
Hierarhhccl classficcation
Figure 2: An illustration of the framework of our method. Given images belonging a hierarchy, the common feature maps of their being the root category and the unique features in different non- root levels that can further distinguish them as finer- finer-grained categories are extracted by the upper residual and bottom 1*1 convolutional branches, respectively. The unique features are transformed as the parameters of Adaptive Instance Normalization (AdaIN) and thus be aggregated into the com- mon feature maps to obtain comprehensive representations of images. To ensure disentanglement of the unique features, different levels of them are randomly combined and then reconstructed the image space where adversarial loss and hierarchical classification loss are elaborately designed.
Network Interpretability. Network interpretability aims to learn how the network works via visu- alizing it from the perspective that humans can understand. These methods can be briefly divided into two groups according to whether the visualization is involved in the network during training, i.e. the off-line methods and online methods. The off-line methods make attempts to visualize patterns in image space that activate each convolutional filter (Zeiler & Fergus 2014 Dosovitskiy & Brox, 2016b at Bau et al., 2017 2019) or to interpret the area an image that is responsible for the net- work prediction (Simonyan an et al. 2013 Fong & Vedaldi 2017: Zintgraf et al., 2017 Abbasi-Asl & Geirhos et al.) 2019). While such methods can explain what has already been learned by the model, they cannot improve the model interpretability in return. Instead, the online works propose to directly learn interpretable representations during training (Li et al.] 2018 Zhang al. 2017). However, these methods mainly focus figuring out the running mechanism of networks while paying less attention to dissecting variations of the features among categories, which cannot make models really understand their inputs.
Yu 2017; Stock & Cissé 2017: Palacio al.) Geihho
Hierarchy-regullrizee Learning, Semantic hierarchies have been explored in object classification task for accelerating recognition (Griffin & Perona| [2088] Marszalek & Schmid 2008), obtaining sequence of predictions (Deng et al.] 20122] et al. 2013). making use of category relation graphs (Deng et al., 2014]
al. (2015). and improving recognition performance as additional supervision (Zhao et al. 2011 Srivastava & Salakhutdinov 2013 Hwang & Sigal 2014) Yan et al., 2013 Goo al. [0166] Ahmed et al.j 2016). While these discriminative classifiatioon works have achieved their expected goals, they usually lack interpretability. To address such issues, (Xie et al., 2017 Zhao et al. 2017) propose use generative models disentangle the factors from lovelevel representations to high-level ones that can construct a specific object. (Singh et al.) 2019) uses an unsupervised generative framework to hierarchically disentangle the background, object shape and appearance from an image. However, they either deal with each category in isolation ignore the discriminability of learned features, and thus cannot accurately disentangle the differences and similarities among categories.
3 HIERARCHICAL REPRESENTATION LEARNING
Supposing that category hierarchy is given in the form shown in Fig. ((a) we use 1 = 1,... L to denote the level of hierarchy (L for the leaf level and for the root level), K; to denote the number of nodes at level I, nf to denote the k-th node at level 1, and C# to denote the number of children for n/. As illustrated in Fig. ((b) given an original object image denoted as 1 our goal is to extract the feature F, in the 1-th level.
Generally speaking, object O can be described as the combination of a group of visual attributes:
0
A + +A + A ... + A₂ A1+1 + Am + A())
(I)
Ievel=1
Jenel=2
Ievel=!
3

---

Under review as a conference paper at ICLR 2020
where AO)) represents currently undefined attributes existing on O. As we have discussed, humans classify O in a particular category level according to a subset of the whole attribute set in Eqn.(T)
Take the object in Fig. [[b] for example, it can be regarded as a furniture since it contains the attribute subset (Ai +... + A; f. and be classified to a table in terms of the attribute subset ( (Aı + + A, +
A1+1 +.. ) present in it. Therefore, the disentangled feature F, for our objectíves in Fig. 1(b) is actually the reflection of the attribute subset formulated in Eqn. (T). Moreover, since the hierarchical correlations (i.e. the inherited relationship) among categories in different hierarchies, obviously the subset {Aı + + A; As+1 + ... + A includes the subset {A1 + + A naturally leading to the disentangled F being the proper subset of Fj.
Taking these into consideration, we design the hierarchical disentangle network (HDN) based on the autoencoder architecture in Fig. 2 The encoder E dissects the hierarchical representations given a semantic hierarchy. The decoder G plays the role of an interpreter to reflect the variations of semantic in the image space for different hierarchical levels guided by the hierarchical discriminator Dady and classifiers Det (they share most network architecture except the output layers).
3.1 TOP-DOWN LEARNING OF HIERARCHICAT. REPRESENTATIONS
Since F(-1 is the proper subset of F,, once F is obtained, only the difference R, (1 < < L) between F, and F needs to be encoded. Considering these, we devise top-down representation extraction scheme.
Given Fi-1 and R,, aggregate them together to obtain the whole representation in the I-th level. Such procedure can be formulated as:
Fi-1 R
where means information aggregation. In summary, for hierarchical disentanglement, the com- mon feature F1 in the root level and the unique ones {R, {R. in deeper levels need to be encoded.
To further interpret the semantics of these features to humans, the decoder reconstructs them in the image space. The semantics of F are shared among all its offspring which can be regarded as the invariant content of the object, while those of (R,) are unique for different levels which plays the role of the variant style of the object. Therefore, F₁ and (Rt) are processed in the upper and bottom branches respectively to make them play different roles during the reconstruction, as shown in Fig.
3.2 CONSTRAINTS FOR THE LEARNING PROCESS
The basic constraints of hierarchical disentanglement are making features in different levels perform their own duties. For an object O, the encoded F, and {Ri are complementary. as the constraints of F2 being the proper subset of F1+ F, should encode just right information to describes its being the root category. Progressively using R,, one can distinguish from other categories in I-th level.
Apart from disentanglementt visualization of features in the image space is also one of our objec- tives. We turn to the popular conditional generative adversarial nets (cGANs) (Mirza & Osindero 2014) which can control generated images given conditions. Our HDN leverages the disentangled features F, and (R to control the variations of reconstructed images in different category levels.
To ensure F1, (R, J be well disentangled, we propose random combination strategy for different levels of features from different objects and control the generated images through these combined features, as shown in Fig 2 Specifically, given F!. (R. L and F?. [ from arbitrary two objects, we obtain the newly combined features F, and {Ri} where VI S R (F, if 1) come from either the first or second object. The newly combined features are aggregated together as the input for the decoder G to generate a new object image 1 Such image should satisfy the following losses:
Hierarchical classification loss. For each level, 1' should be classified to the category that R reflects (root level F, only contains one category), defined as:
1
Jas = =Ep (-
ΣΣ
Σ
(Da.
(3)
4

---

Under review as a conference paper at ICLR 2020
where Jets is cross-entropy loss among local categories in each level which have common parent node such as the dashed rectangled categories in the bottom right corner of 2ig p ()) denotes distribution generated images G(F), {R) (=2)) Dete (19) is probabilistic prediction on the c-th local category, and vf is the ground truth local label of the generated object in the (-th level.
Please note that we only focus on the local brother categories instead of all categories in that level. It makes the disentanglement more flexible. On one hand, the classification in each level can thus only focus on the unique features that are just discriminative among those local brother categories. On the other hand, the duties of different levels can well disentangled, since the semantic information encoded in different levels is tangled, after the random combination and image reconstruction, the hierarchical classifiers would be quite confused.
Adversarial loss. We employ GANs to match the distribution reconstructed images to the real data distribution. Specifically, the LS-GAN Mao et al. (017) loss is adopted in light of its stable training, defined as:
JGAN = Ep-pp(C) [(1 Dade (19)²
(4)
Image reconstruction loss. As for Fi and (2) from one same object, should be able to reconstruct as close to the input as possible.
recon = (r) [|| 11]]
(5)
where p' (G) is the distribution of generations taking F1, (R) 12 from the same objects inputs.
Feature reconstruction loss. Apart from the image reconstruction loss, the feature reconstruction loss added to HDN to stabilize the training process.
F.R recon
(R. [[E (R.))2) - (F1, (R) (2)))
(6)
where E(E) is the distribution of encoded hierarchical features E(I).
Now combine the four loss functions defined in Eqn. 3). Eqn. 4). (5n.(5) and Eqn.66 into one comprehensive loss function for supervising the disentangling of E and visualization of G:
J G) = Jats + JGAN + aJJ.coo
BJER recon
where a and are the hyper-parameters to balance the weights of the four terms.
As for the update of discriminator and hierarchical classifiers, use the following loss:
J(D) = (Fprpp(data)
(flog(Dess(()) (T)))))))))
(8)
1=2
(Frap((data) [1
Dado (T)) Eppp(G) (Dado((9))))
3.3 RELATIONSHIP WITH PREVIOUS WORK
is noted that the recent work DTLC-GAN (Kaneko et al. 2018) has similarities with our method on motivations of learning hierarchical representations. Nevertheless, DTLC-GAN is indeed differ- ent from ours. Specifically, the detailed goals of leveraging hierarchical relationship are different. DTLC-GAN aims maximize the mutual information between conditioned representation and data in each level, i.e. study how the appearance of data varies with more and more specific conditions and thus synthesize data with more fine-grained details. Our method focuses more how humans distinguish objects from categories in different hierarchical levels and wishes such manner of un- derstanding objects can be applied the machine, i.e. learn the commonality and individuality of categories in nature. Therefore, the disentangled features of our method mainly served for down- stream discriminative tasks such as semantic retrieval, open world unseen category recognition as we have attempted in the following experiments. Besides, thanks to the disentangled commonality, method can further realize the semantic translations between images by exchanging the individual parts, which has been popular application in real world.
5

---

Under review as a conference paper at ICLR 2020
Source
Target
F
R2
R3
R4
R₂ R2 Rg R R I(R)
Figure 3: Semantic translation results of the source images controlled by hierarchically disentangled features of the targets on CclebA. Different columns denote results of using FI. {Ri or their combinations disentangled from the target images to replace the corresponding levels of the sources. Ground truths of R2, R3, R2 are gender, smile and hair color, respectively.
4 EXPERIMENTS
Datasets: We conduct experiments on hierarchical annotated data from four datasets, typical ex- amples in the hierarchy are shown in Fig Fig and Fig 10 in Appendix" The first is CelebA dataset (Liu al., 2015). It provides more than 200K face images with 40 attribute annotation- Following the official train/test protocol, we define a four-level hierarchical structure which has explicit attribute difference between any two levels. Specifically, all faces (root category) are first divided into two categories based on gender. Such initial categories are further classified according to the smile expression and hair color in the next two levels. With such ground-truth hierarchical annotations, we can validate our method more easily.
The second dataset named Fashion-MNIST (Xiao et al. 2017) is proposed as a direct drop-in re- placement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same train/test split with MNIST. Since such dataset does not provide any hierarchical structure, cluster T-shirt, coat, pullover one super category and trouser, dress as another super one to construct a three-level hierarchy (root is fashion) according to their appearance similarities.
The other two datasets are 3D data, CADCars (Fidler al. 2012) and ShapeNet (Chang et al. 2015). CADCars contains 183 3D Car models and ShapeNet is constitutive of 51,300 3D models covering 55 common and 205 finer-grained categories. Using the provided tools, we generated 24 2D images with 6 pose and 4 illumination variations for CADCars. These 2D data are clustered into four super categories, i.e. minibus, sedan, sports and SUV, and are further divided into 6 finer-grained categories for each super one based on pose annotations, which defines three-level hierarchy. On ShapeNet, 12 2D images with pose variation are obtained for each 3D model. One three-level category-pose hierarchy named as ShapeNet-P similar to CADCars and one three-level hierarchy named as ShapeNet-C as in Fig [ (a) are defined. Ratio of train/test split is 4:1.
Implementation Details: Our HDN is implemented with Pytorch platforn Design of the back- bone follows recent proposed image generation (Karras et al., 2018) and translation works (Huang et al. 2018). Images resized to 128* 128 resolution for all datasets except Fashion-MNIST which Is resized to 28*28. As shown in Fig 2 to match it with our task, we increase the number of 1*1 convolution branches such that originally one representation is disentangled into multiple hierar- chical levels. We also equip the residual blocks with Adaptive Instance Normalization (AdaIN) whose parameters are dynamically generated by a multi-layer perception (MLP) from the disentan- gled latent codes. Besides, D has L output branches, one for real/fake predictions and the others for hierarchical classifications. More training details are given in the Appendix.

---

Under review as a conference paper at ICLR 2020
24
(a) Root Figure 4: 2D iSNE of disentangled F on test set of CelebA for different levels. For easy understand, M and F mean male and female, S and N mean Smile and Neural, BI, G and Br mean Black, Golden and Brown hair respectively.
(b) Level 2
(c) Level 3
(d) Level 4
R₂
Ry+ R
R3
Ryt R
R4
I(R)
Rg+ R3
E(R)
Figure 5: Interpolations of disentangled R, between the source (first columns) and target (last
columns) images (other levels unchanged for the source images).
4.1 DISENTANGLED RESULTS
Firstly, we replace one or several levels of disentangled features for an image with those of another image, and then observe the visual changes of generated image to validate the semantic consistence with pre-defined hierarchical structure. 3 (and Fig 12 Fig in the Appendix) shows such semantic translation results. It is observed that different level of features perform their own duties, i.e they carry just enough information to control the variations within that level (e.g. gender, smile and hair color for CelebA we specially predefined on CelebA), but would not involve more belongs to other level. For instance, in 3 change features of an image in arbitrary one, two or all levels to those of another image, the semantics would be changed correspondingly. Apart from H the common feature F, also encodes information that is not discriminative among its offspring categories but is necessary to construct the object (e.g. the identity, pose and even the background information of a face image). To give a more intuitive feeling about the ability of disentangled features, we investigate the discriminabilites of them via the popular (SNE tool (Maaten) & Hinton| 2008). As shown in Fig f (and Fig [4] Fig T5] Fig T6 Fig [7 in the Appendix), with only the common feature Fi, samples are mixed together. When progressively be combined with features of deeper levels R, samples are better separated and almost consistent with the hicrarchical structure.
Apart from direct visual edit, we also show that one can transform the source image smoothly by linear interpolation (with 5 equally spaced interpolation coefficients from 0.1 to 0.9) of disentangled features between the source and target. Such examples are shown in Fig 5 We can see that genders, expressions, hair colors and their combinations of the source images (first columns each case) can be changed smoothly towards those of the targets (last columns of each case). Learning smooth feature space with continuous variations a significant issue for representation learning task, which can ensure the generalization ability for unseen similar objects. We have investigated this in Sec 4.3 At the end of this section, a quantitative evaluation of these results is conducted.
It is noted that the focus of this paper is to interpret the hicrarchical structure within data. Therefore, we heuristically construct hierarchical structures based on human priors. One can also automatically obtain rea- sonable hierarchical annotations using machine learning technologics such as unsupervised clustering (Goo)
al. 2016) does.
he source codes will be released the public.
CelebA
Fashion-MNIST
CADCars
ShapeNet-CC
ShapeNet-P
Test
Gen.
Test
Gen.
Test
Gen.
Test
Gen.
Test
Gen.
Level 2
0.9570
0.9387
0.9629
0.9779
0.9781
0.9792
0,9941
0.9941
0,9323
0,8863
Level 3
0.9232
0.9103
0.9336
0.8464
0.9798
0.9670
0.9844
0.8865
0.9190
0.8413
Level 4
0.8932
0.8799
Table 1: Accuracy of hierarchical classifications for test and generated (Gen.) images.
7

---

Under review as a conference paper at ICLR 2020
DSH
DS11-S
HasNNe
HashNei-S
SSDII-S
SurGAN
StylGAN
HDDN-B
Tevel 2
0.9523
0.7619
0.9483
8.8177
0.9593
08120
0.7474
0.5297
039771
0.9747
Level 3
0.5010
0.5564
08374
B6338
088455
0.6887
0.5231
0.2726
03589
0.9006
0.6461
0.6356
0.5016
0,1037
0.7052
Table 2: mAP results of retrieval for compared methods in different semantic levels.
To be specific, we use the learned hierarchical classifier D to evaluate whether the semantics are correctly disentangled and decoded into the randomly translated images as in Fig 3l To ensure D is reliable, the accuracy of hierarchical classifications on the test data is given as a reference. Table[[ the evaluation results. Firstly, can be seen that the semantics of translated images with changing different levels are recognized correctly by the corresponding classifiers. Secondly, the deeper of the level, the more difficult of the translation, since the criteria for distinguish one category from others in the deeper level would become more and more complicated (summation of all criteria above this level). Finally, becomes difficult to transfer the unique features and generate images when that information is difficult to be described and disentangled such as in the leaf-level of Fashion-MNIST and ShapeNet-C, which deserves to make more efforts.
4.2 APPLICATION TO IMAGE RETRIEVAL
One of the objectives of learned representations is to be applied in real-world applications. Semantic image retrieval has been studied for years. Hashing is one effective and space-time efficient solution for this task. However, the semantic of target images that users expect are not always consistent just due to the tangled information of objects in different hierarchical levels. In this section, we conduct retrieval in different levels on CelebA. We compare three competing deep hashing methods, i.e. DSH (Liu et al. 2016a), HashNet (Cao et al. (017)) and SSDH (Yang et al.) 2018). and two strong pre-trained GAN baselines, i.e. supervised StarGAN (Choi et al. 2018) and unsupervised StyleGAN[Karras et al. (2018). The backbones of hashing methods are same with the bottom branch of encoder of HDN and pretrained on CASIA WebFace dataset (Yi et al. 2014). In 1-th level, a model with bit-length as same as the dimension of the concatenation of [R trained. As for StarGAN and StyleGAN, the latent features before the last layer of discriminator are used as the representations of samples. To make fair comparison with hashing methods, also binarize the disentangled features via Sigmoid activation fucntion, which we named as HDN-B. Images of the test are used to retrieve the training set.
Table gives the mAP results in different semantic levels. First, our method achieves the best performance, though we did not impose specific metrie objectives on features while the flat StarGAN and StyleGAN perform not well. Second, HDN is more efficient since it only needs one model owing to disentanglement while hashing methods have train a model in each level. We also tried to use only one model trained in the leaf-level to evaluate in high levels (methods with postfix of "-S"), but the results are inferior to those independently trained for each level. Third, HDN-B is better than HDN, which mainly due to the increased non-linear ability of features. Finally, the retrieval of HDN is more interpretable. As shown in Fig with different parts of features, the returned images satisfy different semantic requirement, while for general method like SSDH one can not interpret the meanings of different code parts.
HDN
SSDH
Query
Figure 6: Top-5 returned images of a retrieval case using different parts of features (i.e. different R of HDN, and different bit parts of SSDH). Green and red boxes are correct and false samples judged by the hierarchy.
8

---

Under review as a conference paper at ICLR 2020
CelebA
Shape
ShapeNet-P
Seen
Unseen
Seen
Unseen
Seen
Unseen
Level-2 Acc.
0.9441
0.9650
1.0
0.7019
0.8563
0.7727
Level-3 Acc.
0.8520
0.9450
Leaf-entropy
0.1779
0.3561
0.1204
0.4015
0.1567
0.4913
Table 3: Hierarchical prediction performance for seen test set and unseen leaf-level categories.
Source Target
Result
Source
Target
Result
Source
Target
Result
ShapeNet-C
ShapeNet-P
CelebA Semantic translation results between seen and unseen objects. Here we replace the source images with all levels of R of the targets, i.c. the right most case in Fig
Figure 7:
4.3 UNSEEN CATEGORY PREDICTION AND SEMANTIC EDIT
Recognition of unseen categories is a challenging task for deep learning models, which has high requirements for the generalization ability of models. As our HDN learns features in different hier- archy levels, it can obtain sequenced category predictions for an object. Therefore, if unseen objects share similarities with seen ones, one should still obtain the right predictions those levels. and the predictions among seen categories in levels where unseen objects have their own features should be confused. For levels with similarities and with their own features, accuracy and entropy of predic- tions of a linear hierarchical classifier trained with the disentangled features are used as evaluation metries respectively.
this section, we test HDN on certain unseen leaf-level categories, i.e. bald and gray hair of CelebA, kinds of new tables and sofas of ShapeNet--C C and objects with other poses of ShapeNet-P (typical examples of these data are shown in Fig in the Appendix).
Table sshow the quantitative results. Two conclusions can be reached. 1). In levels where seen and unseen objects share similarities (i.e. gender, smile levels on CelebA, Sofa/Table level on ShapeNet- C, Loveseat/Club chair/Work table/Billiards on ShapeNet-P), most objects be correctly classi- fied. 2). In the leaf-level, unseen objects have the unique unseen features, leading to the prediction entropy increase obviously compared with that of seen objects. Besides, it is found that the unseen objects are more likely classified to similar seen categories in leaf-level. For instance, about 30% and 56% bald faces are recognized couches are predicted as loveseat and L-couch respectively, 44% and 50% of the frontal sofa/table are classified as the right 30 offset of frontal and left 30 offset of frontal. The semantic translations in F between seen and unseen images also verify such results. The semantics of non-leaf levels can be transferred as usual, but the unseen unique features are not. Bald may be disentangled as golden black hair due to the skin color. The material of leather is ignored in ShapeNet-C since model focus more on shape to distinguish seen objects rather than material during training. The translations frontal pose are also confused as be found in the cases of ShapeNet-P. Through this study, we think that disentangling the visual primitives of objects as learned knowledge is one of the most promising solutions for the ability of open-world recognition.
black and golden hair respectively, 50% and 50% leather
CONCLUSIONS
We propose the hierarchical disentangle network (HDN) which exploits the natural characteristic- s among categories to divide the representation learning a coarse-to-fine manner. Our model achieves promising disentangle results. We also show the applications of such disentangled features on semantic translation, retrieval and even unseen objects prediction. However, our work is just an early step towards the long goal of disentangled representation learning, limited by the capacity of generative models on large scale and heavily tangled categories, the performance HDN is not quite well such as the ImageNet dataset which deserves to make more efforts.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Reza Abbasi-Asl and Bin Yu. Interpreting convolutional neural networks through compression. CoRR, abs/1711.02329) 2017.
Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani. Network of experts for large-scale image categorization. In ECCV, pp. 516-532, 2016.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In IEEE,CVPR, pp- 3319-3327, 2017.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolci Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. In ICIR,, 2019.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3798): 1828, 2013.
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S. Yu. Hashnet: Deep learning to hash by continuation. In IEEE, ICCV, pp. 5609-5618, 2017.
Angel Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: information-rich 3d model repository. CoRR, abs/1512.03012, 2015.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, pp. 2172-2180, 2016.
Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung-Wes Woo Ha, Sunghun Kim, and Jaegul Choo. S- targan: Unified generative adversarial networks for multi-domain image-to-image translation. In IEEE CVPR, pp. 7789-8797, 2018.
Jia Deng, Jonathan Krause, Alexander C. Berg, and Fei-Fei Li. Hedging your bets: Optimizing sccuracy-speciffiity trade-off's in large scale visual recognition. In IVER, pp. 3450-3457,
2012.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs In ECCV, pp. 48-64 2014.
Nan Ding, Jia Deng, Kevin P. Murphy, and Hartmut Neven. Probabilistic label relation graphs with ising models. In IEEE, ICCV, pp. 1161-1169, 2015.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based deep networks. In NIPS, pp. 658-666, 2016a.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In JEEE, CVPR, pp. 4829-4837 2016b.
Sanja Fidler. Sven J. Dickinson, and Raquel Urtasun. 3d object detection and viewpoint estimation with a deformable 3d cuboid model. In NIPS, pp. 620-628, 2012.
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In IEEE ICCV, pp. 3449-3457, 2017.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. In ICLR, 2019.
Abel Gonzalez-Gardae Joost van de Weijer, and Yoshua Bengio. saage-to-image translation for cross-domain disentanglement. CoRR, abs/1 805.09730, 2018.
10

---

Under review as a conference paper at ICLR 2020
Wonjoon Goo, Juyong Kim, Gunhee Kim, and Sung Ju Hwang. Taxonomy-regullaizze semantic deep convolutional neural networks. In ECCV, pp. 86-101, 2016.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, pp. 2672-2680, 2014.
Gregory Griffin and Pietro Peroma. Learning and using taxonomies for fast visual categorization. In IEEE, CVPR, pp. 1-8, 2008.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NIPS, pp- 6626-6637, 2017.
Xun Huang, Ming- Yu Liu, Serge J. Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, pp. 179-196, 2018.
Sung Ju Hwang and Leonid Sigal. A unified semantic embedding: Relating taxonomies and at- tributes. NIPS, pp. 271-279, 2014.
Takuhiro Kaneko, Kaoru Hiramatsu, and Kunio Kashino. Generative adversarial image synthesis with decision tree latent controller. In IEEE, CVPR, pp. 6606-6615, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CoRR, 0bs//812.00494, 2018.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In AAAI, pp. 3530-3537, 2018.
Shan Li, Weihong Deng, and JunPing Du Reliable crowdsourcing and deep learning for expression recognition in the wild. IEEE, CVPR, pp. 2584-2593, 2017.
Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. Deep supervised hashing for fast image retrieval. In JEEE, CVPR, pp. 2064 2072, 2016a.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yane Fu, and Alexander C. Berg. SSD: single shot multibox detector. In ECCV, pp. 21-37, 2016b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In IEEE ICCV, pp. 3730-3738, 2015.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2(50)):5579-2600 2008
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In IEEE, ICCV, pp. 2813-2821, 2017.
Marein Marszalek and Cordelia Schmid. Constructing category hierarchies visual recognition. In ECCV, pp. 479-491, 2008.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Dis- entangling factors of variation in deep representation using adversarial training. In NIPS, pp- 5041-5049, 2016.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, abs/1411.1784, 2014.
Vicente Ordonez, Jia Deng, Yejin Choi, Alexander C. Berg, and Tamara L.. Berg. From large scale image categorization entry-level categories. In IEEE, ICCV, pp. 2768-2775, 2013.
Sebastian Palacio, Joachim Folz, Jöm Hees, Federico Raue, Damian Borth, and Andreas Dengel. What do deep networks like see? CoRR, tbs/1883.08337, 2018.

---

Under review as a conference paper at ICLR 2020
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pp. 779-788, 2016.
Scott E. Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning to disentangle factors of variation with manifold interaction. In ICML, pp. 1431-1439, 2014.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. NIPS, pp. 91-99, 2015.
Salah Rifai, Yoshua Bengio, Aaron C. Courville, Pascal Vincent, and Mehdi Mirza. Disentangling factors of variation for facial expression recognition. In ECCV, pp. 808-822, 2012.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, pp. 1226-2234 2016.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013.
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disen- tanglement for fine-gre tineggrinedd object generation and discovery. In CVPR, pp. 6490-6499, 2019.
Nitish Srivastava and Ruslan Salakhutdinov. Discriminative transfer learning with tree-based priors. In NIPS, pp. 2094-2102, 2013.
Pierre Stock and Moustapha Cissé. Convnets and imagenet beyond accuracy: Explanations, bias detection, adversarial examples and model criticism. CoRR, 0bbs/1711.11444 2017.
Joshua B. Tenenbaum and William T. Freeman. Separating style and content. In NIPS, pp. 662-668, 1996.
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning GAN for pose-invariant face recognition. In IEEE, CVPR, pp. 1283-1292, 2017.
Chaoyue Wang, Chaohui Wang, Chang Xu, and Dacheng Tao. Tag disentangled generative adver- sarial network for object image re-rendering. In IJCAI, pp. 2901-2907, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mniss a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/1700.007747, 2017.
Jianwen Xie, Yifei Xu, Erik Nijkamp. Ying Nian Wu, and Song-Chun Zhu. Generative hierarchical learning of sparse FRAME models. In IEEE, CVPR, pp. 1933-1941, 2017.
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. In IEEE, ICCV, pp. 2740-2748, 2015.
Huei-Fang Yang. Kevin Lin, and Chu-Song Chen. Supervised learning semantics-preservinng hash via deep convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(2):437-455 2018.
Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine- grained categorization and verification. IEEE, CVPR, pp. 3973-3981, 2015,
Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. Learning face representation from scratch. CoRR, Abs//441..992, 2014.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pp. 818-833, 2014.
12

---

Under review as a conference paper at ICLR 2020
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. CoRR, abs/1710.00935, 2017.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In IEEE,CVPR, 2018.
Bin Zhao, Fei-Fei Li, and Eric P. Xing. Large-scale category structure aware image categorization. In NIPS. pp. 1251-1259, 2011.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gener- ative models. In ICML, pp. 4091-4099, 2017.
Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. CoRR, Abs//700.00495, 2017.
A APPENDIX
A.1 NETWORK ARCHITECTURES AND TRAINING DETAILS
A.1.1 NETWORK ARCHITECTURES
Following the backbone designs in (Huang et al. 2018) for image-to-image translation task, let c7sl-k denotes a block with k filters and stride 2. Rk denotes residual block that contains two 3 3 convolution blocks with filters. The last layers for different hierarchy levels (common features of root level encoded by the content encoder) in the style encoder include multiple c/s/-8 branches, i.e. x 1 convolution block with filters and stride 1. uk denotes a 2x nearest-neighhor upsampling layer followed by average pooling layer. Instance Normalization (IN) and Adaptive Instance Normalization (AdaIN) are adopted the content encoder branch and decoder respectively. No normalization used in the style encoder branch. Use ReLU activations in the mncoder-decodee and Leaky ReLU with slope 0.2 in the discriminator and classifier. Multi-scale discriminators with 3 scales (single scale for Fashion- MNIST due to its too small resolutions) are used to ensure both realistic details and global structure. The last layer of the decoder equipped with a tanh activations to normalize the values of generated images to the range of 1]. In the following, we give the detailed architectures of each module on different datasets.
convolution block with filters and stride dk means x convolution
5 convolution block with filters and stride GAP denotes global
CelebA, CADCars & ShapeNet:
Content encoder: c7s1-64, d128, d256, R256, R256, R256
Style encoder: c7s1-64, d1 28, d256, d256, d256, GAP, c1s1-8
Decoder: R256, R256, R256, u128, u64, c7s1-3
Discriminator & Classifier: d64, d128, d256, d512
Fashion-MNIST.:
Content encoder: c7s1-32, d64, d128, R128, R128, R128
Style encoder: c7s1-32, d64, d128, R128, R128. R128, GAP, els1-8
Decoder: R128, R128, R128, u64, u.32 c7s1-1
Discriminator & Classifier: d32, d64, d128, d256
A.1.2 TRAINING HYPERPARAMETERS
We use the Adam optimizer with 31 = 0.5, B3 = 0.999, and initial learning rate of 0.0001. We train HDN on all datasets for 300K iterations and half decay the learning rate every 100K iterations. We set batch size to 16. The loss weights a and Eqn. are set as 10 and respectively. Random mirroring is applied during training.
13

---

Under review as a conference paper at ICLR 2020
mmile
Black Hair
Golden Hair
Brown Hair
Male
Neutral
Black Hair
Golden Hair
Brown Hair
mmile
Black Hair
Golden Hair
Brown Hair
Female
Neutral
Black Hair
Golden Hair
Brown Hair
Figure 8: Typical samples of hierarchical data on CelebA. Images within purple rectangular box are some instances of a leaf-level category. Categories within a green rectangular box belong to common super-category. The super-categories within red rectangular box share common ancestor.
Worktable
T-shirt
Table
Billiard
Pullover
Tennis
Fumiture
Coat
Fashion
Club chair
Trouser
Sofa
Loveseat
Bottom
Dress
L-couch
(a) Fashion-MNIST
(b) ShapeNet
Figure 9: Typical samples of hierarchical data on Fashion-MNIST (a) and ShapeNet (b). Images within purple rectangular box some instances of leaf-level category. Categories within green rectangular box belong to one common super-category. The super-categories within a red rectangular box share common ancestor. On ShapeNet, categories within one purple rectangu- lar box can be further divided into four child categories based on pose variations. Therefore, one hierarchy named Shape-C (Category) and another one named ShapeNet-P (Pose) are defined.
A.2 HIERARCHICAL DATA CONSTRUCTION
In this section, Fig Fig and Fig 10 provide leaf-level examples for better understanding the commonalities and individualities among categories in different hierarchy levels. Take the CelebA for example, the root category face has two children distinguished by gender attribute. For each of the two super categories, it includes two finer granular children which are further divided by the smile expression. Finally in the level, each local branch are classified according to their hair colors, i.c. black, golden and brown hair. Within each leaf-level category, samples mainly contain intra-class variations caused by identities, age, pose, etc.
14

---

Under review as a conference paper at ICLR 2020
Minibus
Pose
Pose
Pose 5
Pose
SUV
Pose 1
Psse? 2
Pose
Car
Pose
Pase 5
Sports
Pose 1
Pose
Pose 3
Pose
Pose
Sedan
Fose
Pase
Pose 3
Pose 4
Pose
Pose
Figure 10: Typical samples of hierarchical data on CADCars. Images within a purple rectangular box are some instances of a leaf-level category. Categories within green rectangular box belong to one common super-category. The super-categoric, within a red rectangular box share one common ancestor.
Bald
CelebA
Gray hair
VA
Sofa
ShapeNet-C
Table
ShapeNet-P
Pose
Figure 11: Typical samples unseen leaf-level data on CelebA (bald and gray hair), ShapeNet-C (kinds of tables and sofas) and ShapeNet-P (more poses). Large differences with seen hicrarchical data can be found here.
15

---

Under review as a conference paper at ICLR 2020
Source Target
F2
Rg
Ra
Ry+R
Source
Target
224
(a) CADCars
(b) Fashion-MNN Mishion-MNIST
Figure 12: Semantic translation results of the source images controlled by hierarchically disentan-
gled features of the targets on CADCars(a) and Fashion-MNIST (b).
Source Target
F
Rg
R2+
Source
Target
F1
R2
Ra
RgrR3
ShapeNet-C
(b) ShapeNet-P
Semantic translation results of the source images controlled by hierarchically disentan- gled features of the targets on ShapeNet-C (a) and ShapeNet-PP (b).
Figure 13:
A.3 DISENTANGLE RESULTS ON MORE DATASETS
In this section, give disentangled results CADCars. Fashion-MNIST and ShapeNet. Fig and [3 show the semantics of disentangled features which can well change the variations of generated images. Similarly, [4] Fig F Fig and T7 show that progressively involving more features of deeper levels, samples become better separated, which verifies the results of disen- tanglement consistent with the semantic hierarchical structures.
A.4 CROSS DATASET EVALUATIONS
Learning general representation that can be applied across datasets is one the long goals for ma- chine. In this section, briefly evaluate method datasets which have similar categories but quite different styles. To be specific, we evaluate HDN on a challenging facial expression dataset called RAF (Li et al. 2017) and cars dataset named CompCars (Yang et al. 2015), using models trained on CelebA and CADCars respectively. RAF is a large-scale facial expression database with around 30K great- dreat-diverse facial images downloaded from the Internet. provides expressions, race, age range and gender attributes annotations. Besides. the released images are compactly aligned which have little information about hair colors. Therefore, the leaf-level categories are obtained ac- cording to the race or age range. As for CompCars, it contains 163 car makes with 1,716 car models. Besides, it also labels the car type (i.e. SUV, Sedan, Sports, etc.). Based on these annotations, we
20
60
40
40
30
20
10
10
30
(a) Root
(b) Level 2
Level
Figure 14: 2D tSNE of disentangled F, on test set of Fashion-MNIST for different levels. T and B mean Top and Bottom, Ts, P, C, Tr and D mean T-shirt, Polo shirt, Coat, Trousers and Dresses respectively.
16

---

Under review as a conference paper at ICLR 2020
20
20
G2b 9yy
47
40
20
(b) Level 2 Figure 15: 2D iSNE of disentangled F; test set of CADCars for different levels. M SU, Sp and Se mean Minibus, SUV, Sports and Sedan, P1~ P4 and P6 mean Pose 1, 2, 3, 4 and 6 respectively.
(a) Root
Level 3
20
20
30
20
20
10
15
35
20
(b) Level 2 Figure 16: 2D ISNE of disentangled F, on test set of ShapeNet-C for different levels. S and T mean Sofa and Table, Lo. C. Lc. W, B and Te mean Loveseat, Club chair, L-couch, Work table, Billiards and Tennis-table respectively.
(a) Root
(c) Level 3
29
2000
10
10
20
20
10
10
20
10
20
10
10
(a) Root
(b) Level 2
(c) Level 3
Figure 17: 2D (NNE of disentangled F, test of ShapeNet-P for different levels. L, C, W and mean Loveseat, Club chair, Work table and Billiards respectively.
CompCars
Male
Female
Minbus
Convertble
Hutchback
Pickup
SUV
Figure 18: Typical samples of hierarchical data on RAF and CompCars. Each image is a case of leaf-level category (e.g. different races age ranges, models). Images within green rectangular box belong to one common super-category. The super-categories within red rectangular box share one common ancestor.
17

---

Under review as a conference paper at ICLR 2020
CelebA
RAF
CADCars
CompCars
Seen
Unseen
Seen
Unseen
Level-2 Acc.
0.9441
0.6601
0.9375
0.4510
Level-3 Acc.
0.8520
0.305
0.9375
0.2941
Leaf-entropy
0.1779
0.6153
Table 4: Hierarchical prediction performance of models trained on seen dateset and evaluated on unseen dataset.
"The expression recognition precision is about 50% and that of compound is about 30% using the VGG nets, as reported in (Li et al. 2017). Here level-3 can be regarded as the compound of gender and expressions.
Source
Target Result
Source
Target Result
Figure 19: Semantic translation results between images from different datasets (i.e. image pairs from CelebA and RAF, and from CADCars and CompCars.) using all levels of features of the targets. Face images with compact crop are from RAF. The information gender and smile is correctly transferred. Cars in the right second columns except the last row from CompCars dataset. The car types and poses are changed accordingly.
replace the pose categories in the leaf-level with different car models, using only the profile pose images. Typical examples of the two hierarchical data are shown Fig
Table4 gives the hierarchical prediction comparison between seen (CelebA and CADCars) and un- seen (RAF and CompCars) datasets. Firstly, though data become challenging and have large domain shift, the learned models can still recognize partial objects in high levels. Secondly, due to loss of hair regions, the entropy of RAF data in the leaf-level is quite high. As for CompCars, the adopted images in this paper all profile pose, accuracy of the compound of pose and car type prediction the leaf-level is about 30%, i.e. the poses of most cars (30/45) are correctly predicted. Apart from quantitative results, Fig 19 shows the semantic translation results across datasets. It is observed that information of gender and smile is correctly disentangled and transferred. For the translation be- tween CADCars and CompCars, given the unseen type Hatchback (fifth row), the translated result of the SUV looks like nothing on the earth. Besides, we also find difficult to transfer the images from unseen dataset shown in the last case, which mainly due the domain shift for the generator.
A.5 ABLATION STUDY
In this section, make a justification of several choices made in method. including usage of local 'brother' categories for classification learning, image/feature reconstruction losses in Eqn. and Eqn.(6). Specifically, replace the local classification loss with the global one in each level
18

---

Under review as a conference paper at ICLR 2020
w/o reconfea
w/o reconimg
w/o local
HDN-full
Real
Level 2
0,9600
0.9434
0.8246
0.9387
0.9570
Level 3
0.9020
0.9068
0.7791
0,9103
0,9232
Level 4 0.8661
0.8662
0.8587
0.8799
0.8932
Table 5: Accuracy of hierarchical classifications for real and generated images by baselines and our full method in different semantic levels on CelebA. w/o reconfea, reconimg, local denote HDN trained without the feature reconstruction, image reconstruction or local classification loss, respec- tively. Real means the result of real images in test set.
Source Target
F1
R2
R3
R4
XR)
w/o reconfea w/o reconimg w/o local
HDN-full
Figure 20: Semantic translation results of baselines and HDN-full CclebA. Different columns denote results of using Fi, (Ri or their combinations disentangled from the target images to replace the corresponding levels of the sources. Ground truths of R2, R3, R, gender, smile and hair color, respectively.
to verify the effectiveness of local discriminability for disentangling hierarchical features. For the roles of reconstruction loss, we simply drop during training.
Firstly, we compare baselines with our full method in terms of the classification performance on the generated images controlled by disentangled features. From Table5 we see that HDN- full overall performs better. Replacing the local classification loss with the global one in non-root levels would heavily do harm the goal of hierarchical disentanglement, all categories in that level into consideration which needs the discriminative information in both parent and current levels, while we aim to separate such information in different levels, leading to conflicting objectives. Such conflict can be found in the translated cases shown in Fig Without the local classification, only changing features of one level results in ambiguous generations (the third row), which can also be reflected in the quantitative evaluation of image quality in Table As for the reconstruction losses, they mainly stable the adversarial training. Without them, the quality of generated images would decrease in some extent. Besides, the feature reconstruction loss can boost the disentangling degree of features. As the 2D iSNE results in dedonssttsrssssss without such loss, the intra-class compactness and inter-class discriminability of samples in the embedding space become poor.
the global one takes
A.6 QUALITY COMPARISON OF TRANSLATED IMAGES
In this section, we evaluate the quality of translated images controlled by hierarchical features on test set of CelebA. Since the disentangling paradigm of our method is similar to the image-to-image translation task, we further compare one of such kinds of cGAN-based works, i.e. StarGAN (Choi) et al.] 2018) which has been popular framework for the multi- translation task. Besides, we also compare a disentanglement work named ELEGANT We follow the hyper-parameter settings on CelebA in their publicly released codes. Due to resources-cost, the StyleGAN is only trained
Thii method is good at disentangling two factors in model and can only change factor of varition given reference. The performance would become unstable for than two factors, which has been verified
19

---

Under review as a conference paper at ICLR 2020
16
(a) Root
(b) Level 2
(c) Level 3
(d) Level 4
Figure 21: 2D ISNE of disentangled F, by HDN without reconstruction feature loss on test set of CelebA for different levels. For easy understand, M and F mean male and female, and N mean Smile and Neural, B1, G and Br mean Black, Golden and Brown hair respectively.
wrl reconiec
w/o reconing
wio local
SarGAN
ELEGANT-2
ELEGANTS
Styl-GAN
HDD full
Real
2.75
2.42
3:34
2,59
2.54
3.63
2008
2.70
2,7
FTD
20.07
LPIPS
0,411
0.408
0,400
0.044
0.49
0
0416
Table 6: Comparisons of image quality of baselines and HDN. IS and FID measure the fidelity, and LPIPS measures the diversity of images. For IS and LPIPS, larger is better. For FID, lower is better.
for 128*128 resolution for 300 epochs. We use the Inception Score (IS) (Salimans et al. 2016) and Frehet Inception Distance (FID) (Heusel et al. 2017) to measure semantics of images, and leverage the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al.) 2018) to measure the diversity of generated visual modes. In Table it is observed HDN achieves comparable and even slightly better semantics compared with the state-of-the-art image-to-image translation method, which demonstrates that HDN can not only extract primitives of objects for discriminative tasks but also be applied to such graphical applications. Besides, we observed that IS and LPIPS sensitive to artifacts while FID is more stable, which can be found in qualitative results in the following.
Apart from these quantitative measurements, Fig 22 compares the translated results. Our method performs comparable with StarGAN and better than ELEGANT. ELEGANT is also for disentan- gling, the results of which for few attributes (no more than two as suggested by the authors) looks good but would become much poor when multiple factors need to be dealt with.
A.7 FAILURE CASE ANALYSIS
In this section, show results of HDN the challenging ImageNet dataset and analyze the limita- tions of current method on too complex dataset. We collect images from super categorics including house cats, dogs and big cats of ImageNet. Each super category contains 4 fine-grained categories, which thus constructs a three-level hierarchy (root is animal). Following (Huang et al.) 2018)), all images split by official train/test protocol are processed by a pre-trained faster-renn head detector, and then cropped and resized to 128* resolution as the inputs for HDN, Examples of such hi- erarchical data are shown in Fig 23 Network architecture and training hypee-parameters are same with those of CelebA, CADCars and ShapeNet introduced above.
We first quantitatively evaluate the disentangled features as we did in See 4.1 accuracy of test set in level 2 and 3 is 0.9293 and 0.8760, and that of generated images is 0,9493 and 0.8160, respectively. Fig 24 shows the tSNE embedding results using different levels of F,. From these results, we may infer that our method has successfully disentangled the desired semantic fea- tures in different levels which have progressively increased discriminability and good generalization ability for test and generated images. However, qualitative investigation reveals that it is not the truth of all. Fig[5]shows shows some semantic translation results of source objects using the disentangled hierarchical features of the targets. It is observed that the disentangled features can only change par- tial appearance (e.g. the textures or colors of objects) of the source images which leaves out other necessary and even the key information to recognize the objects in that level from the perspective of humans (e.g. the shape of the lion other than the fur color). Besides, purely change only one level features, the translated images would look strange.
The classification
by the authors of this work. We trained ELEGANT-22 for disentangling gender and smile, and E...EAATTT for all 5 attributes used.
20

---

Under review as a conference paper at ICLR 2020
Surrce
Target
R2
R3
R4 I(R)
ELEGANT -2
ELEGANT -5
StarGAN
HDN
Semantic translation results of compared cGANs and HDN CelebA. R2, R3, R₁ gender, smile and hair color, respectively. StarGAN only needs attribute conditions to generate images, and the target in its has for it. ELEGANT-2 is trained with only gender and smile attributes. The ELEGANT models can only change one attribute from the target each time.
Figure 22:
Egyptian
Persian
maaesse
Tabby
House Cat
Corgi
Shepherd
Husky
5 Samoyed
Animal
Dog
Cougar
Leopard
Tiger
Big Cat
Figure 23: Typical samples of hierarchical data on ImageNet. Images within a purple rectangular box are some instances of a leaf-level category. Categories within a green rectangular box belong to one common super-category. The super-categories within red rectangular box share one common ancestor.
15
15
-12
115
1117
(a) Root
(b) Level 2
(c) Level 3
Figure 24: 2D ISNE of disentangled F, on test set of ImageNet for different levels. H, and mean House cat, Dog and Big cat, E, Si, Ta, Cor, G, Hu, Sa, Cou, Le, Li and Ti mean Egyptian, Persian, Siamese, Tabby cat, Corgi, German shepherd, Husky, Samoyed, Cougar, Leopard, Lion, and Tiger respectively.
21

---

Under review as a conference paper at ICLR 2020
Source
Target
F
R₂
Rg
R2 R2+
Figure 25:
Semantic translation results of the source images controlled by hierarchically disentan- gled features of the targets on ImageNet.
Reasons for these phenomena are mainly in two folds. For one thing, there exists too much in- formation that can be leveraged for classification, since these ImageNet categories themselves are complex and the differences among them are in many aspects. Therefore, classifiers find "shortcuts" and extract only partial primitives constituting the objects in that level. Sometimes these "shorteuts" are even wrong ways, which is the so-called bias problem of ImageNet classifica- tion models (e.g. images containing black man are predicted as basketball) (Stock & Cissé [0177] Geirhos al., 2019). In the qualitative results of HDN on ImageNet, we also find that the semantics of disentangled features are not the whole to interpret the objects in that level and sometimes are even hard to understand by humans. This tells us that sometimes deep features can perform well in terms of certain measurements but may not work as our human expected, while our HDN can diagnose this kind of features as done in Fig For another, the poor image quality is partially owing to the capacity of GAN. Generating high-quality images ImageNet is the well-known tough problem which has not been well addressed by GANs until now due to the much complex data distribution. In framework, in order to disentangle semantics, it is required to synthesize some nonexistent categories combined by semantics from different categories, which further makes distribution fitting harder for discriminator. We believe that HDN could be improved ImageNet dataset by dis- entangling large scale of categories organized in a hierarchical structure, given a powerful enough generative framework.
easily
22


Reviews:
[{"review_id": "r1eVNoThcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for supervising networks for image classification and reconstruction with the object's hierarchical categories in mind. The claimed benefits are the improved generalizability and interpretability. The paper reports per-category-level analysis on the semantic image reconstruction task and retrieval on seen and unseen object categories.\n\nI am currently leaning towards weak accept because I find the paper's claim and technical details generally convincing and simultaneously extracting low-level and high-level features trained using the hierarchical levels of categories is novel. Generalization to unseen categories tends to be a good proxy for real world performance and directly learning the high level categories is a useful idea for doing so.\n\nAlthough I am leaning towards weak accept, I think this paper is close to borderline because the findings do not seem experimentally well validated. It would be more interesting to see Table 3 on multiple unseen categories instead of one special case per dataset. Another idea for experiments is doing cross-dataset evaluations where different datasets may have different leaf categories but shared high level ones. I think it may also be interesting to compare with a non-hierarchical retrieval model and then obtain their high-level prediction accuracy using the corresponding parent level categories.\n\nThe paper generally needs polishing. Minor typos I found:\n\nPage 5: classificaiton, classifers\nPage 6: intuitional->intuitive\nPage 7: an significant\nPage 8: leraning\nPage 1: human\nFigure 3: arbitary", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "S1e_7VtxqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of learning disentangled representation in a hierarchical manner. It proposed a hierarchical disentangle network (HDN) which tackles the disentangling process in a coarse-to-fine manner. Specifically, common representations are captured at root level and unique representations are learned at lower hierarchical level. The HDN is trained in a generative adversarial network (GAN) manner, with additional hierarchical classification loss enforcing the disentanglement. Experiments are conducted on CelebA (attributes), Fashion-MNIST (category), and CAD Cars (category & pose).\n\nOverall, this paper’s contribution seems quite outdated and presentation is not very clear.\n\n(1) Learning hierarchical representation using GAN has been explored in Kaneko et al. 2018 but not even mentioned in the paper. \n\nGenerative Adversarial Image Synthesis with Decision Tree Latent Controller. Kaneko et al. In CVPR 2018.\n\nAs far as reviewer understands, the bottomline for publication at ICLR is to demonstrate significant improvement/technical novelty compared to prior art. \n\nThis paper should also compare against GANs or other state-of-the-art generative models with flat representation (especially on face generation) in terms of SSIM, inception score, and FID score. Without such comparisons, it is unclear what is the value of hierarchical representation proposed here.\n\n-- Glow: Generative Flow with Invertible 1x1 Convolutions. Kingma and Dhariwal. In NeurIPS 2018.\n-- Progressive Growing of GANs for Improved Quality, Stability and Variation. Karras et al. In ICLR 2018.\n-- A Style-based Generator Architecture for Generative Adversarial Networks. Karras et al. In CVPR 2019.\n\n(2) The interpolation results (see Figure 5) look a bit strange as the transition between last columns are not very smooth. Also, please provide details about this experiment: are you applying linear interpolation? What’s the interpolation parameter for each of the column?\n\n(3) For image retrieval experiment, it is not clear if the proposed method is better than any state-of-the-art generative models with flat representations. One strong baseline is to use the latent representation of a pre-trained GAN model as comparison.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxXXnZ25S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed the hierarchical disentangle network (HDN) that leverages hierarchical characteristics of object categories to learn disentangled representation in multiple levels. Their coarse-to-fine manner approach allows each level to focus on learning specific representations in its granularity. This is achieved through supervised learning on each level where they train classifiers to distinguish each particular category from its ‘sibling’ categories which are close to each other. Experiments are conducted on four datasets to validate the method.  \n\nExploiting object hierarchy to learn disentangled representation is a promising direction but I lean towards rejecting this submission due to\n1. No results on commonly used disentanglement metrics (e.g. see [1])\n2. No comparison with existing supervised/unsupervised methods on disentangled representations (e.g. [2][3])\n3. The needs for full supervision on each level and manually designed fixed hierarchy require labels for the full hierarchy and make it not applicable to many existing data. This probably is why the proposed approach did not work well for more complex datasets like ImageNet.\n\n\nThese also should be addressed:\n1. The choice of adaptive instance normalization should be discussed. AdaIN could be used to account for small changes like color or local changes, but can it be used for larger and more global change (for example from animal category to human). If not, is it a limitation of this method?\n2. Justification for several choices made in the method, for example in the form of qualitative/quantitative ablation studies - usage of local ‘brother’ categories, image/feature reconstruction losses\n3. Can the metric in Table 1 prove disentanglement is achieved? What if E and G learned some way to fool the classifiers\n4. Authors use conditional generative adversarial networks but it seems that there is no noise.\n5. Discussion of failure cases. For example, the authors mentioned that the proposed approach did not work well for ImageNet. Why is this the case?\n\n\nMinor comments:\n- some citations are not properly formatted\n\n[1] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, Locatello et al.\n[2] Disentangled Sequential Autoencoder, Li and Mandt\n[3] Exploring Disentangled Feature Representation Beyond Face Identification, Liu et al.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HyeYaubg5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for solving the following problem: given an Image I from a labelled dataset with a label hierarchy as a tree of depth L, produce a set of vector representations {F_1, F_2 ... F_L}, such that a) the set can be used to reconstruct I as well as possible and b) each representation in the set only contains information about the label at the corresponding level in the label tree.\n\nWhile I am not extremely familiar with work in disentangled representation learning, the authors claim is true to my knowledge that most work on disentangling factors does not explicitly take into account hierarchical structure as this work does. Therefore, this work appears novel and interesting to me. I will leave the assessment of the degree of novelty to other reviewers/AC who may be more familiar with the literature.\n\nThe approach is also effective, and the authors demonstrate through visualizations and experiments that the proposed model can be trained and accomplishes its objectives reasonably well. My overall decision is to accept this paper, but there are some improvements I'd like to see since I found it difficult to understand in some places. \n\n- There is repeated use of the term \"granularity\" in the abstract and Sec. 1 which is undefined. What, according to the authors, is the difference between having a hierarchical structure and multi-granularity? I suggest clarifying what is meant by this, or avoid using the term (used hierarchy instead).\n\n- In Sec. 3.2, it appears that what is meant by R_l is actually the set {R_1, ..., R_L}. This would imply that the R's from different levels are randomly combined, and the number of representations combined is always L. Is this correct? In either case, what happens here should be made much clearer. It took me several readings to arrive at this interpretation.\n\n- It took me a while to infer how the results in Figure 3 are generated. There is a sudden switch in Sec. 4.1 from model training details to its use for semantic translation, which was not explained.\n\nMinor suggestions:\n\n- Please use parenthetical citations throughout the paper where appropriate (use \\citep{}) to avoid breaking the flow of reading.\n- Pg. 1, last line: \"us human\" -> \"humans\"\n- Pg. 2, line 1: \"with others\" -> \"to others\"\n- Pg. 2, line 2: \"hierarchy structure\" -> \"hierarchical structure\"\n- Pg. 2: \"multi-granularity nature\" -> \"multi-granular nature\" or \"hierarchical nature\"\n- Pg. 7, line 2: \"an significant\" -> \"a significant\"\n- Pg. 7, line 4: \"At the last\" -> \"At the end\"\n- Pg. 7, \"it can be reached\" -> \"it can be seen that\"\n- Pg. 7: \"Table.4.1 gives the evaluation results\" -? \"Table 1 gives ...\"\n- Pg. 7, Please revise: \"which is deserved to make more efforts\"\n- Pg. 7: \"to applied\" -> \"to be applied\"\n- Pg. 8, line 1: Remove \"quite\"; typo in \"leraning\"", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new method for learning hierarchically disentangled representations. One reviewer is positive, one is between weak accept and borderline and two reviewers recommend rejection, and keep their assessment after rebuttal and a discussion. The main criticism is the lack of disentanglement metrics and comparisons. After reading the paper and the discussion, the AC tends to agree with the negative reviewers. Authors are encouraged to strengthen their work and resubmit to a future venue.
Decision: Reject