Title: HIERARCHICAL DISENTANGLE NETWORK FOR OB-JECT REPRESENTATION LEARNING

Abstract: An object can be described as the combination of primary visual attributes. Disentangling such underlying primitives is the long-term objective of representation learning. It is observed that categories have the natural hierarchical characteristics, i.e. any two objects can share some common primitives in a particular category level while they may possess their unique ones in another. However, previous works usually operate in a flat manner (i.e. in a particular level) to disentangle the representations of objects. Though they may obtain the primitives to constitute objects as the categories in that level, their results are obviously not efficient and complete. In this paper, we propose the hierarchical disentangle network (HDN) to exploit the rich hierarchical characteristics among categories to divide the disentangling process in a coarse-to-fine manner, such that each level only focuses on learning the specific representations and finally the common and unique representations in all levels jointly constitute the raw object. Specifically, HDN is designed based on an encoder-decoder architecture. To simultaneously ensure the disentanglement and interpretability of the encoded representations, a novel hierarchical generative adversarial network (GAN) is elaborately designed. Quantitative and qualitative evaluations on four object datasets validate the effectiveness of our method.

1. INTRODUCTION:

Representation learning, as one basic and hot topic in machine learning and computer vision community, has achieved significant progress in recent years on different tasks such as recognition (Russakovsky et al., 2015) , detection (Ren et al., 2015; Redmon et al., 2016; Liu et al., 2016b) and generation (Goodfellow et al., 2014) , benefiting from the rapid development of representation learned by deep neural networks. Considering the strong capacity of deep representation, in this paper, we mainly focus on the deep representation learning framework.

Despite great success the deep representations have achieved as mentioned above, two important problems are still unresolved or less considered, i.e. the interpretability and the disentanglement of the learned representations. In the past decades, various works have been developed to reveal the black box of deep learning (Zeiler & Fergus, 2014; Dosovitskiy & Brox, 2016b; Bau et al., 2017; Simonyan et al., 2013; Stock & Cissé, 2017; Zhang et al., 2017) and move us closer to the goal of disentangling the variations within data (Reed et al., 2014; Mathieu et al., 2016; Rifai et al., 2012; Tran et al., 2017; Gonzalez-Garcia et al., 2018; Huang et al., 2018; Chen et al., 2016) . Even though they have brought great insights to us, they still have some limitations. For instance, (Chen et al., 2016; Xie et al., 2017; Zhao et al., 2017) learn to disentangle variation factors within each category using generative models, instead of investigating the similarities and differences among categories, leading to poor discriminability. Therefore, the learned representations would not well conform to human perception. Though (Gonzalez-Garcia et al., 2018; Huang et al., 2018) try to obtain the domain-invariant and domain-specific knowledge, they can only handle two categories one time, which is not that efficient. In this paper, we attempt to learn disentangled representations in a more natural and efficient manner.

Let us first discuss how humans understand an object. Generally speaking, an object can be regarded as the combination of many semantic attributes. Hundreds of thousands of objects in the world can be clustered and recognized by humans just because we can figure out the common and unique Figure 1 : An illustration of a hierarchical structure (a) and extracting the hierarchical features that constitute a leaf-level image (b). In (b), the common features that only contain the information of its being the root category are first extracted. By tracing from the root to leaf, the unique features that contain additional information of its being the finer-grained category are further extracted.

attributes of an object compared to others. Besides, a man who never play the billiards can only recognize a table in an image, while a sports fan may regard it as a billiard table. Both of them are right since categories have natural hierarchical structure. As shown in Fig. 1 (a), given six leaf-level categories, they can be organized in a three-level hierarchy considering the common and different features they have. Each child category in the hierarchy is a special case of its parent category since it inherits all features from its parent category and has extra features that are not present in its parent category. From another perspective, each parent category is the abstraction of all its child categories considering it contains the attributes that are present in all its child categories. Then we come back to the task of disentangling representation learning. It aims to learn the representation encoding useful information that can be applied in other tasks (e.g. building classifiers and predictors) (Bengio et al., 2013) . Taking the hierarchical nature of categories into account, if we only learn the representations of an object in a flat manner for a specific category level as previous works do, it will not be scalable and comprehensive for the machine to be qualified for various tasks in the real world.

Our work aims to exploit the natural hierarchical characteristics among categories to divide the representation learning in a coarse-to-fine manner, such that each level only focuses on learning the specific representations. For instance, given a billiard table image in Fig. 1 (b), it tangles the information of being a furniture, a table and a billiard table. We first extract the features that only contain the information of furniture from the image. By tracing from the root to leaf level, more and more information is extracted until we can recognize its belonging categories in all hierarchical levels. By doing so, the disentangled representations are expected to find wide and promising applications. For example, one can transfer the semantics in a specific category level from one object to another while keep information of other levels unchanged. Besides, it would help for the hierarchical image compression task using different levels of the disentangled representations. To achieve the objective of hierarchical disentangling and simultaneously interpreting the results so that humans can understand, we propose the hierarchical disentangle network (HDN), which draws lessons from hierarchical classification and the recent proposed generative adversarial nets (Goodfellow et al., 2014) . Extensive experiments are conducted on four popular object datasets to validate the effectiveness of our method.

2. RELATED WORKS:

Disentangling Deep Representations. The goal of disentangling representation learning is to discover factors of variation within data (Bengio et al., 2013) . Recent years have witnessed a substantial interest on such research area (Tenenbaum & Freeman, 1996) , including works based on deep learning (Reed et al., 2014; Mathieu et al., 2016; Rifai et al., 2012; Wang et al., 2017; Tran et al., 2017; Gonzalez-Garcia et al., 2018; Huang et al., 2018; Chen et al., 2016) . (Rifai et al., 2012) is probably the earliest to learn disentangled representations using deep networks for the task of emotion recognition. (Reed et al., 2014) is based on a higher-order Boltzmann machine and regards each factor variation of the manifold as its sub-manifold. (Mathieu et al., 2016) and (Chen et al., 2016) leverage the generative adversarial nets (GAN) to learn factors of variation. Recently, cross-domain translation methods (Gonzalez-Garcia et al., 2018; Huang et al., 2018) learn the domain-invariant and domain-specific representations. These works ignore the existing natural and inherent hierarchy relationships among categories, with which we can conduct the disentangling in a coarse-to-fine manner such that each level only focuses on learning the specific representations. Given images belonging to a hierarchy, the common feature maps of their being the root category and the unique features in different nonroot levels that can further distinguish them as finer-grained categories are extracted by the upper residual and bottom 1*1 convolutional branches, respectively. The unique features are transformed as the parameters of Adaptive Instance Normalization (AdaIN) and thus be aggregated into the common feature maps to obtain comprehensive representations of images. To ensure disentanglement of the unique features, different levels of them are randomly combined and then reconstructed in the image space where adversarial loss and hierarchical classification loss are elaborately designed.

Network Interpretability. Network interpretability aims to learn how the network works via visualizing it from the perspective that humans can understand. These methods can be briefly divided into two groups according to whether the visualization is involved in the network during training, i.e. the off-line methods and online methods. The off-line methods make attempts to visualize patterns in image space that activate each convolutional filter (Zeiler & Fergus, 2014; Dosovitskiy & Brox, 2016b; a; Bau et al., 2017; 2019) or to interpret the area in an image that is responsible for the network prediction (Simonyan et al., 2013; Fong & Vedaldi, 2017; Zintgraf et al., 2017; Abbasi-Asl & Yu, 2017; Stock & Cissé, 2017; Palacio et al., 2018; Geirhos et al., 2019) . While such methods can explain what has already been learned by the model, they cannot improve the model interpretability in return. Instead, the online works propose to directly learn interpretable representations during training (Li et al., 2018; Zhang et al., 2017) . However, these methods mainly focus on figuring out the running mechanism of networks while paying less attention to dissecting variations of the features among categories, which cannot make models really understand their inputs.

Hierarchy-regularized Learning. Semantic hierarchies have been explored in object classification task for accelerating recognition (Griffin & Perona, 2008; Marszalek & Schmid, 2008) , obtaining a sequence of predictions (Deng et al., 2012; Ordonez et al., 2013) , making use of category relation graphs (Deng et al., 2014; Ding et al., 2015) , and improving recognition performance as additional supervision (Zhao et al., 2011; Srivastava & Salakhutdinov, 2013; Hwang & Sigal, 2014; Yan et al., 2015; Goo et al., 2016; Ahmed et al., 2016) . While these discriminative classification works have achieved their expected goals, they usually lack interpretability. To address such issues, (Xie et al., 2017; Zhao et al., 2017) propose to use generative models to disentangle the factors from low-level representations to high-level ones that can construct a specific object. (Singh et al., 2019) uses an unsupervised generative framework to hierarchically disentangle the background, object shape and appearance from an image. However, they either deal with each category in isolation or ignore the discriminability of learned features, and thus cannot accurately disentangle the differences and similarities among categories.

3. HIERARCHICAL REPRESENTATION LEARNING:

Supposing that a category hierarchy is given in the form shown in Fig. 1 (a), we use l = 1, ..., L to denote the level of hierarchy (L for the leaf level and 1 for the root level), K l to denote the number of nodes at level l, n k l to denote the k-th node at level l, and C k l to denote the number of children for n k l . As illustrated in Fig. 1 (b), given an original object image denoted as I o , our goal is to extract the feature F l in the l-th level.

Generally speaking, an object O can be described as the combination of a group of visual attributes:

EQUATION

where ∆(O) represents currently undefined attributes existing on O. As we have discussed, humans classify O in a particular category level according to a subset of the whole attribute set in Eqn.(1). Take the object in Fig. 1 Taking these into consideration, we design the hierarchical disentangle network (HDN) based on the autoencoder architecture in Fig. 2 . The encoder E dissects the hierarchical representations given a semantic hierarchy. The decoder G plays the role of an interpreter to reflect the variations of semantic in the image space for different hierarchical levels guided by the hierarchical discriminator D adv and classifiers D cls (they share most network architecture except the output layers).

3.1. TOP-DOWN LEARNING OF HIERARCHICAL REPRESENTATIONS:

Since F l-1 is the proper subset of F l , once F l-1 is obtained, only the difference R l (1 < l ≤ L) between F l and F l-1 needs to be encoded. Considering these, we devise a top-down representation extraction scheme.

Given F l-1 and R l , we aggregate them together to obtain the whole representation in the l-th level. Such procedure can be formulated as:

F l = F l-1 ⊕ R l

(2) where ⊕ means information aggregation. In summary, for hierarchical disentanglement, the common feature F 1 in the root level and the unique ones {R l } L l=2 in deeper levels need to be encoded. To further interpret the semantics of these features to humans, the decoder reconstructs them in the image space. The semantics of F 1 are shared among all its offspring which can be regarded as the invariant content of the object, while those of {R l } L l=2 are unique for different levels which plays the role of the variant style of the object. Therefore, F 1 and {R l } L l=2 are processed in the upper and bottom branches respectively to make them play different roles during the reconstruction, as shown in Fig. 2 .

3.2. CONSTRAINTS FOR THE LEARNING PROCESS:

The basic constraints of hierarchical disentanglement are making features in different levels perform their own duties. For an object O, the encoded F 1 and {R l } L l=2 are complementary, as the constraints of F l being the proper subset of F l+1 . F 1 should encode just right information to describes its being the root category. Progressively using R l , one can distinguish it from other categories in l-th level.

Apart from disentanglement, visualization of features in the image space is also one of our objectives. We turn to the popular conditional generative adversarial nets (cGANs) (Mirza & Osindero, 2014) which can control generated images given conditions. Our HDN leverages the disentangled features F 1 and {R l } L l=2 to control the variations of reconstructed images in different category levels. To ensure F 1 , {R l } L l=2 be well disentangled, we propose a random combination strategy for different levels of features from different objects and control the generated images through these combined features, as shown in Fig. 2 . Specifically, given

F 1 1 , {R 1 l } L l=2 and F 2 1 , {R 2 l } L l=2

from arbitrary two objects, we obtain the newly combined features F 1 and {R l } L l=2 , where ∀1 ≤ l ≤ L, R l (F 1 if l = 1) come from either the first or second object. The newly combined features are aggregated together as the input for the decoder G to generate a new object image I g . Such image should satisfy the following losses:

-Hierarchical classification loss. For each level, I g should be classified to the category that R i l reflects (root level F 1 only contains one category), defined as:

EQUATION

where J cls is cross-entropy loss among local categories in each level which have a common parent node k such as the dashed rectangled categories in the bottom right corner of Fig.

2. p(G) denotes distribution of generated images G(F i 1 , {R i l } L l=2 ). D cls (I g ) c

l is probabilistic prediction on the c-th local category, and y c l is the ground truth local label of the generated object in the l-th level. Please note that we only focus on the local brother categories instead of all categories in that level. It makes the disentanglement more flexible. On one hand, the classification in each level can thus only focus on the unique features that are just discriminative among those local brother categories. On the other hand, the duties of different levels can be well disentangled, since if the semantic information encoded in different levels is tangled, after the random combination and image reconstruction, the hierarchical classifiers would be quite confused.

-Adversarial loss. We employ GANs to match the distribution of reconstructed images to the real data distribution. Specifically, the LS-GAN (Mao et al., 2017) loss is adopted in light of its stable training, defined as:

EQUATION

-Image reconstruction loss. As for F 1 and {R 1 l } L l=2 from one same object, we should be able to reconstruct it as close to the input as possible.

EQUATION

where p (G) is the distribution of generations taking F 1 , {R l } L l=2 from the same objects as inputs. -Feature reconstruction loss. Apart from the image reconstruction loss, the feature reconstruction loss is added to HDN to stabilize the training process.

EQUATION

where p(E) is the distribution of encoded hierarchical features E(I o ).

Now we combine the four loss functions defined in Eqn.

(3), Eqn.(4), Eqn.(5) and Eqn.(6) into one comprehensive loss function for supervising the disentangling of E and visualization of G:

J(E, G) = J cls + J GAN + αJ I recon + βJ F,R recon (7)

where α and β are the hyper-parameters to balance the weights of the four terms.

As for the update of discriminator and hierarchical classifiers, we use the following loss:

EQUATION

3.3. RELATIONSHIP WITH PREVIOUS WORK:

It is noted that the recent work DTLC-GAN (Kaneko et al., 2018) has similarities with our method on motivations of learning hierarchical representations. Nevertheless, DTLC-GAN is indeed different from ours. Specifically, the detailed goals of leveraging hierarchical relationship are different. DTLC-GAN aims to maximize the mutual information between conditioned representation and data in each level, i.e. study how the appearance of data varies with more and more specific conditions and thus synthesize data with more fine-grained details. Our method focuses more on how humans distinguish objects from categories in different hierarchical levels and wishes such manner of understanding objects can be applied to the machine, i.e. learn the commonality and individuality of categories in nature. Therefore, the disentangled features of our method are mainly served for downstream discriminative tasks such as semantic retrieval, open world unseen category recognition as we have attempted in the following experiments. Besides, thanks to the disentangled commonality, our method can further realize the semantic translations between images by exchanging the individual parts, which has been a popular application in real world. 

4. EXPERIMENTS:

Datasets: We conduct experiments on hierarchical annotated data from four datasets, typical examples in the hierarchy are shown in Fig. 8 , Fig. 9 and Fig. 10 in Appendix 1 . The first is CelebA dataset (Liu et al., 2015) . It provides more than 200K face images with 40 attribute annotations. Following the official train/test protocol, we define a four-level hierarchical structure which has explicit attribute difference between any two levels. Specifically, all faces (root category) are first divided into two categories based on gender. Such initial categories are further classified according to the smile expression and hair color in the next two levels. With such ground-truth hierarchical annotations, we can validate our method more easily.

The second dataset named Fashion-MNIST (Xiao et al., 2017) is proposed as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same train/test split with MNIST. Since such dataset does not provide any hierarchical structure, we cluster T-shirt, coat, pullover as one super category and trouser, dress as another super one to construct a three-level hierarchy (root is fashion) according to their appearance similarities.

The other two datasets are 3D data, CADCars (Fidler et al., 2012) and ShapeNet (Chang et al., 2015) . CADCars contains 183 3D Car models and ShapeNet is constitutive of 51,300 3D models covering 55 common and 205 finer-grained categories. Using the provided tools, we generated 24 2D images with 6 pose and 4 illumination variations for CADCars. These 2D data are clustered into four super categories, i.e. minibus, sedan, sports and SUV, and are further divided into 6 finer-grained categories for each super one based on pose annotations, which defines a three-level hierarchy. On ShapeNet, 12 2D images with pose variation are obtained for each 3D model. One three-level category-pose hierarchy named as ShapeNet-P similar to CADCars and one three-level hierarchy named as ShapeNet-C as in Fig. 1 .(a) are defined. Ratio of train/test split is 4:1.

Implementation Details: Our HDN is implemented with Pytorch platform 2 . Design of the backbone follows recent proposed image generation (Karras et al., 2018) and translation works (Huang et al., 2018) . Images are resized to 128*128 resolution for all datasets except Fashion-MNIST which is resized to 28*28. As shown in Fig. 2 , to match it with our task, we increase the number of 1*1 convolution branches such that originally one representation is disentangled into multiple hierarchical levels. We also equip the residual blocks with Adaptive Instance Normalization (AdaIN) whose parameters are dynamically generated by a multi-layer perception (MLP) from the disentangled latent codes. Besides, D has L output branches, one for real/fake predictions and the others for hierarchical classifications. More training details are given in the Appendix. 

4.1. DISENTANGLED RESULTS:

Firstly, we replace one or several levels of disentangled features for an image with those of another image, and then observe the visual changes of generated image to validate the semantic consistence with pre-defined hierarchical structure. Fig. 3 (and Fig. 12 , Fig. 13 in the Appendix) shows such semantic translation results. It is observed that different level of features perform their own duties, i.e. they carry just enough information to control the variations within that level (e.g. gender, smile and hair color for CelebA we specially predefined on CelebA), but would not involve more belongs to other level. For instance, in Fig. 3 change features of an image in arbitrary one, two or all levels to those of another image, the semantics would be changed correspondingly. Apart from {R l } L l=2 , the common feature F 1 also encodes information that is not discriminative among its offspring categories but is necessary to construct the object (e.g. the identity, pose and even the background information of a face image). To give a more intuitive feeling about the ability of disentangled features, we investigate the discriminabilites of them via the popular tSNE tool (Maaten & Hinton, 2008) . As shown in Fig.4 (and Fig.14, Fig.15, Fig.16, Fig.17 in the Appendix), with only the common feature F 1 , samples are mixed together. When progressively be combined with features of deeper levels R l , samples are better separated and almost consistent with the hierarchical structure.

Apart from direct visual edit, we also show that one can transform the source image smoothly by linear interpolation (with 5 equally spaced interpolation coefficients from 0.1 to 0.9) of disentangled features between the source and target. Such examples are shown in Fig. 5 . We can see that genders, expressions, hair colors and their combinations of the source images (first columns in each case) can be changed smoothly towards those of the targets (last columns of each case). Learning a smooth feature space with continuous variations is a significant issue for representation learning task, which can ensure the generalization ability for unseen similar objects. We have investigated this in Sec.4.3. At the end of this section, a quantitative evaluation of these results is conducted.

1 It is noted that the focus of this paper is to interpret the hierarchical structure within data. Therefore, we heuristically construct hierarchical structures based on human priors. One can also automatically obtain reasonable hierarchical annotations using machine learning technologies such as unsupervised clustering as (Goo et al., 2016) does. 2 The source codes will be released to the public. To be specific, we use the learned hierarchical classifier D to evaluate whether the semantics are correctly disentangled and decoded into the randomly translated images as in Fig. 3 . To ensure D is reliable, the accuracy of hierarchical classifications on the test data is given as a reference.

CelebA:

Table .1 gives the evaluation results. Firstly, it can be seen that the semantics of translated images with changing different levels are recognized correctly by the corresponding classifiers. Secondly, the deeper of the level, the more difficult of the translation, since the criteria for distinguish one category from others in the deeper level would become more and more complicated (summation of all criteria above this level). Finally, it becomes difficult to transfer the unique features and generate images when that information is difficult to be described and disentangled such as in the leaf-level of Fashion-MNIST and ShapeNet-C, which deserves to make more efforts.

4.2. APPLICATION TO IMAGE RETRIEVAL:

One of the objectives of learned representations is to be applied in real-world applications. Semantic image retrieval has been studied for years. Hashing is one effective and space-time efficient solution for this task. However, the semantic of target images that users expect are not always consistent just due to the tangled information of objects in different hierarchical levels. In this section, we conduct retrieval in different levels on CelebA. We compare three competing deep hashing methods, i.e. DSH (Liu et al., 2016a) , HashNet (Cao et al., 2017) and SSDH (Yang et al., 2018) , and two strong pre-trained GAN baselines, i.e. supervised StarGAN (Choi et al., 2018) and unsupervised StyleGAN Karras et al. (2018) . The backbones of hashing methods are same with the bottom branch of encoder E of HDN and pretrained on CASIA WebFace dataset (Yi et al., 2014) . In l-th level, a model with bit-length as same as the dimension of the concatenation of {R l } l 2 is trained. As for StarGAN and StyleGAN, the latent features before the last layer of discriminator are used as the representations of samples. To make a fair comparison with hashing methods, we also binarize the disentangled features via Sigmoid activation fucntion, which we named as HDN-B. Images of the test set are used to retrieve the training set.

Table .2 gives the mAP results in different semantic levels. First, our method achieves the best performance, though we did not impose specific metric objectives on features while the flat StarGAN and StyleGAN perform not well. Second, HDN is more efficient since it only needs one model owing to disentanglement while hashing methods have to train a model in each level. We also tried to use only one model trained in the leaf-level to evaluate in high levels (methods with postfix of "-S"), but the results are inferior to those independently trained for each level. Third, HDN-B is better than HDN, which mainly due to the increased non-linear ability of features. Finally, the retrieval of HDN is more interpretable. As shown in Fig. 6 , with different parts of features, the returned images satisfy different semantic requirement, while for general method like SSDH one can not interpret the meanings of different code parts. 2). In the leaf-level, unseen objects have the unique unseen features, leading to the prediction entropy increase obviously compared with that of seen objects. Besides, it is found that the unseen objects are more likely classified to similar seen categories in leaf-level. For instance, about 30% and 56% bald faces are recognized as black and golden hair respectively, 50% and 50% leather couches are predicted as loveseat and L-couch respectively, 44% and 50% of the frontal sofa/table are classified as the right 30 • offset of frontal and left 30 • offset of frontal. The semantic translations in Fig. 7 between seen and unseen images also verify such results. The semantics of non-leaf levels can be transferred as usual, but the unseen unique features are not. Bald may be disentangled as golden or black hair due to the skin color. The material of leather is ignored in ShapeNet-C since model focus more on shape to distinguish seen objects rather than material during training. The translations to frontal pose are also confused as can be found in the cases of ShapeNet-P. Through this study, we think that disentangling the visual primitives of objects as learned knowledge is one of the most promising solutions for the ability of open-world recognition.

HDN SSDH Query:



5. CONCLUSIONS:

We propose the hierarchical disentangle network (HDN) which exploits the natural characteristics among categories to divide the representation learning in a coarse-to-fine manner. Our model achieves promising disentangle results. We also show the applications of such disentangled features on semantic translation, retrieval and even unseen objects prediction. However, our work is just an early step towards the long goal of disentangled representation learning, limited by the capacity of generative models on large scale and heavily tangled categories, the performance of HDN is not quite well such as on the ImageNet dataset which deserves to make more efforts. 

A APPENDIX A.1 NETWORK ARCHITECTURES AND TRAINING DETAILS:

A.1.1 NETWORK ARCHITECTURES Following the backbone designs in (Huang et al., 2018) for image-to-image translation task, let c7s1-k denotes a 7 × 7 convolution block with k filters and stride 1. dk means a 4 × 4 convolution block with k filters and stride 2. Rk denotes a residual block that contains two 3 × 3 convolution blocks with k filters. The last layers for different hierarchy levels (common features of root level are encoded by the content encoder) in the style encoder include multiple c1s1-8 branches, i.e.

1 × 1 convolution block with 8 filters and stride 1. uk denotes a 2× nearest-neighbor upsampling layer followed by a 5 × 5 convolution block with k filters and stride 1. GAP denotes a global average pooling layer. Instance Normalization (IN) and Adaptive Instance Normalization (AdaIN) are adopted to the content encoder branch and decoder respectively. No normalization is used in the style encoder branch. Use ReLU activations in the encoder-decoder and Leaky ReLU with slope 0.2 in the discriminator and classifier. Multi-scale discriminators with 3 scales (single scale for Fashion-MNIST due to its too small resolutions) are used to ensure both realistic details and global structure.

The last layer of the decoder is equipped with a tanh activations to normalize the values of generated images to the range of [-1, 1]. In the following, we give the detailed architectures of each module on different datasets.

CelebA, CADCars & ShapeNet::

Content encoder: c7s1-64, d128, d256, R256, R256, R256

Style encoder: c7s1-64, d128, d256, d256, d256, GAP, c1s1-8

Decoder: R256, R256, R256, u128, u64, c7s1-3

Discriminator & Classifier: d64, d128, d256, d512

Fashion-MNIST:

Content encoder: c7s1-32, d64, d128, R128, R128, R128

Style encoder: c7s1-32, d64, d128, R128, R128, R128, GAP, c1s1-8

Decoder: R128, R128, R128, u64, u,32 c7s1-1

Discriminator & Classifier: d32, d64, d128, d256

A.1.2 TRAINING HYPERPARAMETERS:

We use the Adam optimizer with β 1 = 0.5, β 2 = 0.999, and initial learning rate of 0.0001. We train HDN on all datasets for 300K iterations and half decay the learning rate every 100K iterations. We set batch size to 16. The loss weights α and β in Eqn.( 7) are set as 10 and 1 respectively. Random mirroring is applied during training. 

A.3 DISENTANGLE RESULTS ON MORE DATASETS:

In this section, we give disentangled results on CADCars, Fashion-MNIST and ShapeNet. Fig. 12 and Fig. 13 show the semantics of disentangled features which can well change the variations of generated images. Similarly, Fig. 14 , Fig. 15 , Fig. 16 and Fig. 17 show that progressively involving more features of deeper levels, samples become better separated, which verifies the results of disentanglement are consistent with the semantic hierarchical structures.

A.4 CROSS DATASET EVALUATIONS:

Learning general representation that can be applied across datasets is one of the long goals for machine. In this section, we briefly evaluate our method on datasets which have similar categories but quite different styles. To be specific, we evaluate HDN on a challenging facial expression dataset called RAF (Li et al., 2017) and a cars dataset named CompCars (Yang et al., 2015) , using models trained on CelebA and CADCars respectively. RAF is a large-scale facial expression database with around 30K great-diverse facial images downloaded from the Internet. It provides expressions, race, age range and gender attributes annotations. Besides, the released images are compactly aligned which have little information about hair colors. Therefore, the leaf-level categories are obtained according to the race or age range. As for CompCars, it contains 163 car makes with 1,716 car models. Besides, it also labels the car type (i.e. SUV, Sedan, Sports, etc.). Based on these annotations, we Firstly, we compare baselines with our full method in terms of the classification performance on the generated images controlled by disentangled features. From Table5, we can see that HDNfull overall performs better. Replacing the local classification loss with the global one in non-root levels would heavily do harm to the goal of hierarchical disentanglement, as the global one takes all categories in that level into consideration which needs the discriminative information in both parent and current levels, while we aim to separate such information in different levels, leading to conflicting objectives. Such conflict can be found in the translated cases shown in Fig. 20 . Without the local classification, only changing features of one level results in ambiguous generations (the third row), which can also be reflected in the quantitative evaluation of image quality in Table6. As for the reconstruction losses, they mainly stable the adversarial training. Without them, the quality of generated images would decrease in some extent. Besides, the feature reconstruction loss can boost the disentangling degree of features. As the 2D tSNE results in Fig. 21 demonstrates, without such loss, the intra-class compactness and inter-class discriminability of samples in the embedding space become poor.

A.6 QUALITY COMPARISON OF TRANSLATED IMAGES:

In this section, we evaluate the quality of translated images controlled by hierarchical features on test set of CelebA . Since the disentangling paradigm of our method is similar to the image-to-image translation task, we further compare one of such kinds of cGAN-based works, i.e. StarGAN (Choi et al., 2018) which has been a popular framework for the multi-attribute translation task. Besides, we also compare a disentanglement work named ELEGANT3 . We follow the hyper-parameters settings on CelebA in their publicly released codes. Due to resources-cost, the StyleGAN is only trained for 128*128 resolution for 300 epochs. We use the Inception Score (IS) (Salimans et al., 2016) and Frchet Inception Distance (FID) (Heusel et al., 2017) to measure semantics of images, and leverage the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) to measure the diversity of generated visual modes. In Table6, it is observed HDN achieves comparable and even slightly better semantics compared with the state-of-the-art image-to-image translation method, which demonstrates that HDN can not only extract primitives of objects for discriminative tasks but also be applied to such graphical applications. Besides, we observed that IS and LPIPS are sensitive to artifacts while FID is more stable, which can be found in qualitative results in the following.

Apart from these quantitative measurements, Fig. 22 compares the translated results. Our method performs comparable with StarGAN and better than ELEGANT. ELEGANT is also for disentangling, the results of which for few attributes (no more than two as suggested by the authors) looks good but would become much poor when multiple factors need to be dealt with.

A.7 FAILURE CASE ANALYSIS:

In this section, we show results of HDN on the challenging ImageNet dataset and analyze the limitations of current method on too complex dataset. We collect images from 3 super categories including house cats, dogs and big cats of ImageNet. Each super category contains 4 fine-grained categories, which thus constructs in a three-level hierarchy (root is animal). Following (Huang et al., 2018) We first quantitatively evaluate the disentangled features as we did in Sec.4.1. The classification accuracy of test set in level 2 and 3 is 0.9293 and 0.8760, and that of generated images is 0.9493 and 0.8160, respectively. Fig. 24 shows the tSNE embedding results using different levels of F l . From these results, we may infer that our method has successfully disentangled the desired semantic features in different levels which have progressively increased discriminability and good generalization ability for test and generated images. However, qualitative investigation reveals that it is not the truth of all. Fig. 25 shows some semantic translation results of source objects using the disentangled hierarchical features of the targets. It is observed that the disentangled features can only change partial appearance (e.g. the textures or colors of objects) of the source images which leaves out other necessary and even the key information to recognize the objects in that level from the perspective of humans (e.g. the shape of the lion other than the fur color). Besides, purely change only one level features, the translated images would look strange.

by the authors of this work. We trained ELEGANT-2 for disentangling gender and smile, and ELEGANT-5 for all 5 attributes we used. Reasons for these phenomena are mainly in two folds. For one thing, there exists too much information that can be leveraged for classification, since these ImageNet categories themselves are too complex and the differences among them are in many aspects. Therefore, classifiers can easily find "shortcuts" and extract only partial primitives constituting the objects in that level. Sometimes these "shortcuts" are even wrong ways, which is the so-called bias problem of ImageNet classification models (e.g. images containing black man are predicted as basketball) (Stock & Cissé, 2017; Geirhos et al., 2019) . In the qualitative results of HDN on ImageNet, we also find that the semantics of disentangled features are not the whole to interpret the objects in that level and sometimes are even hard to understand by humans. This tells us that sometimes deep features can perform well in terms of certain measurements but may not work as our human expected, while our HDN can diagnose this kind of features as done in Fig. 25 . For another, the poor image quality is partially owing to the capacity of GAN. Generating high-quality images on ImageNet is the well-known tough problem which has not been well addressed by GANs until now due to the much complex data distribution.

In our framework, in order to disentangle semantics, it is required to synthesize some nonexistent categories combined by semantics from different categories, which further makes distribution fitting harder for discriminator. We believe that HDN could be improved on ImageNet dataset by disentangling large scale of categories organized in a hierarchical structure, given a powerful enough generative framework.

:

This method is good at disentangling two factors in a model and can only change one factor of varition given a reference. The performance would become unstable for more than two factors, which has been verified

Figures and Tables:

Figure2: An illustration of the framework of our method. Given images belonging to a hierarchy, the common feature maps of their being the root category and the unique features in different nonroot levels that can further distinguish them as finer-grained categories are extracted by the upper residual and bottom 1*1 convolutional branches, respectively. The unique features are transformed as the parameters of Adaptive Instance Normalization (AdaIN) and thus be aggregated into the common feature maps to obtain comprehensive representations of images. To ensure disentanglement of the unique features, different levels of them are randomly combined and then reconstructed in the image space where adversarial loss and hierarchical classification loss are elaborately designed.

Figure 3: Semantic translation results of the source images controlled by hierarchically disentangled features of the targets on CelebA. Different columns denote results of using F 1 , {R l } L l=2 or their combinations disentangled from the target images to replace the corresponding levels of the sources. Ground truths of R 2 , R 3 , R 4 are gender, smile and hair color, respectively.

Figure 4: 2D tSNE of disentangled F l on test set of CelebA for different levels. For easy understand, M and F mean male and female, S and N mean Smile and Neural, Bl, G and Br mean Black, Golden and Brown hair respectively.

Figure 5: Interpolations of disentangled R l between the source (first columns) and target (last columns) images (other levels unchanged for the source images).

Figure 6: Top-5 returned images of a retrieval case using different parts of features (i.e. different R l of HDN, and different bit parts of SSDH). Green and red boxes are correct and false samples judged by the hierarchy.

Figure 7: Semantic translation results between seen and unseen objects. Here we replace the source images with all levels of R l of the targets, i.e. the right most case in Fig.3.

Figure 10: Typical samples of hierarchical data on CADCars. Images within a purple rectangular box are some instances of a leaf-level category. Categories within a green rectangular box belong to one common super-category. The super-categories within a red rectangular box share one common ancestor.

Figure11: Typical samples of unseen leaf-level data on CelebA (bald and gray hair), ShapeNet-C (kinds of tables and sofas) and ShapeNet-P (more poses). Large differences with seen hierarchical data can be found here.

Figure 14: 2D tSNE of disentangled F l on test set of Fashion-MNIST for different levels. T and B mean Top and Bottom, Ts, P, C, Tr and D mean T-shirt, Polo shirt, Coat, Trousers and Dresses respectively.

Figure 15: 2D tSNE of disentangled F l on test set of CADCars for different levels. M SU, Sp and Se mean Minibus, SUV, Sports and Sedan, P1∼ P4 and P6 mean Pose 1, 2, 3, 4 and 6 respectively.

Figure 16: 2D tSNE of disentangled F l on test set of ShapeNet-C for different levels. S and T mean Sofa andTable, Lo, C, Lc, W, B and Te mean Loveseat, Club chair, L-couch, Work table, Billiards and Tennis-table respectively.

Figure 17: 2D tSNE of disentangled F l on test set of ShapeNet-P for different levels. L, C, W and B mean Loveseat, Club chair, Work table and Billiards respectively.

Figure 20: Semantic translation results of baselines and HDN-full on CelebA. Different columns denote results of using F 1 , {R l } L l=2 or their combinations disentangled from the target images to replace the corresponding levels of the sources. Ground truths of R 2 , R 3 , R 4 are gender, smile and hair color, respectively.

, all images split by official train/test protocol are processed by a pre-trained faster-rcnn head detector, and then cropped and resized to 128*128 resolution as the inputs for HDN. Examples of such hierarchical data are shown in Fig.23. Network architecture and training hyper-parameters are same with those of CelebA, CADCars and ShapeNet introduced above.

Figure 22: Semantic translation results of compared cGANs and HDN on CelebA. R 2 , R 3 , R 4 are gender, smile and hair color, respectively. StarGAN only needs attribute conditions to generate images, and the target in its row has no use for it. ELEGANT-2 is trained with only gender and smile attributes. The ELEGANT models can only change one attribute from the target each time.

Figure 25: Semantic translation results of the source images controlled by hierarchically disentangled features of the targets on ImageNet.



Table content: <table><tr><td>chair Loveseat</td><td>L-couch</td><td>Work</td></tr></table>

) Hierarchical structure

Table content: <table><tr><td>Common</td><td/></tr><tr><td>features</td><td>Furniture</td></tr></table>



Table content: <table><tr><td/><td>Aggregation</td></tr><tr><td>Unique</td><td>Aggregation</td></tr><tr><td>features</td><td/></tr><tr><td colspan="2">(b) Hierarchical features extraction</td></tr></table>

(b)  for example, it can be regarded as a furniture since it contains the attribute subset {A 1 + ... + A i }, and be classified to a table in terms of the attribute subset {A 1 + ... + A i + A i+1 + ... + A j } present in it. Therefore, the disentangled feature F l for our objectives in Fig.1(b) is actually the reflection of the attribute subset formulated in Eqn.(1). Moreover, since the hierarchical correlations (i.e. the inherited relationship) among categories in different hierarchies, obviously the subset {A 1 + ... + A i + A i+1 + ... + A j } includes the subset {A 1 + ... + A i }, naturally leading to the disentangled F l-1 being the proper subset of F l .

Table content: <table/>

+𝑅𝑅 4 𝑅𝑅 3 +𝑅𝑅 4 𝑅𝑅 2 +𝑅𝑅 3

Table content: <table><tr><td>Source Target</td><td>𝐹𝐹 1</td><td>𝑅𝑅 2</td><td>𝑅𝑅 3</td><td>𝑅𝑅 4</td><td>𝑅𝑅 2</td><td>Σ(𝑅𝑅)</td></tr></table>

Accuracy of hierarchical classifications for test and generated (Gen.) images.

Table content: <table><tr><td/><td/><td colspan="2">Fashion-MNIST</td><td colspan="2">CADCars</td><td colspan="2">ShapeNet-C</td><td colspan="2">ShapeNet-P</td></tr><tr><td>Test</td><td>Gen.</td><td>Test</td><td>Gen.</td><td>Test</td><td>Gen.</td><td>Test</td><td>Gen.</td><td>Test</td><td>Gen.</td></tr><tr><td colspan="10">Level 2 0.9570 0.9387 0.9629 0.9779 0.9781 0.9792 0.9941 0.9941 0.9323 0.8863</td></tr><tr><td colspan="10">Level 3 0.9232 0.9103 0.9336 0.8464 0.9798 0.9670 0.9844 0.8865 0.9190 0.8413</td></tr><tr><td colspan="2">Level 4 0.8932 0.8799</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></table>

mAP results of retrieval for compared methods in different semantic levels.

Table content: <table/>

Hierarchical prediction performance for seen test set and unseen leaf-level categories.

Table content: <table><tr><td>Source Target Result</td><td/><td/><td/></tr><tr><td/><td>Source Target</td><td>Result</td><td>Source Target</td><td>Result</td></tr><tr><td>CelebA</td><td>ShapeNet-C</td><td/><td>ShapeNet-P</td></tr></table>



Table content: <table/>

Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. CoRR, abs/1710.00935, 2017. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE,CVPR, 2018.

Table content: <table><tr><td>Bin Zhao, Fei-Fei Li, and Eric P. Xing. Large-scale category structure aware image categorization.</td></tr><tr><td>In NIPS, pp. 1251-1259, 2011.</td></tr><tr><td>Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gener-</td></tr><tr><td>ative models. In ICML, pp. 4091-4099, 2017.</td></tr><tr><td>Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network</td></tr><tr><td>decisions: Prediction difference analysis. CoRR, abs/1702.04595, 2017.</td></tr></table>

Table, Lo, C, Lc, W, B and Te mean Loveseat, Club chair, L-couch, Work table, Billiards and Tennis-table respectively.

Table content: <table/>

table and Billiards respectively. Accuracy of hierarchical classifications for real and generated images by baselines and our full method in different semantic levels on CelebA. w/o reconfea, reconimg, local denote HDN trained without the feature reconstruction, image reconstruction or local classification loss, respectively. Real means the result of real images in test set.

Table content: <table><tr><td/><td>RAF</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Male</td><td/><td colspan="2">Female</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Smile</td><td>Neutral</td><td>Smile</td><td>Neutral</td><td>Sedan</td><td>Minibus</td><td>Convertible</td><td>Hatchback</td><td>Pickup</td><td>SUV</td></tr></table>

Figure21: 2D tSNE of disentangled F l by HDN without reconstruction feature loss on test set of CelebA for different levels. For easy understand, M and F mean male and female, S and N mean Smile and Neural, Bl, G and Br mean Black, Golden and Brown hair respectively. Comparisons of image quality of baselines and HDN. IS and FID measure the fidelity, and LPIPS measures the diversity of images. For IS and LPIPS, larger is better. For FID, lower is better.

Table content: <table><tr><td>M+S+Bl M+S+G M+S+Br M+N+Bl M+N+G M+N+Br F+S+Bl F+S+G F+S+Br F+N+Bl F+N+G F+N+Br</td></tr><tr><td>(d) Level 4</td></tr></table>

Peer review (iclr2020):
This paper proposes an algorithm for supervising networks for image classification and reconstruction with the object*s hierarchical categories in mind. The claimed benefits are the improved generalizability and interpretability. The paper reports per-category-level analysis on the semantic image reconstruction task and retrieval on seen and unseen object categories. I am currently leaning towards weak accept because I find the paper*s claim and technical details generally convincing and simultaneously extracting low-level and high-level features trained using the hierarchical levels of categories is novel. Generalization to unseen categories tends to be a good proxy for real world performance and directly learning the high level categories is a useful idea for doing so. Although I am leaning towards weak accept, I think this paper is close to borderline because the findings do not seem experimentally well validated. It would be more interesting to see Table 3 on multiple unseen categories instead of one special case per dataset. Another idea for experiments is doing cross-dataset evaluations where different datasets may have different leaf categories but shared high level ones. I think it may also be interesting to compare with a non-hierarchical retrieval model and then obtain their high-level prediction accuracy using the corresponding parent level categories. The paper generally needs polishing. Minor typos I found: Page 5: classificaiton, classifers Page 6: intuitional->intuitive Page 7: an significant Page 8: leraning Page 1: human Figure 3: arbitary