Hierarchical Disentangle Network for Object Representation Learning

1 INTRODUCTION . Representation learning , as one basic and hot topic in machine learning and computer vision community , has achieved significant progress in recent years on different tasks such as recognition ( Russakovsky et al. , 2015 ) , detection ( Ren et al. , 2015 ; Redmon et al. , 2016 ; Liu et al. , 2016b ) and generation ( Goodfellow et al. , 2014 ) , benefiting from the rapid development of representation learned by deep neural networks . Considering the strong capacity of deep representation , in this paper , we mainly focus on the deep representation learning framework . Despite great success the deep representations have achieved as mentioned above , two important problems are still unresolved or less considered , i.e . the interpretability and the disentanglement of the learned representations . In the past decades , various works have been developed to reveal the black box of deep learning ( Zeiler & Fergus , 2014 ; Dosovitskiy & Brox , 2016b ; Bau et al. , 2017 ; Simonyan et al. , 2013 ; Stock & Cissé , 2017 ; Zhang et al. , 2017 ) and move us closer to the goal of disentangling the variations within data ( Reed et al. , 2014 ; Mathieu et al. , 2016 ; Rifai et al. , 2012 ; Tran et al. , 2017 ; Gonzalez-Garcia et al. , 2018 ; Huang et al. , 2018 ; Chen et al. , 2016 ) . Even though they have brought great insights to us , they still have some limitations . For instance , ( Chen et al. , 2016 ; Xie et al. , 2017 ; Zhao et al. , 2017 ) learn to disentangle variation factors within each category using generative models , instead of investigating the similarities and differences among categories , leading to poor discriminability . Therefore , the learned representations would not well conform to human perception . Though ( Gonzalez-Garcia et al. , 2018 ; Huang et al. , 2018 ) try to obtain the domain-invariant and domain-specific knowledge , they can only handle two categories one time , which is not that efficient . In this paper , we attempt to learn disentangled representations in a more natural and efficient manner . Let us first discuss how humans understand an object . Generally speaking , an object can be regarded as the combination of many semantic attributes . Hundreds of thousands of objects in the world can be clustered and recognized by humans just because we can figure out the common and unique attributes of an object compared to others . Besides , a man who never play the billiards can only recognize a table in an image , while a sports fan may regard it as a billiard table . Both of them are right since categories have natural hierarchical structure . As shown in Fig . 1 ( a ) , given six leaf-level categories , they can be organized in a three-level hierarchy considering the common and different features they have . Each child category in the hierarchy is a special case of its parent category since it inherits all features from its parent category and has extra features that are not present in its parent category . From another perspective , each parent category is the abstraction of all its child categories considering it contains the attributes that are present in all its child categories . Then we come back to the task of disentangling representation learning . It aims to learn the representation encoding useful information that can be applied in other tasks ( e.g . building classifiers and predictors ) ( Bengio et al. , 2013 ) . Taking the hierarchical nature of categories into account , if we only learn the representations of an object in a flat manner for a specific category level as previous works do , it will not be scalable and comprehensive for the machine to be qualified for various tasks in the real world . Our work aims to exploit the natural hierarchical characteristics among categories to divide the representation learning in a coarse-to-fine manner , such that each level only focuses on learning the specific representations . For instance , given a billiard table image in Fig . 1 ( b ) , it tangles the information of being a furniture , a table and a billiard table . We first extract the features that only contain the information of furniture from the image . By tracing from the root to leaf level , more and more information is extracted until we can recognize its belonging categories in all hierarchical levels . By doing so , the disentangled representations are expected to find wide and promising applications . For example , one can transfer the semantics in a specific category level from one object to another while keep information of other levels unchanged . Besides , it would help for the hierarchical image compression task using different levels of the disentangled representations . To achieve the objective of hierarchical disentangling and simultaneously interpreting the results so that humans can understand , we propose the hierarchical disentangle network ( HDN ) , which draws lessons from hierarchical classification and the recent proposed generative adversarial nets ( Goodfellow et al. , 2014 ) . Extensive experiments are conducted on four popular object datasets to validate the effectiveness of our method . 2 RELATED WORKS . Disentangling Deep Representations . The goal of disentangling representation learning is to discover factors of variation within data ( Bengio et al. , 2013 ) . Recent years have witnessed a substantial interest on such research area ( Tenenbaum & Freeman , 1996 ) , including works based on deep learning ( Reed et al. , 2014 ; Mathieu et al. , 2016 ; Rifai et al. , 2012 ; Wang et al. , 2017 ; Tran et al. , 2017 ; Gonzalez-Garcia et al. , 2018 ; Huang et al. , 2018 ; Chen et al. , 2016 ) . ( Rifai et al. , 2012 ) is probably the earliest to learn disentangled representations using deep networks for the task of emotion recognition . ( Reed et al. , 2014 ) is based on a higher-order Boltzmann machine and regards each factor variation of the manifold as its sub-manifold . ( Mathieu et al. , 2016 ) and ( Chen et al. , 2016 ) leverage the generative adversarial nets ( GAN ) to learn factors of variation . Recently , cross-domain translation methods ( Gonzalez-Garcia et al. , 2018 ; Huang et al. , 2018 ) learn the domain-invariant and domain-specific representations . These works ignore the existing natural and inherent hierarchy relationships among categories , with which we can conduct the disentangling in a coarse-to-fine manner such that each level only focuses on learning the specific representations . Network Interpretability . Network interpretability aims to learn how the network works via visualizing it from the perspective that humans can understand . These methods can be briefly divided into two groups according to whether the visualization is involved in the network during training , i.e . the off-line methods and online methods . The off-line methods make attempts to visualize patterns in image space that activate each convolutional filter ( Zeiler & Fergus , 2014 ; Dosovitskiy & Brox , 2016b ; a ; Bau et al. , 2017 ; 2019 ) or to interpret the area in an image that is responsible for the network prediction ( Simonyan et al. , 2013 ; Fong & Vedaldi , 2017 ; Zintgraf et al. , 2017 ; Abbasi-Asl & Yu , 2017 ; Stock & Cissé , 2017 ; Palacio et al. , 2018 ; Geirhos et al. , 2019 ) . While such methods can explain what has already been learned by the model , they can not improve the model interpretability in return . Instead , the online works propose to directly learn interpretable representations during training ( Li et al. , 2018 ; Zhang et al. , 2017 ) . However , these methods mainly focus on figuring out the running mechanism of networks while paying less attention to dissecting variations of the features among categories , which can not make models really understand their inputs . Hierarchy-regularized Learning . Semantic hierarchies have been explored in object classification task for accelerating recognition ( Griffin & Perona , 2008 ; Marszalek & Schmid , 2008 ) , obtaining a sequence of predictions ( Deng et al. , 2012 ; Ordonez et al. , 2013 ) , making use of category relation graphs ( Deng et al. , 2014 ; Ding et al. , 2015 ) , and improving recognition performance as additional supervision ( Zhao et al. , 2011 ; Srivastava & Salakhutdinov , 2013 ; Hwang & Sigal , 2014 ; Yan et al. , 2015 ; Goo et al. , 2016 ; Ahmed et al. , 2016 ) . While these discriminative classification works have achieved their expected goals , they usually lack interpretability . To address such issues , ( Xie et al. , 2017 ; Zhao et al. , 2017 ) propose to use generative models to disentangle the factors from low-level representations to high-level ones that can construct a specific object . ( Singh et al. , 2019 ) uses an unsupervised generative framework to hierarchically disentangle the background , object shape and appearance from an image . However , they either deal with each category in isolation or ignore the discriminability of learned features , and thus can not accurately disentangle the differences and similarities among categories . 3 HIERARCHICAL REPRESENTATION LEARNING . Supposing that a category hierarchy is given in the form shown in Fig . 1 ( a ) , we use l = 1 , ... , L to denote the level of hierarchy ( L for the leaf level and 1 for the root level ) , Kl to denote the number of nodes at level l , nkl to denote the k-th node at level l , and C k l to denote the number of children for nkl . As illustrated in Fig . 1 ( b ) , given an original object image denoted as I o , our goal is to extract the feature Fl in the l-th level . Generally speaking , an object O can be described as the combination of a group of visual attributes : O = A1 + ... + Ai︸ ︷︷ ︸ level=1 + Ai+1 + ... + Aj ︸ ︷︷ ︸ level=2 ... + Aj+1 + ... + Am ︸ ︷︷ ︸ level=l + ∆ ( O ) ( 1 ) where ∆ ( O ) represents currently undefined attributes existing on O . As we have discussed , humans classify O in a particular category level according to a subset of the whole attribute set in Eqn. ( 1 ) . Take the object in Fig . 1 ( b ) for example , it can be regarded as a furniture since it contains the attribute subset { A1 + ... + Ai } , and be classified to a table in terms of the attribute subset { A1 + ... + Ai + Ai+1 + ... +Aj } present in it . Therefore , the disentangled feature Fl for our objectives in Fig . 1 ( b ) is actually the reflection of the attribute subset formulated in Eqn. ( 1 ) . Moreover , since the hierarchical correlations ( i.e . the inherited relationship ) among categories in different hierarchies , obviously the subset { A1 + ... + Ai + Ai+1 + ... + Aj } includes the subset { A1 + ... + Ai } , naturally leading to the disentangled Fl−1 being the proper subset of Fl . Taking these into consideration , we design the hierarchical disentangle network ( HDN ) based on the autoencoder architecture in Fig . 2 . The encoder E dissects the hierarchical representations given a semantic hierarchy . The decoder G plays the role of an interpreter to reflect the variations of semantic in the image space for different hierarchical levels guided by the hierarchical discriminator Dadv and classifiers Dcls ( they share most network architecture except the output layers ) .

Summary: This paper studies the problem of learning disentangled representation in a hierarchical manner. It proposed a hierarchical disentangle network (HDN) which tackles the disentangling process in a coarse-to-fine manner. Specifically, common representations are captured at root level and unique representations are learned at lower hierarchical level. The HDN is trained in a generative adversarial network (GAN) manner, with additional hierarchical classification loss enforcing the disentanglement. Experiments are conducted on CelebA (attributes), Fashion-MNIST (category), and CAD Cars (category & pose).