Connecting Sphere Manifolds Hierarchically for Regularization

1 INTRODUCTION . Applying inductive biases or prior knowledge to inference models is a popular strategy to improve their generalization performance ( Battaglia et al. , 2018 ) . For example , a hierarchical structure is found based on the similarity or shared characteristics between samples and thus becomes a basic criterion to categorize particular objects . The known hierarchical structures provided by the datasets ( e.g. , ImageNet ( Deng et al. , 2009 ) classified based on the WordNet graph ; CIFAR100 ( Krizhevsky , 2009 ) in ten different groups ) can help the network identify the similarity between the given samples . In classification tasks , the final layer of neural networks maps embedding vectors to a discrete target space . However , there is no mechanism forcing similar categories to be distributed close to each other in the embedding . Instead , we may observe classes to be uniformly distributed after training , as this simplifies the separation by the last fully-connected layer . This behavior is a consequence of seeing the label structure as ‘ flat , ’ i.e. , when we omit to consider the hierarchical relationships between classes ( Bilal et al. , 2017 ) . To alleviate this problem , in this study , we force similar classes to be closer in the embedding by forcing their hyperplanes to follow a given hierarchy . One way to realize that is by making children nodes dependent on parent nodes and constraining their distance through a regularization term . However , the norm itself does not give a relevant information on the closeness between classifiers . Indeed , two classifiers are close if they classify two similar points in the same class . This means similar classifiers have to indicate a similar direction . Therefore , we have to focus on the angle between classifiers , which can be achieved through spherical constraints . Contributions . In this paper , we propose a simple strategy to incorporate hierarchical information in deep neural network architectures with minimal changes to the training procedure , by modifying only the last layer . Given a hierarchical structure in the labels under the form of a tree , we explicitly force the classifiers of classes to belong to a sphere , whose center is the classifier of their super-class , recursively until we reach the root ( see Figure 2 ) . We introduce the spherical fully-connected layer and the hierarchically connected layer , whose combination implements our technique . Finally , we investigate the impact of Riemannian optimization instead of simple norm normalization . By its nature , the proposed technique is quite versatile because the modifications only affect the structure of last fully-connected layer of the neural network . Thus , it can be combined with many other strategies ( like spherical CNN from Xie et al . ( 2017 ) , or other deep neural network architectures ) . Related works . Hierarchical structures are well-studied , and their properties can be effectively learned using manifold embedding . The design of the optimal embedding to learn the latent hierarchy is a complex task , and was extensively studied in the past decade . For example , Word2Vec ( Mikolov et al. , 2013b ; a ) and Poincaré embedding ( Nickel & Kiela , 2017 ) showed a remarkable performance in hierarchical representation learning . ( Du et al. , 2018 ) forced the representation of sub-classes to “ orbit ” around the representation of their super-class to find similarity based embedding . Recently , using elliptical manifold embedding ( Batmanghelich et al. , 2016 ) , hyperbolic manifolds ( Nickel & Kiela , 2017 ; De Sa et al. , 2018 ; Tifrea et al. , 2018 ) , and a combination of the two ( Gu et al. , 2019 ; Bachmann et al. , 2019 ) , shown that the latent structure of many data was non-Euclidean ( Zhu et al. , 2016 ; Bronstein et al. , 2017 ; Skopek et al. , 2019 ) . ( Xie et al. , 2017 ) showed that spheres ( with angular constraints ) in the hidden layers also induce diversity , thus reducing over-fitting in latent space models . Mixing hierarchical information and structured prediction is not new , especially in text analysis ( Koller & Sahami , 1997 ; McCallum et al. , 1998 ; Weigend et al. , 1999 ; Wang et al. , 1999 ; Dumais & Chen , 2000 ) . Partial order structure of the visual-semantic hierarchy is exploited using a simple order pair with max-margin loss function in ( Vendrov et al. , 2016 ) . The results of previous studies indicate that exploiting hierarchical information during training gives better and more resilient classifiers , in particular when the number of classes is large ( Cai & Hofmann , 2004 ) . For a given hierarchy , it is possible to design structured models incorporating this information to improve the efficiency of the classifier . For instance , for support vector machines ( SVMs ) , the techniques reported in ( Cai & Hofmann , 2004 ; 2007 ; Gopal et al. , 2012 ; Sela et al. , 2011 ) use hierarchical regularization , forcing the classifier of a super-class to be close to the classifiers of its sub-classes . However , the intuition is very different in this case , because SVMs do not learn the embedding . In this study , we consider that the hierarchy of the class labels is known . Moreover , we do not change prior layers of the deep neural network , and only work on the last layer that directly contributed to build hyperplanes for a classification purpose . Our work is thus orthogonal to those works on embedding learning , but not incompatible . Comparison with hyperbolic/Poincaré/graph networks . Hyperbolic network is a recent technique that shows impressive results for hierarchical representation learning . Poincaré networks ( Nickel & Kiela , 2017 ) were originally designed to learn the latent hierarchy of data using low-dimension embedding . To alleviate their drawbacks due to a transductive property which can not be used for unseen graph inference , hyperbolic neural networks equipped set aggregation operations have been proposed ( Chami et al. , 2019 ; Liu et al. , 2019 ) . These methods have been mostly focused on learning embedding using a hyperbolic activation function for hierarchical representation . Our technique is orthogonal to these works : First , we assume that the hierarchical structure is not learnt but already known . Second , our model focuses on generating individual hyperplanes of embedding vectors given by the network architecture . While spherical geometry has a positive curvature , moreover , that of hyperbolic space has a constant negative curvature . However , our technique and hyperbolic networks are not mutually exclusive . Meanwhile focusing on spheres embedded in Rd in this study , it is straightforward to consider spheres embedded in hyperbolic spaces . 2 HIERARCHICAL REGULARIZATION . 2.1 DEFINITION AND NOTATIONS We assume we have samples with hierarchically ordered classes . For instance , apple , banana , and orange are classes that may belong to the super-class “ fruits. ” This represents hierarchical relationships with trees , as depicted in Figure 1 . We identify nodes in the graph through the path taken in the tree . To represent the leaf ( highlighted in blue in Figure 1 ) , we use the notation n { 1,3,2 } . This means it is the second child of the super-class n { 1,3 } , and recursively , until we reach the root . More formally , we identify nodes as np , where p is the path to the node . A path uniquely defines a node where only one possible path exists . Using the concatenation , between the path p and its child i , a new path p̃ can be defined as follows , p̃ = 〈p , i〉 ( 1 ) We denote P the set of all paths in the tree starting from the root , with cardinality |P| . Notice that |P| is also the number of nodes in the tree ( i.e. , number of classes and super-classes ) . We distinguish the set P from the set L , the set of paths associated to nodes whose label appears in the dataset . Although L may equal to P , this is not the case in our experiments . We show an example in Appendix A . 2.2 SIMILARITY BETWEEN OBJECTS AND THEIR REPRESENTATION . Let X be the network input ( e.g . an image ) , and φθ ( X ) be its representation , i.e. , the features of X extracted by a deep neural network parameterized by θ . We start with the following observation : Given a representation , super-class separators should be similar to separators for their sub-classes . This assumption implies the following direct consequence . All objects whose labels belong to the same super-class have a similar representation . That is a natural property that we may expect from a good representation . For instance , two dogs from different breeds should share more common features than that of a dog shares with an apple . Therefore , the parameter of the classifiers that identify dog ’ s breed should also be similar . Their difference lies in the parameters associated to some specific features that differentiate breeds of dogs . Although this is not necessarily satisfied with arbitrary hierarchical classification , we observe this in many existing datasets . For instance , Caltech-UCSD Birds 200 and Stanford dogs are datasets that classify , respectively , birds and dogs in term of their breeds . A possible example where this assumption may not be satisfied is a dataset whose super-classes are “ labels whose first letter is « · » . ” 2.3 HIERARCHICAL REGULARIZATION . Starting from a simple observation in the previous section , we propose a regularization technique that forces the network to have similar representation for classes along a path p , which implies having similar representation between similar objects . More formally , if we have an optimal classifier wp for the super-class p and a classifier w〈p , i〉 for the class 〈p , i〉 , we expect that ‖wp − w〈p , i〉‖ is small . ( 2 ) If this is satisfied , separators for objects in the same super-class are also similar because ‖w〈p , i〉 − w〈p , j〉‖ = ‖ ( w〈p , i〉 − wp ) − ( w〈p , j〉 − wp ) ‖ ≤ ‖wp − w〈p , i〉‖︸ ︷︷ ︸ small + ‖wp − w〈p , j〉‖︸ ︷︷ ︸ small . ( 3 ) However , the optimal classifier for an arbitrary representation φθ ( X ) may not satisfy equation 2 . The naive and direct way to ensure equation 2 is through hierarchical regularization , which forces classifiers in the same path to be close to each other . 2.4 HIERARCHICAL LAYER AND HIERARCHICALLY CONNECTED LAYER . In the previous section , we described the hierarchical regularization technique given a hierarchical structure in the classes . In this section , we show how to conveniently parametrize equation 2 . We first express the classifier as a sum of vectors δ defined recursively as follows : w〈p , i〉 = wp + δ〈p , i〉 , δ { } = 0 , ( 4 ) where { } is the root . It is possible to consider δ { } 6= 0 , which shifts separating hyper-planes . We do not consider this case in this paper . Given equation 4 , we have that ‖δ〈p , i〉‖ is small in equation 2 . Finally , it suffices to penalize the norm of δ〈p , i〉 during the optimization . Notice that , by construction , the number of δ ’ s is equal to the number of nodes in the hierarchical tree . Next , consider the output of CNNs for classification , φθ ( · ) TW , ( 5 ) where θ denotes the parameters of the hidden layers , W = [ w1 , . . . , w|L| ] denotes the last fullyconnected layer , and wi denotes the separator for the class i . For simplicity , we omit potential additional nonlinear functions , such as a softmax , on top of the prediction . We have parametrized wi following the recursive formula in equation 4 . To define the matrix formulation of equation 4 , we first introduce the Hierarchical layer H which plays an important role . This hierarchical layer can be identified to the adjacency matrix of the hierarchical graph . Definition 1 . ( Hierarchical layer ) . Consider ordering over the sets P and L , i.e. , for i = 1 , . . . , |P| and j = 1 , . . . , |L| , P = { p1 , . . . , pi , . . . , p|P| } and L = { p1 , . . . , pj , . . . , p|L| } . In other words , we associate to all nodes an index . Then , the hierarchical layer H is defined as H ∈ B|P|×|L| , Hi , j = 1 if npi npj , 0 otherwise . ( 6 ) where npi npj means npj is a parent of npi . We illustrate an example of H in Appendix A . The next proposition shows that equation 5 can be written using a simple matrix-matrix multiplication , involving the hierarchical layer . Proposition 1 . Consider a representation φθ ( · ) , where φθ ( · ) ∈ Rd . LetW be the matrix of separators W = [ wp1 , . . . , wp|L| ] , pi ∈ L , ( 7 ) where the separators are parametrized as equation 4 . Let ∆ be defined as ∆ ∈ Rd×|P| , ∆ = [ δp1 , . . . , δp|P| ] , ( 8 ) where P and L are defined in Section 2.1 . Consider the hierarchical layer defined in Definition 1 . Then , the matrix of separators W can be expressed as W = ∆H . ( 9 ) We can see W = ∆H as a combination of an augmented fully-connected layer , combined with the hierarchical layer that selects the right columns of ∆ , hence the term hierarchically connected layer . The ` 2 regularization of the δ can be conducted by the parameter weight decay , which is widely used in training of neural networks . The hierarchical layer H is fixed , while ∆ is learnable . This does not affect the complexity of the back-propagation significantly , as ∆H is a simple linear form . The size of the last layer slightly increases , from |L| × d to |P| × d , where d is the dimension of the representation φθ ( · ) . For instance , in the case of Tiny-ImageNet , the number of parameters of the last layer only increases by roughly 36 % ; nevertheless , the increased number of parameters of the last layer is still usually negligible in comparison with the total number of parameters for classical network architectures .

Summary: In this paper, the authors proposed a novel reparameterization framework of the last network layer that takes semantic hierarchy into account. Specifically, the authors assume a predefined hierarchy graph, and model the classifier of child classes as a parent classifier plus offsets $\delta$ recursively. The authors show that such hierarchy can be parameterized a matrix multiplication $\Delta \mathbf{H}$ where $\mathbf{H}$ is predefined by the graph. In addition, the authors further propose to fix the norm of $\delta$ in a decaying manner with respect to path length. The resulting spherical objective is optimized via Riemannian gradient descent.