Building Deep Equivariant Capsule Networks

1 INTRODUCTION . The hierarchical component-structure of visual objects motivates their description as instances of class-dependent spatial grammars . The production-rules of such grammars specify this structure by laying out valid type-combinations for components of an object , their inter-geometry , as well as the behaviour of these with respect to transformations on the input . A system that aims to truly understand a visual scene must accurately learn such grammars for all constituent objects - in effect , learning their aggregational structures . One means of doing so is to have the internal representation of a model serve as a component-parsing of an input across several semantic resolutions . Further , in order to mimic latent compositionalities in objects , such a representation must be reflective of detected strengths of possible spatial relationships . A natural structure for such a representation is a parse-tree whose nodes denote components , and whose weighted parent-child edges denote the strengths of detected aggregational relationships . Capsule networks ( Hinton et al. , 2011 ) , ( Sabour et al. , 2017 ) are a family of deep neural networks that aim to build such distributed , spatially-aware representations in a multi-class setting . Each layer of a capsule network represents and detects instances of a set of components ( of a visual scene ) at a particular semantic resolution . It does this by using vector-valued activations , termed ’ capsules ’ . Each capsule is meant to be interpreted as being representative of a set of generalised pose-coordinates for a visual object . Each layer consists of capsules of several types that may be instantiated at all spatial locations depending on the nature of the image . Thus , given an image , a capsule network provides a description of its components at various ’ levels ’ of semantics . In order that this distributed representation across layers be an accurate component-parsing of a visual scene , and capture meaningful and inherent spatial relationships , deeper capsules are constructed from shallower capsules using a mechanism that combines backpropagation-based learning , and consensus-based heuristics . Briefly , the mechanism of creating deeper capsules from a set of shallower capsules is as follows . Each deeper capsule of a particular type receives a set of predictions for its pose from a local pool of shallower capsules . This happens by using a set of trainable neural networks that the shallower capsules are given as input into . These networks can be interpreted as aiming to capture possible part-whole relationships between the corresponding deeper and shallower capsules . The predictions thus obtained are then combined in a manner that ensures that the result reflects agreement among them . This is so that capsules are activated only when their component-capsules are in the right spatial relationship to form an instance of the object-type it represents . The agreement-based aggregation described just now is termed ’ routing ’ . Multiple routing algorithms exist , for example dynamic routing ( Sabour et al. , 2017 ) , EM-routing ( Hinton et al. , 2018 ) , SVD-based routing ( Bahadori , 2018 ) , and routing based on a clustering-like objective function ( Wang & Liu , 2018 ) . Based on their explicit learning of compositional structures , capsule networks can be seen as an alternative ( to CNNs ) for better learning of compositional representations . Indeed , CNN-based models do not have an inherent mechanism to explicitly learn or use spatial relationships in a visual scene . Further , the common use of layers that enforce local transformation-invariance , such as pooling , further limit their ability to accurately detect compositional structures by allowing for relaxations in otherwise strict spatial relations ( Hinton et al. , 2011 ) . Thus , despite some manner of hierarchical learning - as seen in their layers capturing simpler to more complex features as a function of depth - CNNs do not form the ideal representational model we seek . It is our belief that capsule-based models may serve us better in this regard . This much said , research in capsule networks is still in its infancy , and several issues have to be overcome before capsule networks can become universally applicable like CNNs . We focus on two of these that we consider as fundamental to building better capsule network models . First , most capsule-network models , in their current form , do not scale well to deep architectures . A significant factor is the fact that all pair-wise relationships between capsules of two layers ( upto a local pool ) are explicitly modelled by a unique neural network . Thus , for a ’ convolutional capsule ’ layer - the number of trainable neural networks depends on the product of the spatial extent of the windowing and the product of the number of capsule-types of each the two layers . We argue that this design is not only expensive , but also inefficient . Given two successive capsule-layers , not all pairs of capsule-types have significant relationships . This is due to them either representing object-components that are part of different classes , or being just incompatible in compositional structures . The consequences of this inefficiency go beyond poor scalability . For example , due to the large number of prediction-networks in this design , only simple functions - often just matrices - are used to model part-whole relationships . While building deep capsule networks , such a linear inductive bias can be inaccurate in layers where complex objects are represented . Thus , for the purpose of building deeper architectures , as well as more expressive layers , this inefficiency in the prediction phase must be handled . The second issue with capsule networks is more theoretical , but nonetheless has implications in practice . This is the lack , in general , of theoretical guarantees on equivariance . Most capsule networks only use intuitive heuristics to learn transformation-robust spatial relations among components . This is acceptable , but not ideal . A capsule network model that can detect compositionalities in a provablyinvariant manner are more useful , and more in line with the basic motivations for capsules . Both of the above issues are remedied in the following description of our model . First , instead of learning pair-wise relationships among capsules , we learn to projectively encode a description of each capsule-type for every layer . This we do by associating each capsule-type with a vector-valued function , given by a trainable neural network . This network assumes the role of the prediction mechanism in capsule networks . We interpret the role of this network as a means of encoding the manifold of legal pose-variations for its associated capsule-type . It is expected that , given proper training , shallower capsules that have no relationship with a particular capsule-type will project themselves to a vector of low activation ( for example , 2-norm ) , when input to the corresponding network . As an aside , it is this mechanism that gives the name to our model . We term this manifold the ’ space-of-variation ’ of a capsule-type . Since , we attempt to learn such spaces at each layer , we name our model ’ space-of-variation ’ networks ( SOVNET ) . In this design , the number of trainable networks for a given layer depend on the number of capsule-types of that layer . As mentioned earlier , the choice of prediction networks and routing algorithm is important to having guarantees on learning transformation-invariant compositional relationships . Thus , in order to ensure equivariance , which we show is sufficient for the above , we use group-equivariant convolutions ( GCNN ) ( Cohen & Welling , 2016 ) in the prediction phase . Thus , shallower capsules of a fixed type are input to a GCNN associated with a deeper capsule-type to obtain predictions for it . Apart from ensuring equivariance to transformations , GCNNs also allow for greater parameter-sharing ( across a set of transformations ) , resulting in greater awareness of local object-structures . We argue that this could potentially improve the quality of predictions when compared to isolated predictions made by convolutional capsule layers , such as those of ( Hinton et al. , 2018 ) . The last contribution of this paper is an equivariant degree-centrality based routing algorithm . The main idea of this method is to treat each prediction for a capsule as a vertex of a graph , whose weighted edges are given by a similarity measure on the predictions themselves . Our method uses the softmaxed values of the degree scores of the affinity matrix of this graph as a set of weights for aggregating predictions . The key idea being that predictions that agree with a majority of other predictions for the same capsule get a larger weight - following the principle of routing-by-agreement . While this method is only heuristic in the sense of optimality , it is provably equivariant and preserves the capsule-decomposition of an input . We summarise the contributions of this paper in the following : 1 . A general framework for a scalable capsule-network model . 2 . A particular instantiation of this model that uses equivariant convolutions , and an equivariant , degree-centrality-based routing algorithm . 3 . A graph-based framework for studying the representation of a capsule network , and the proof of the sufficiency of equivariance for the ( qualified ) preservation of this representation under transformations of the input . 4 . A set of proof-of-concept , evaluative experiments on affinely transformed variations of MNIST , FASHIONMNIST , and CIFAR10 , as well as separate experiments on KMNIST and SVHN that showcase the superior adapatability of SOVNET architectures to train and test-time geometric perturbations of the data , as well as their general performance . 2 SOVNET , EQUIVARIANCE , AND COMPOSITIONALITY . We begin with essential definitions for a layer of SOVNET , and the properties we wish to guarantee . Given a group ( G , ◦ ) , we formally describe the lth layer of a SOVNET architecture as the set of function-tuples { ( f li , ali ) : 0 ≤ i ≤ Nl − 1 ; f li : G → Rd l ; ali : G → [ 0 , 1 ] } . Here , Nl denotes the number of capsule-types at the lth layer , f li is a functional description of the d l-dimensional posevectors of instances of the ith capsule-type , and ali is a functional description of the corresponding activations . We model each capsule-type as a function over a group of transformations so as to allow for formal guarantees on transformation-equivariance . Thus , we also model images as function from a group to a representation-space . The main assumption being that the translation-group is a subgroup of the group in question . This is similar in approach to ( Cohen & Welling , 2016 ) . We wish for each capsule-type , both pose and activation-wise , to display equivariance . We present a formal definition of this notion . Consider a group ( G , ◦ ) and vector spaces V , W . Let T and T ′ be two group-representations for elements of G over V and W , respectively . Φ : V → W is said to be equivariant with respect to T and T ′ if ∀g ∈ G , ∀x ∈ V , Φ ( Tgx ) = T ′gΦ ( x ) . This definition translates to a preservation on transformations in the input-space to the output-space - something that allows no loss of information in compositional structures . As in ( Cohen & Welling , 2016 ) , we restrict the notion of equivariance in our model by using the operator Lg in place of the group-representation . Lg is given by [ Lgf ] ( x ) = f ( g−1x ) . Thus , if ⊗ denotes an operation between two functions , we require ( [ Lgf ] ⊗Ψ ) ( x ) = [ Lg ( f ⊗Ψ ) ] ( x ) . The operator ⊗ describes the change in representation space , and is dependent on the nature of the deep learning model . In the case of capsule networks ( and SOVNET ) , this change is given by routing among capsules as described in subsection 2.1 .

Summary: This paper combines CapsuleNetworks and GCNNs with a novel formulation. First they modify the CapsNet formulation by replacing the linear transformation between two capsule layers with a group convolution. Second they share the group equivarient convolution filters per all capsules of the lower layer.  Third, they change the similarity metric from a lower-upper similarity into a pairwise lower similarity and aggregation which makes it keep the equivarience. Since the cij does not depend on upper capsule anymore they only perform 1 routing iteration (no modification of the routing factors).