Siamese Capsule Networks

Capsule Networks have shown encouraging results on defacto benchmark computer vision datasets such as MNIST , CIFAR and smallNORB . Although , they are yet to be tested on tasks where ( 1 ) the entities detected inherently have more complex internal representations and ( 2 ) there are very few instances per class to learn from and ( 3 ) where point-wise classification is not suitable . Hence , this paper carries out experiments on face verification in both controlled and uncontrolled settings that together address these points . In doing so we introduce Siamese Capsule Networks , a new variant that can be used for pairwise learning tasks . The model is trained using contrastive loss with ` 2-normalized capsule encoded pose features . We find that Siamese Capsule Networks perform well against strong baselines on both pairwise learning datasets , yielding best results in the few-shot learning setting where image pairs in the test set contain unseen subjects . 1 INTRODUCTION . Convolutional Neural networks ( CNNs ) have been a mainstay model for a wide variety of tasks in computer vision . CNNs are effective at detecting local features in the receptive field , although the spatial relationship between features is lost when crude routing operations are performed to achieve translation invariance , as is the case with max and average pooling . Essentially , pooling results in viewpoint invariance so that small perturbations in the input do not effect the output . This leads to a significant loss of information about the internal properties of present entities ( e.g location , orientation , shape and pose ) in an image and relationships between them . The issue is usually combated by having large amounts of annotated data from a wide variety of viewpoints , albeit redundant and less efficient in many cases . As noted by hinton1985shape , from a psychology perspective of human shape perception , pooling does not account for the coordinate frames imposed on objects when performing mental rotation to identify handedness Rock ( 1973 ) ; McGee ( 1979 ) ; Humphreys ( 1983 ) . Hence , the scalar output activities from local kernel regions that summarize sets of local inputs are not sufficient for preserving reference frames that are used in human perception , since viewpoint information is discarded . Spatial Transformer Networks ( STN ) Jaderberg et al . ( 2015 ) have acknowledged the issue by using dynamic spatial transformations on feature mappings to enhance the geometric invariance of the model , although this approach addresses changes in viewpoint by learning to remove rotational and scale variance , as opposed to viewpoint variance being reflected in the model activations . Instead of addressing translation invariance using pooling operations , Hinton et al . ( 2011 ) have worked on achieving translation equivariance.The recently proposed Capsule Networks Sabour et al . ( 2017 ) ; Hinton et al . ( 2018 ) have shown encouraging results to address these challenges . Thus far , Capsule Networks have only been tested on datasets that have ( 1 ) a relatively sufficient number of instances per class to learn from and ( 2 ) utilized on tasks in the standard classification setup . This paper extends Capsule Networks to the pairwise learning setting to learn relationships between whole entity encodings , while also demonstrating their ability to learn from little data that can perform few-shot learning where instances from new classes arise during testing ( i.e zero-shot prediction ) . The Siamese Capsule Network is trained using a contrastive loss with ` 2-normalized encoded features and demonstrated on two face verification tasks . 2 CAPSULE NETWORKS . Hinton et al . ( 2011 ) first introduced the idea of using whole vectors to represent internal properties ( referred to as instantiation parameters that include pose ) of an entity with an associated activation probability where each capsule represents a single instance of an entity within in an image . This differs from the single scalar outputs in conventional neural networks where pooling is used as a crude routing operation over filters . Pooling performs sub-sampling so that neurons are invariant to viewpoint change , instead capsules look to preserve the information to achieve equivariance , akin to perceptual systems . Hence , pooling is replaced with a dynamic routing scheme to send lowerlevel capsule ( e.g nose , mouth , ears etc . ) outputs as input to parent capsule ( e.g face ) that represent part-whole relationships to achieve translation equivariance and untangles the coordinate frame of an entity through linear transformations . The idea has its roots in computer graphics where images are rendered given an internal hierarchical representation , for this reason the brain is hypothesized to solve an inverse graphics problem where given an image the cortex deconstructs it to its latent hierarchical properties . The original paper by Sabour et al . ( 2017 ) describes a dynamic routing scheme that represent these internal representations as vectors given a group of designated neurons called capsules , which consist of a pose vector u ∈ Rd and activation α ∈ [ 0 , 1 ] . The architecture consists of two convolutional layers that are used as the initial input representations for the first capsule layer that are then routed to a final class capsule layer . The initial convolutional layers allow learned knowledge from local feature representations to be reused and replicated in other parts of the receptive field . The capsule inputs are determined using a Iterative Dynamic Routing scheme . A transformation Wij is made to output vector ui of capsule CLi . The length of the vector ui represents the probability that this lower-level capsule detected a given object and the direction corresponds to the state of the object ( e.g orientation , position or relationship to upper capsule ) . The output vector ui is transformed into a prediction vector ûj|i , where ûj|i = Wijui . Then , ûj|i is weighted by a coupling coefficient cij to obtain sj = ∑ i cij ûj|i , where coupling coefficients for each capsule ∑ j cij = 1 and cij is got by log prior probabilities bij from a sigmoid function , followed by the softmax , cij = ebij/ ∑ k e bik . If ûLj|i has high scalar magnitude when multiplied by uL+1j then the coupling coefficient cij is increased and the remaining potential parent capsules coupling coefficients are decreased . Routing By Agreement is then performed using coincidence filtering to find tight clusters of nearby predictions . The entities output vector length is represented as the probability of an entity being present by using the nonlinear normalization shown in Equation 1 where vote vj is the output from total input sj , which is then used to compute the agreement aij = vj ûj|i that is added to the log prior bij . vj = ||sj ||2 1 + ||sj ||2 sj ||sj || ( 1 ) The capsule is assigned a high log-likelihood if densely connected clusters of predictions are found from a subset of s. The centroid of the dense cluster is output as the entities generalized pose . This coincidence filtering step can also be achieved by traditional outlier detection methods such as Random sample consensus ( RANSAC ) Fischler & Bolles ( 1987 ) and classical Hough Transforms Ballard ( 1987 ) for finding subsets of the feature space with high agreement . Although , the motivation for using the vector normalization of the instantiation parameters is to force the network to preserve orientation . Lastly , a reconstruction loss on the images was used for regularization which constrains th capsules to learn properties that can better encode the entities . In this paper , we do not use such regularization scheme by autoencoding pairs of input images , instead we use a variant of dropout . Extensions Of Capsule Networks Hinton et al . ( 2018 ) recently describe matrix capsules that perform routing by agreement using the expectation maximization ( EM ) algorithm , motivated by computer graphics where pose matrices are used to define rotations and translations of objects to account for viewpoint changes . Each parent capsule is considered a Gaussian and the pose matrix of each child capsule are considered data samples of the Gaussian . A given layer L contains a set of capsules CL such that ∀C ` ∃ { M ` , α ` } ∈ CL where pose matrix M ` ∈ Rn×n ( n = 4 ) and activation α ` ∈ [ 0 , 1 ] are the outputs . A vote is made V ` ij = M ` iW ` ij for the pose matrix of CL+1j where Wij ∈ Rn×n is a learned viewpoint invariant transformation matrix from capsule CLi → C L+1 j . EM determines the activation of C L+1 j as aj = σ ( λ ( βa − βu ∑ i rij − ∑ h cost h j ) ) where the costhj is the negative log-probability density weighted by the assignment probabilities rij , −βu is the negative log probability density per pose matrix computed to describe CL+1j . If C L+1 j is activated −βa is the cost for describing ( µj , σ2j ) from lower-level pose data samples along with rij and λ is the inverse temperature so as the assignment probability becomes higher the slope of the sigmoid curve becomes steeper ( represents the presence of an entity instead of the nonlinear vector normalization seen in Equation 1 ) . The network uses 1 standard convolutional layer , a primary capsule layer , 2 intermediate capsule convolutional layer , followed by the final class capsule layer . The matrix capsule network significantly outperformed CNNs on the SmallNORB dataset . LaLonde & Bagci ( 2018 ) introduce SegCaps which uses a locally connected dynamic routing scheme to reduce the number of parameters while using deconvolutional capsules to compensate for the loss of global information , showing best performance for segmenting pathological lungs from low dose CT scans . The model obtained a 39 % and 95 % reduction in parameters over baseline architectures while outperforming both . Bahadori ( 2018 ) introduced Spectral Capsule Networks demonstrated on medical diagnosis . The method shows faster convergence over the EM algorithm used with pose vectors . Spatial coincidence filters align extracted features on a 1-d linear subspace . The architecture consists of a 1d convolution followed by 3 residual layers with dilation . Residual blocks R are used as nonlinear transformations for the pose and activation of the first primary capsule instead of the linear transformation that accounts for rotations in CV , since deformations made in healthcare imaging are not fully understood . The weighted votes are obtained as sj , i = αiRj ( ui ) ∀i where Sj is a matrix of concatenated votes that are then decomposed using SVD , where the first singular value dimension s̃1 is used to capture most of the variance between votes , thus the activation aj activation is computed as σ ( η ( s21/ ∑ k s 2 k − b ) ) where s21/ ∑ k s 2 k is the ratio of all variance explained for all right singular vectors in V , b is optimized and η is decreased during training . The model is trained by maximizing the log-likelihood showing better performance than the spread loss used with matrix capsules and mitigates the problem of capsules becoming dormant . Wang & Liu ( 2018 ) formalize the capsule routing strategy as an optimization of a clustering loss and a KL regularization term between the coupling coefficient distribution and its past states . The proposed objective function follows as minC , S { L ( C , S ) : = − ∑ i ∑ j cij〈oj|i , sj〉 + α ∑ i ∑ j cij log cij } where oj|i = Tijµi/||Tij ||F and ||Tij ||F is the Frobenious norm of Tij . This routing scheme shows significant benefit over the original routing scheme by Sabour et al . ( 2017 ) as the number of routing iterations increase . Evidently , there has been a surge of interest within the research community . In contrast , the novelty presented in this paper is the pairwise learning capsule network scheme that proposes a different loss function , a change in architecture that compares images , aligns entities across images and describes a method for measuring similarity between final layer capsules such that inter-class variations are maximized and intra-class variations are minimized . Before describing these points in detail , we briefly describe the current state of the art work ( SoTA ) in face verification that have utilized Siamese Networks .

Summary: Authors present an adaptation of Capsule Networks for pairwise learning tasks. The pose vectors of final capsule layers for each tower is concatenated and passed through a fully connected layer to calculate the embedding for each tower's input. Then a contrastive loss based on a distance metric is optimized for the embeddings. An interesting regularizer (?) is used which is a dropout based on Capsule activation for connecting last layer capsules to the fully connected layer.