Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring in Data

1 INTRODUCTION . Thorough experimentation in the fields of psychology and neuroscience has provided support to the intuition that our visual perception and cognition systems are able to identify familiar objects despite modifications in size , location , background , viewpoint and lighting ( Bruce & Humphreys , 1994 ) . Interestingly , we are not just able to recognize such modified objects , but are able to characterize which modifications have been applied to them as well . As an example , when we see a picture of a cat , we are not just able to tell that there is a cat in it , but also its position , its size , facts about the lighting conditions of the picture , and so forth . Such observations suggest that the human visual system is equivariant to a large transformation group containing translation , rotation , scaling , among others . In other words , the mental representation obtained by seeing a transformed version of an object , is equivalent to that of seeing the original object and transforming it mentally next . These fascinating abilities exhibited by biological visual systems have inspired a large field of research towards the development of neural architectures able to replicate them . Among these , the most popular and successful approach is the Convolutional Neural Network ( CNN ) ( LeCun et al. , 1989 ) , which incorporates equivariance to translation via convolution . Unfortunately , in counterpart to the human visual system , CNNs do not exhibit equivariance to other transformations encountered in visual data ( e.g. , rotations ) . Interestingly , however , if an ordinary CNN happens to learn rotated copies of the same filter , the stack of feature maps becomes equivariant to rotations even though individual feature maps are not ( Cohen & Welling , 2016 ) . Since ordinary CNNs must learn such rotated copies independently , they effectively utilize an important number of network parameters suboptimally to this end ( see Fig . 3 in Krizhevsky et al . ( 2012 ) ) . Based on the idea that equivariance in CNNs can be extended to larger transformation groups by stacking convolutional feature maps , several approaches have emerged to extend equivariance to , e.g. , planar rotations ( Dieleman et al. , 2016 ; Marcos et al. , 2017 ; Weiler et al. , 2018 ; Li et al. , 2018 ) , spherical rotations ( Cohen et al. , 2018 ; Worrall & Brostow , 2018 ; Cohen et al. , 2019 ) , scaling ( Marcos et al. , 2018 ; Worrall & Welling , 2019 ) and general transformation groups ( Cohen & Welling , 2016 ) , such that transformed copies of a single entity are not required to be learned independently . Although incorporating equivariance to arbitrary transformation groups is conceptually and theoretically similar1 , evidence from real-world experiences motivating their integration might strongly differ . Several studies in neuroscience and psychology have shown that our visual system does not react equally to all transformations we encounter in visual data . Take , for instance , translation and rotation . Although we easily recognize objects independently of their position of appearance , a large corpus of experimental research has shown that this is not always the case for in-plane rotations . Yin ( 1969 ) showed that mono-oriented objects , i.e. , complex objects such as faces which are customarily seen in one orientation , are much more difficult to be accurately recognized when presented upsidedown . This behaviour has been reproduced , among others , for magazine covers ( Dallett et al. , 1968 ) , symbols ( Henle , 1942 ) and even familiar faces ( e.g. , from classmates ) ( Brooks & Goldstein , 1963 ) . Intriguingly , Schwarzer ( 2000 ) found that this effect exacerbates with age ( adults suffer from this effect much more than children ) , but , adults are much faster and accurate in detecting mono-oriented objects in usual orientations . Based on these studies , we draw the following conclusions : • The human visual system does not perform ( fully ) equivariant feature transformations to visual data . Consequently , it does not react equally to all possible input transformations encountered in visual data , even if they belong to the same transformation group ( e.g. , in-plane rotations ) . • The human visual system does not just encode familiarity to objects but seems to learn through experience the poses in which these objects customarily appear in the environment to assist and improve object recognition ( Freire et al. , 2000 ; Riesenhuber et al. , 2004 ; Sinha et al. , 2006 ) . Complementary studies ( Tarr & Pinker , 1989 ; Oliva & Torralba , 2007 ) suggest that our visual system encodes orientation atypicality relative to the context rather than on an absolute manner ( Fig . 1 ) . Motivated by the aforementioned observations we state the co-occurrence envelope hypothesis : The Co-occurrence Envelope Hypothesis . By allowing equivariant feature mappings to detect transformations that co-occur in the data and focus learning on the set formed by these co-occurrent transformations ( i.e. , the co-occurrence envelope of the data ) , one is able to induce learning of more representative feature representations of the data , and , resultantly , enhance the descriptive power of neural networks utilizing them . We refer to one such feature mapping as co-attentive equivariant . Identifying the co-occurrence envelope . Consider a rotation equivariant network receiving two copies of the same face ( Fig . 2a ) . A conventional rotation equivariant network is required to perform inference and learning on the set of all possible orientations of the visual patterns constituting a face regardless of the input orientation ( Fig . 2b ) . However , by virtue of its rotation equivariance , it is able to recognize rotated faces even if it is trained on upright faces only . A possible strategy to simplify the task at hand could be to restrict the network to react exclusively to upright faces ( Fig . 2c ) . In this case , the set of relevant visual pattern orientations becomes much smaller , at the expense of disrupting equivariance to the rotation group . Resultantly , the network would risk becoming unable to detect faces in any other orientation than those it is trained on . A better strategy results from restricting the set of relevant pattern orientations by defining them relative to one another 1It is achieved by developing feature mappings that utilize the transformation group in the feature mapping itself ( e.g. , translating a filter in the course of a feature transformation is used to obtain translation equivariance ) . ( e.g. , mouth orientation relative to the eyes ) as opposed to absolutely ( e.g. , upright mouth ) ( Fig . 2d ) . In such a way , we are able to exploit information about orientation co-occurrences in the data without disrupting equivariance . The set of co-occurrent orientations in Fig . 2d corresponds to the co-occurrence envelope of the samples in Fig . 2a for the transformation group defined by rotations . In this work , we introduce co-attentive equivariant feature mappings and apply them on existing equivariant neural architectures . To this end , we leverage the concept of attention ( Bahdanau et al. , 2014 ) to modify existing mathematical frameworks for equivariance , such that co-occurrent transformations can be detected . It is critical not to disrupt equivariance in the attention procedure as to preserve it across the entire network . To this end , we introduce cyclic equivariant self-attention , a novel attention mechanism able to preserve equivariance to cyclic groups . Experiments and results . We explore the effects of co-attentive equivariant feature mappings for single and multiple symmetry groups . Specifically , we replace conventional rotation equivariant mappings in p4-CNNs ( Cohen & Welling , 2016 ) and DRENs ( Li et al. , 2018 ) with co-attentive ones . We show that co-attentive rotation equivariant neural networks consistently outperform their conventional counterparts in fully ( rotated MNIST ) and partially ( CIFAR-10 ) rotational settings . Subsequently , we generalize cyclic equivariant self-attention to multiple similarity groups and apply it on p4m-CNNs ( Cohen & Welling , 2016 ) ( equivariant to rotation and mirror reflections ) . Our results are in line with those obtained for single symmetry groups and support our stated hypothesis . Contributions . • We propose the co-occurrence envelope hypothesis and demonstrate that conventional equivariant mappings are consistently outperformed by our proposed co-attentive equivariant ones . • We generalize co-attentive equivariant mappings to multiple symmetry groups and provide , to the best of our knowledge , the first attention mechanism acting generally on symmetry groups . 2 PRELIMINARIES . Equivariance . We say that a feature mapping f : X → Y is equivariant to a ( transformation ) group G ( orG-equivariant ) if it commutes with actions of the groupG acting on its domain and codomain : f ( TXg ( x ) ) = T Y g ( f ( x ) ) ∀g ∈ G , x ∈ X ( 1 ) where T ( · ) g denotes a group action in the corresponding space . In other words , the ordering in which we apply a group action Tg and the feature mapping f is inconsequential . There are multiple reasons as of why equivariant feature representations are advantageous for learning systems . Since group actions TXg produce predictable and interpretable transformations T Y g in the feature space , the hypothesis space of the model is reduced ( Weiler et al. , 2018 ) and the learning process simplified ( Worrall et al. , 2017 ) . Moreover , equivariance allows the construction of L-layered networks by stacking several equivariant feature mappings { f ( 1 ) , ... , f ( l ) , ... , f ( L ) } such that the input structure as regarded by the group G is preserved ( e.g. , CNNs and input translations ) . As a result , an arbitrary intermediate network representation ( f ( l ) ◦ ... ◦ f ( 1 ) ) ( x ) , l ∈ L , is able to take advantage of the structure of x as well . Invariance is an special case of equivariance in which TYg = IdY , the identity , and thus all group actions in the input space are mapped to the same feature representation . Equivariant neural networks . In neural networks , the integration of equivariance to arbitrary groups G has been achieved by developing feature mappings f that utilize the actions of the group G in the feature mapping itself . Interestingly , equivariant feature mappings encode equivariance as parameter sharing with respect to G , i.e. , the same weights are reused for all g ∈ G. This makes the inclusion of larger groups extremely appealing in the context of parameter efficient networks . Conventionally , the l-th layer of a neural network receives a signal x ( l ) ( u , λ ) ( where u ∈ Z2 is the spatial position and λ ∈ Λl is the unstructured channel index , e.g. , RGB channels in a color image ) , and applies a feature mapping f ( l ) : Z2 × Λl → Z2 × Λl+1 to generate the feature representation x ( l+1 ) ( u , λ ) . In CNNs , the feature mapping f ( l ) : = f ( l ) T is defined by a convolution 2 ( ? R2 ) between the input signal x ( l ) and a learnable convolutional filter W ( l ) λ′ , λ , λ ′ ∈ Λl , λ ∈ Λl+1 : x ( l+1 ) ( u , λ ) = [ x ( l ) ? R2 W ( l ) λ′ , λ ] ( u , λ ) = ∑ λ′ , u′ x ( l ) ( u+ u′ , λ′ ) W ( l ) λ′ , λ ( u ′ ) ( 2 ) By sliding W ( l ) λ′ , λ across u , CNNs are able to preserve the spatial structure of the input x through the feature mapping f lT and successfully provide equivariance to the translation group T = ( Z2 , + ) . The underlying idea for the extension of equivariance to larger groups in CNNs is conceptually equivalent to the strategy utilized by LeCun et al . ( 1989 ) for translation equivariance . Consider , for instance , the inclusion of equivariance to the set of rotations by θr degrees3 : Θ = { θr = r 2πrmax } rmax r=1 . To this end , we modify the feature mapping f ( l ) : = f ( l ) R : Z2×Θ×Λl → Z2×Θ×Λl+1 to include the rotations defined by Θ . Let x ( l ) ( u , r , λ ) and W ( l ) λ′ , λ ( u , r ) be the input and the convolutional filter with an affixed index r for rotation . The roto-translational convolution ( ? R2oΘ ) f ( l ) R is defined as : x ( l+1 ) ( u , r , λ ) = [ x ( l ) ? R2oΘ W ( l ) λ′ , λ ] ( u , r , λ ) = ∑ λ′ , r′ , u′ x ( l ) ( u+ u′ , r′ , λ′ ) W ( l ) λ′ , λ ( θru ′ , r′ − r ) ( 3 ) Since f ( l ) R produces ( dim ( Θ ) = rmax ) times more output feature maps than f ( l ) T , we need to learn much smaller convolutional filters W ( l ) λ′ , λ to produce the same number of output feature channels . Learning equivariant neural networks . Consider the change of variables g = u , G = Z2 , g ∈ G and g = ( u , r ) , G = Z2oΘ , g ∈ G in Eq . 2 and Eq . 3 , respectively . In general , neural networks are learned via backpropagation ( LeCun et al. , 1989 ) by iteratively applying the chain rule of derivation to update the network parameters . Intuitively , the networks outlined in Eq . 2 and Eq . 3 obtain feedback from all g ∈ G and , resultantly , are inclined to learn feature representations that perform optimally on the entire group G. However , as outlined in Fig . 2 and Section 1 , several of those feature combinations are not likely to simultaneously appear . Resultantly , the hypothesis space of the model as defined by Weiler et al . ( 2018 ) might be further reduced . Note that this reasoning is tightly related to existing explanations for the large success of spatial ( Xu et al. , 2015 ; Woo et al. , 2018 ; Zhang et al. , 2018 ) and temporal ( Luong et al. , 2015 ; Vaswani et al. , 2017 ; Mishra et al. , 2017 ; Zhang et al. , 2018 ) attention in deep learning architectures .

Summary: This paper describes an approach to applying attention in equivariant image classification CNNs so that the same transformation (rotation+mirroring) is selected for each kernel. For example, if the image is of an upright face, the upright eyes will be selected along with the upright nose, as opposed to allowing the rotation of each to be independent. Applying this approach to several different models on rotated MNIST and CIFAR-10 lead to smaller test errors in all cases.