CO-ATTENTIVE EQUIVARIANT NEURAL NETWORKS: FOCUSING EQUIVARIANCE ON TRANSFORMATIONS CO-OCCURRING IN DATA
Year: 2020

Abstract:
Equivariance is a nice property to have as it produces much more parameter efficient neural architectures and preserves the structure of the input through the feature mapping. Even though some combinations of transformations might never appear (e.g., an upright face with a horizontal nose), current equivariant architectures consider the set of all possible transformations in a transformation group when learning feature representations. Contrarily, the human visual system is able to attend to the set of relevant transformations occurring in the environment and utilizes this information to assist and improve object recognition. Based on this observation, we modify conventional equivariant feature mappings such that they are able to attend to the set of co-occurring transformations in data and generalize this notion to act on groups consisting of multiple symmetries. We show that our proposed co-attentive equivariant neural networks consistently outperform conventional rotation equivariant and rotation & reflection equivariant neural networks on rotated MNIST and CIFAR-10.

heading: 1 INTRODUCTION
text: Thorough experimentation in the fields of psychology and neuroscience has provided support to the intuition that our visual perception and cognition systems are able to identify familiar objects despite modifications in size, location, background, viewpoint and lighting (Bruce & Humphreys, 1994). Interestingly, we are not just able to recognize such modified objects, but are able to characterize which modifications have been applied to them as well. As an example, when we see a picture of a cat, we are not just able to tell that there is a cat in it, but also its position, its size, facts about the lighting conditions of the picture, and so forth. Such observations suggest that the human visual system is equivariant to a large transformation group containing translation, rotation, scaling, among others. In other words, the mental representation obtained by seeing a transformed version of an object, is equivalent to that of seeing the original object and transforming it mentally next.
These fascinating abilities exhibited by biological visual systems have inspired a large field of research towards the development of neural architectures able to replicate them. Among these, the most popular and successful approach is the Convolutional Neural Network (CNN) (LeCun et al., 1989), which incorporates equivariance to translation via convolution. Unfortunately, in counterpart to the human visual system, CNNs do not exhibit equivariance to other transformations encountered in visual data (e.g., rotations). Interestingly, however, if an ordinary CNN happens to learn rotated copies of the same filter, the stack of feature maps becomes equivariant to rotations even though individual feature maps are not (Cohen & Welling, 2016). Since ordinary CNNs must learn such rotated copies independently, they effectively utilize an important number of network parameters suboptimally to this end (see Fig. 3 in Krizhevsky et al. (2012)). Based on the idea that equivariance in CNNs can be extended to larger transformation groups by stacking convolutional feature maps, several approaches have emerged to extend equivariance to, e.g., planar rotations (Dieleman et al., 2016; Marcos et al., 2017; Weiler et al., 2018; Li et al., 2018), spherical rotations (Cohen et al., 2018; Worrall & Brostow, 2018; Cohen et al., 2019), scaling (Marcos et al., 2018; Worrall & Welling, 2019) and general transformation groups (Cohen & Welling, 2016), such that transformed copies of a single entity are not required to be learned independently.
Although incorporating equivariance to arbitrary transformation groups is conceptually and theoretically similar1, evidence from real-world experiences motivating their integration might strongly differ. Several studies in neuroscience and psychology have shown that our visual system does not react equally to all transformations we encounter in visual data. Take, for instance, translation and rotation. Although we easily recognize objects independently of their position of appearance, a large corpus of experimental research has shown that this is not always the case for in-plane rotations. Yin (1969) showed that mono-oriented objects, i.e., complex objects such as faces which are customarily seen in one orientation, are much more difficult to be accurately recognized when presented upsidedown. This behaviour has been reproduced, among others, for magazine covers (Dallett et al., 1968), symbols (Henle, 1942) and even familiar faces (e.g., from classmates) (Brooks & Goldstein, 1963). Intriguingly, Schwarzer (2000) found that this effect exacerbates with age (adults suffer from this effect much more than children), but, adults are much faster and accurate in detecting mono-oriented objects in usual orientations. Based on these studies, we draw the following conclusions:
• The human visual system does not perform (fully) equivariant feature transformations to visual data. Consequently, it does not react equally to all possible input transformations encountered in visual data, even if they belong to the same transformation group (e.g., in-plane rotations).
• The human visual system does not just encode familiarity to objects but seems to learn through experience the poses in which these objects customarily appear in the environment to assist and improve object recognition (Freire et al., 2000; Riesenhuber et al., 2004; Sinha et al., 2006).
Complementary studies (Tarr & Pinker, 1989; Oliva & Torralba, 2007) suggest that our visual system encodes orientation atypicality relative to the context rather than on an absolute manner (Fig. 1). Motivated by the aforementioned observations we state the co-occurrence envelope hypothesis:
The Co-occurrence Envelope Hypothesis. By allowing equivariant feature mappings to detect transformations that co-occur in the data and focus learning on the set formed by these co-occurrent transformations (i.e., the co-occurrence envelope of the data), one is able to induce learning of more representative feature representations of the data, and, resultantly, enhance the descriptive power of neural networks utilizing them. We refer to one such feature mapping as co-attentive equivariant.
Identifying the co-occurrence envelope. Consider a rotation equivariant network receiving two copies of the same face (Fig. 2a). A conventional rotation equivariant network is required to perform inference and learning on the set of all possible orientations of the visual patterns constituting a face regardless of the input orientation (Fig. 2b). However, by virtue of its rotation equivariance, it is able to recognize rotated faces even if it is trained on upright faces only. A possible strategy to simplify the task at hand could be to restrict the network to react exclusively to upright faces (Fig. 2c). In this case, the set of relevant visual pattern orientations becomes much smaller, at the expense of disrupting equivariance to the rotation group. Resultantly, the network would risk becoming unable to detect faces in any other orientation than those it is trained on. A better strategy results from restricting the set of relevant pattern orientations by defining them relative to one another
1It is achieved by developing feature mappings that utilize the transformation group in the feature mapping itself (e.g., translating a filter in the course of a feature transformation is used to obtain translation equivariance).
(e.g., mouth orientation relative to the eyes) as opposed to absolutely (e.g., upright mouth) (Fig. 2d). In such a way, we are able to exploit information about orientation co-occurrences in the data without disrupting equivariance. The set of co-occurrent orientations in Fig. 2d corresponds to the co-occurrence envelope of the samples in Fig. 2a for the transformation group defined by rotations.
In this work, we introduce co-attentive equivariant feature mappings and apply them on existing equivariant neural architectures. To this end, we leverage the concept of attention (Bahdanau et al., 2014) to modify existing mathematical frameworks for equivariance, such that co-occurrent transformations can be detected. It is critical not to disrupt equivariance in the attention procedure as to preserve it across the entire network. To this end, we introduce cyclic equivariant self-attention, a novel attention mechanism able to preserve equivariance to cyclic groups.
Experiments and results. We explore the effects of co-attentive equivariant feature mappings for single and multiple symmetry groups. Specifically, we replace conventional rotation equivariant mappings in p4-CNNs (Cohen & Welling, 2016) and DRENs (Li et al., 2018) with co-attentive ones. We show that co-attentive rotation equivariant neural networks consistently outperform their conventional counterparts in fully (rotated MNIST) and partially (CIFAR-10) rotational settings. Subsequently, we generalize cyclic equivariant self-attention to multiple similarity groups and apply it on p4m-CNNs (Cohen & Welling, 2016) (equivariant to rotation and mirror reflections). Our results are in line with those obtained for single symmetry groups and support our stated hypothesis.
Contributions.
• We propose the co-occurrence envelope hypothesis and demonstrate that conventional equivariant mappings are consistently outperformed by our proposed co-attentive equivariant ones.
• We generalize co-attentive equivariant mappings to multiple symmetry groups and provide, to the best of our knowledge, the first attention mechanism acting generally on symmetry groups.

heading: 2 PRELIMINARIES
text: Equivariance. We say that a feature mapping f : X → Y is equivariant to a (transformation) group G (orG-equivariant) if it commutes with actions of the groupG acting on its domain and codomain:
f(TXg (x)) = T Y g (f(x)) ∀g ∈ G, x ∈ X (1)
where T (·)g denotes a group action in the corresponding space. In other words, the ordering in which we apply a group action Tg and the feature mapping f is inconsequential. There are multiple reasons as of why equivariant feature representations are advantageous for learning systems. Since group actions TXg produce predictable and interpretable transformations T Y g in the feature space, the hypothesis space of the model is reduced (Weiler et al., 2018) and the learning process simplified (Worrall et al., 2017). Moreover, equivariance allows the construction of L-layered networks by
stacking several equivariant feature mappings {f (1), ..., f (l), ..., f (L)} such that the input structure as regarded by the group G is preserved (e.g., CNNs and input translations). As a result, an arbitrary intermediate network representation (f (l) ◦ ... ◦ f (1))(x), l ∈ L, is able to take advantage of the structure of x as well. Invariance is an special case of equivariance in which TYg = IdY , the identity, and thus all group actions in the input space are mapped to the same feature representation.
Equivariant neural networks. In neural networks, the integration of equivariance to arbitrary groups G has been achieved by developing feature mappings f that utilize the actions of the group G in the feature mapping itself. Interestingly, equivariant feature mappings encode equivariance as parameter sharing with respect to G, i.e., the same weights are reused for all g ∈ G. This makes the inclusion of larger groups extremely appealing in the context of parameter efficient networks.
Conventionally, the l-th layer of a neural network receives a signal x(l)(u, λ) (where u ∈ Z2 is the spatial position and λ ∈ Λl is the unstructured channel index, e.g., RGB channels in a color image), and applies a feature mapping f (l) : Z2 × Λl → Z2 × Λl+1 to generate the feature representation x(l+1)(u, λ). In CNNs, the feature mapping f (l) := f (l)T is defined by a convolution
2 (?R2 ) between the input signal x(l) and a learnable convolutional filter W (l)λ′,λ, λ ′ ∈ Λl, λ ∈ Λl+1:
x(l+1)(u, λ) = [x(l) ?R2 W (l) λ′,λ](u, λ) = ∑ λ′,u′ x(l)(u+ u′, λ′)W (l) λ′,λ(u ′) (2)
By sliding W (l)λ′,λ across u, CNNs are able to preserve the spatial structure of the input x through the feature mapping f lT and successfully provide equivariance to the translation group T = (Z2,+).
The underlying idea for the extension of equivariance to larger groups in CNNs is conceptually equivalent to the strategy utilized by LeCun et al. (1989) for translation equivariance. Consider, for instance, the inclusion of equivariance to the set of rotations by θr degrees3: Θ = {θr = r 2πrmax } rmax r=1. To this end, we modify the feature mapping f (l) := f (l)R : Z2×Θ×Λl → Z2×Θ×Λl+1 to include the rotations defined by Θ. Let x(l)(u, r, λ) and W (l)λ′,λ(u, r) be the input and the convolutional filter with an affixed index r for rotation. The roto-translational convolution (?R2oΘ) f (l) R is defined as:
x(l+1)(u, r, λ) = [x(l) ?R2oΘ W (l) λ′,λ](u, r, λ) = ∑ λ′,r′,u′ x(l)(u+ u′, r′, λ′)W (l) λ′,λ(θru ′, r′ − r) (3)
Since f (l)R produces (dim(Θ) = rmax) times more output feature maps than f (l) T , we need to learn much smaller convolutional filters W (l)λ′,λ to produce the same number of output feature channels.
Learning equivariant neural networks. Consider the change of variables g = u, G = Z2, g ∈ G and g = (u, r), G = Z2oΘ, g ∈ G in Eq. 2 and Eq. 3, respectively. In general, neural networks are learned via backpropagation (LeCun et al., 1989) by iteratively applying the chain rule of derivation to update the network parameters. Intuitively, the networks outlined in Eq. 2 and Eq. 3 obtain feedback from all g ∈ G and, resultantly, are inclined to learn feature representations that perform optimally on the entire group G. However, as outlined in Fig. 2 and Section 1, several of those feature combinations are not likely to simultaneously appear. Resultantly, the hypothesis space of the model as defined by Weiler et al. (2018) might be further reduced.
Note that this reasoning is tightly related to existing explanations for the large success of spatial (Xu et al., 2015; Woo et al., 2018; Zhang et al., 2018) and temporal (Luong et al., 2015; Vaswani et al., 2017; Mishra et al., 2017; Zhang et al., 2018) attention in deep learning architectures.

heading: 3 CO-ATTENTIVE EQUIVARIANT NEURAL NETWORKS
text: In this section we define co-attentive feature mappings and apply them in the context of equivariant neural networks (Figure 3). To this end, we introduce cyclic equivariant self-attention and utilize it to construct co-attentive rotation equivariant neural networks. Subsequently, we show that cyclic equivariant self-attention is extendable to larger symmetry groups and make use of this fact to construct co-attentive neural networks equivariant to rotations and mirror reflections.
2Formally it is as a correlation. However, we hold on to the standard deep learning terminology. 3The reader may easily verify that Θ (and hence Z2 o Θ, with (o) the semi-direct product) forms a group.

heading: 3.1 CO-ATTENTIVE ROTATION EQUIVARIANT NEURAL NETWORKS
text: To allow rotation equivariant networks to utilize and learn co-attentive equivariant representations, we introduce an attention operator A(l) on top of the roto-translational convolution f (l)R with which discernment along the rotation axis r of the generated feature responses x(l)(u, r, λ) is possible. Formally, our co-attentive rotation equivariant feature mapping f (l)R is defined as follows:
x(l+1) = f (l) R (x (l)) = A(l)(f (l)R (x (l))) = A(l) ( [x(l) ?R2oΘ W (l) λ′,λ]) (4)
Theoretically, A(l) could be defined globally over f (l)R (x(l)) (i.e., simultaneously along u, r, λ) as depicted in Eq. 4. However, we apply attention locally to: (1) grant the algorithm enough flexibility to attend locally to the co-occurrence envelope of feature representations and, (2) utilize attention exclusively along the rotation axis r, such that our contributions are clearly separated from those possibly emerging from spatial attention. To this end, we apply attention pixel-wise on top of f
(l) R (x (l)) (Eq. 5). Furthermore, we assign a single attention instance A(l)λ to each learned feature representation and utilize it across the spatial dimension of the output feature maps4:
x(l+1)(u, r, λ) = A(l)λ ({x (l+1)(u, r̂, λ)}rmaxr̂=1)(r) (5)
Attention and self-attention. Consider a source vector x = (x1, ..., xn) and a target vector y = (y1, ..., ym). In general, an attention operator A leverages information from the source vector x (or multiple feature mappings thereof) to estimate an attention matrix A ∈ [0, 1]n×m, such that: (1) the elementAi,j provides an importance assessment of the source element xi with reference to the target element yj and (2) the sum of importance over all xi is equal to one: ∑ iAi,j = 1. Subsequently, the matrix A is utilized to modulate the original source vector x as to attend to a subset of relevant source positions with regard to yj : x̃j = (A:,j)T x (where is the Hadamard product). A special case of attention is that of self-attention (Cheng et al., 2016), in which the target and the source vectors are equal (y := x). In other words, the attention mechanism estimates the influence of the sequence x on the element xj for its weighting.
4For a more meticulous discussion on how Eq. 5 attains co-occurrent attention, see Appendix A.
In general, the attention matrix5 A ∈ [0, 1]n×m is constructed via nonlinear space transformations fÃ : Rn → Rn×m of the source vector x, on top of which the softmax function is applied: A:,j = softmax(fÃ(x):,j). This ensures that the properties previously mentioned hold. Typically, the mappings fÃ found in literature take feature transformation pairs of x as input (e.g., {s,H} in RNNs (Luong et al., 2015), {Q,K} in self-attention networks (Vaswani et al., 2017)), and perform (non)-linear mappings on top of it, ranging from multiple feed-forward layers (Bahdanau et al., 2014) to several operations between the transformed pairs (Luong et al., 2015; Vaswani et al., 2017; Mishra et al., 2017; Zhang et al., 2018). Due to the computational complexity of these approaches and the fact that we do extensive pixel-wise usage of fÃ on every network layer, their direct integration in our framework is computationally prohibitive. To circumvent this problem, we modify the usual self-attention formulation as to enhance its descriptive power in a much more compact setting.
Compact local self-attention. Initially, we relax the range of values of A from [0, 1]n×n to Rn×n. This allows us to encode much richer relationships between element pairs (xi, xj) at the cost of less interpretability. Subsequently, we define A = xT Ã, where Ã ∈ Rn×n is a matrix of learnable parameters. Furthermore, instead of directly applying softmax on the columns of A, we first sum over the contributions of each element xi to obtain a vector a = { ∑ iAi,j}nj=1, which is then passed to the softmax function. Following Vaswani et al. (2017), we prevent the softmax function from reaching regions of low gradient by scaling its argument by ( √ dim(A))−1 = (1/n): ã = softmax((1 / n) a). Lastly, we counteract the contractive behaviour of the softmax function by normalizing ã before weighting x as to preserve the magnitude range of its argument. This allows us to use A in deep architectures. Our compact self-attention mechanism is summarized as follows:
a = { ∑ iAi,j} n j=1 = ∑ i(x
T Ã)i,j = xÃ (6) ã = softmax((1 / n) a) (7)
x̂ = A(x) = (ã /max(ã)) x (8) The cyclic equivariant self-attention operator AC . Consider {x(u, r, λ)}rmaxr=1, the vector of responses generated by a roto-translational convolution fR stacked along the rotation axis r. By applying self-attention along r, we are able to generate an importance matrix A ∈ Rrmax×rmax relating all pairs of (θi, θj)-rotated responses in the rotational group Θ at a certain position. We refer to this attention mechanism as full self-attention (AF ). Although AF is able to encode arbitrary linear source-target relationships for each target position, it is not restricted to conserve equivariance to Θ. Resultantly, we risk incurring into the behavior outlined in Fig. 2c. Before we further elaborate on this issue, we introduce the cyclic permutation operator Pi, which induces a cyclic shift of i positions on its argument: σP i
(xj) = x(j+i)mod(dim(x)), ∀xj ∈ x.
Consider a full self-attention operator AF acting on top of a roto-translational convolution fR. Let p be an input pattern to which fR only produces a strong activation in the feature map x(r̂) = fR(p)(r̂), r̂ ∈ {r}rmaxr=1. Intuitively, during learning, only the corresponding attention coefficients Ã;,r̂ in AF would be significantly increased. Now, consider the presence of the input pattern θip, a θi-rotated variant of p. By virtue of the rotational equivariance property of the feature mapping fR, we obtain (locally) an exactly equal response to that of p up to a cyclic permutation of i positions on r, and thus, we obtain a strong activation in the feature map Pi(x(r̂)) = x(σPi(r̂)). We encounter two problems in this setting: AF is not be able to detect that p and θip correspond to the exact same input pattern and, as each but the attention coefficients Ã:,j is small, the network might considerably damp the response generated by θip. As a result, the network might (1) squander important feedback information during learning and (2) induce learning of repeated versions of the same pattern for different orientations. In other words, AF does not behave equivariantly as a function of θi. Interestingly, we are able to introduce prior-knowledge into the attention model by restricting the structure of Ã. By leveraging the idea of equivariance to the cyclic group Cn, we are able to solve the problems exhibited byAF and simultaneously reduce the number of additional parameters required by the self-attention mechanism (from r2max to rmax). Consider again the input patterns p and θip. We incorporate the intuition that p and θip are one and the same entity, and thus, fR (locally) generates the same output feature map up to a cyclic permutation Pi: fR(θip) = Pi(fR(p)). Consequently, the attention mechanism should produce the exact same output for both p and θip up to the same cyclic permutation Pi. In other words,A (and thus Ã) should be equivariant to cyclic permutations.
5Technically, each column of A is restricted to a simplex and hence A lives in a subspace of [0, 1]n×m.
A well-known fact in mathematics is that a matrix A is equivariant with respect to cyclic permutations of the domain if and only if it is circulant (Alkarni, 2001; Åhlander & Munthe-Kaas, 2005). We make use of this certitude and leverage the concept of circulant matrices to impose cyclic equivariance to the structure of Ã. Formally, a circulant matrix C ∈ Rn×n is composed of n cyclic permutations of its defining vector c = {ci}ni=1, such that its j-th column is a cyclic permutation of j − 1 positions of c: C:,j = Pj−1(c)T . We construct our cyclic equivariant self-attention operator AC by defining Ã as a circulant matrix specified by a learnable attention vector aC = {aCi } rmax i=1:
Ã = {Pj−1(aC)T }nj=1 (9)
and subsequently applying Eqs. 6 - 8. Resultantly, AC is able to assign the responses generated by fR for rotated versions of an input pattern p to a unique entity: fR(θip) = Pi(fR(p)), and dynamically adjust its output to the angle of appearance θi, such that the attention operation does not disrupt its propagation downstream the network: AC(fR(θip)) = Pi(AC(fR(p))). Consequently, the attention weights aC are updated equally regardless of specific values of θi. Due to these properties, AC does not incur in any of the problems outlined earlier in this section. Conclusively, our co-attentive rotation equivariant feature mapping f (l)R is defined as follows:
x(l+1)(u, r, λ) = f (l) R (x (l))(u, r, λ) = AC(l)λ ( [x(l) ?R2oΘ W (l) λ′,λ])(u, r, λ) (10)
Note that a co-attentive equivariant feature mapping fR is approximately equal (up to a normalized softmax operation (Eq. 8)) to a conventional equivariant one fR, if Ã = αI for any α ∈ R.

heading: 3.2 EXTENDING AC TO MULTIPLE SYMMETRY GROUPS
text: The self-attention mechanisms outlined in the previous section are easily extendable to larger groups consisting of multiple symmetries. Consider, for instance, the group θrm of rotations by θr degrees and mirror reflections m defined analogously to the group p4m in Cohen & Welling (2016). Let p(u, r,m, λ) be an input signal with an affixed index m ∈ {m0,m1} for mirror reflections (m1 indicates mirrored) and fθrm be a group convolution (Cohen & Welling, 2016) on the θrm group. The group convolution fθrm produces two times as many output channels (2rmax : m0rmax+m1rmax) as those generated by the roto-translational convolution fR (Eq. 3, Fig. 3).
Full self-attention AF can be integrated directly by modulating the output of fθrm as depicted in Sec. 3.1 with Ã ∈ R2rmax×2rmax . Here, AF relates each of the group convolution responses with one another. However, just as for fR, AF disrupts the equivariance property of fθrm to the θrm group. Similarly, the cyclic equivariant self-attention operator AC can be extended to multiple symmetry groups as well. Before we continue, we introduce the cyclic permutation operator Pi,t, which induces a cyclic shift of i positions on its argument along the transformation axis t. Consider the input patterns p and θip outlined in the previous section and mp, a mirrored instance of p. Let x(u, r,m, λ) = fθrm(p)(u, r,m, λ) be the response of the group convolution fθrm for the input pattern p. By virtue of the rotation equivariance property of fθrm, the generated response for θip is equivalent to that of p up to a cyclic permutation of i positions along the rotation axis r: fθrm(θip)(u, r,m, λ) = Pi,r(fθrm(p))(u, r,m, λ) = x(u, σP i
(r),m, λ). Similarly, by virtue of the mirror equivariance property of fθrm, the response generated by mp is equivalent to that of p up to a cyclic permutation of one position along the mirroring axis m: fθrm(mp)(u, r,m, λ) = P1,m(fθrm(p))(u, r,m, λ) = x(u, r, σP 1
(m), λ). Note that if we take two elements from a group g, h, their composition (gh) is also an element of the group. Resultantly, fθrm((mθ
i)p)(u, r,m, λ) = (P1,m ◦ Pi,r)(fθrm(p))(u, r,m, λ) = P1,m(Pi,r(x))(u, r,m, λ) = P1,m(x)(u, σPi(r),m, λ) = x(u, σPi(r), σP1(m), λ). In other words, in order to extend AC to the θrm group, it is necessary to restrict the structure of Ã such that it respects the permutation laws imposed by the equivariant mapping fθrm. Let us rewrite x(u, r,m, λ) as x(u, g, λ), g = (mr) ∈ {m0,m1} × {r̂}rmaxr̂=1. In this case, we must impose a circulant block matrix structure on Ã such that: (1) the composing blocks permute internally as defined by Pi,r and (2) the blocks themselves permute with one another as defined by P1,m. Formally, Ã is defined as:
Ã = [ Ã1 Ã2 Ã2 Ã1 ] (11)
where {Ãi ∈ Rrmax×rmax}, i ∈ {1, 2} are circulant matrices (Eq. 9). Importantly, the ordering of the permutation laws in Ã is interchangeable if the input vector is modified accordingly, i.e., g = (rm).
Conclusively, cyclic equivariant self-attention AC is directly extendable to act on any G-equivariant feature mapping fG, and for any symmetry group G, if the group actions TYg produce cyclic permutations on the codomain of fG. To this end, one must restrict the structure of Ã to that of a circulant block matrix, such that all permutation laws of TYg hold: T Y g (AC(fG)) = AC(TYg (fG)), ∀g ∈ G.

heading: 4 EXPERIMENTS
text: Experimental Setup. We validate our approach by exploring the effects of co-attentive equivariant feature mappings for single and multiple symmetry groups on existing equivariant architectures. Specifically, we replace conventional rotation equivariant mappings in p4-CNNs (Cohen & Welling, 2016) and DRENs (Li et al., 2018) with co-attentive equivariant ones and evaluate their effects in fully (rotated MNIST) and partially (CIFAR-10) rotational settings. Similarly, we evaluate coattentive equivariant maps acting on multiple similarity groups by replacing equivariant mappings in p4m-CNNs (Cohen & Welling, 2016) (equivariant to rotation and mirror reflections) likewise. Unless otherwise specified, we replicate as close as possible the same data processing, initialization strategies, hyperparameter values and evaluation strategies utilized by the baselines in our experiments. Note that the goal of this paper is to study and evaluate the relative effects obtained by co-attentive equivariant networks with regard to their conventional counterparts. Accordingly, we do not perform any additional tuning relative to the baselines. We believe that improvements on our reported results are feasible by performing further parameter tuning (e.g., on the network structure or the used hyperparameters) on the proposed co-attentive equivariant networks.
The additional learnable parameters, i.e., those associated to the cyclic self-attention operator (Ã) are initialized identically to the rest of the layer. Subsequently, we replace the values of Ã along the diagonal by 1 (i.e., diag(Ãinit) = 1) such that Ãinit approximately resembles the identity I and, hence, co-attentive equivariant layers are initially approximately equal to equivariant ones.
Rotated MNIST. The rotated MNIST dataset (Larochelle et al., 2007) contains 62000 gray-scale 28x28 handwritten digits uniformly rotated on the entire circle [0, 2π). The dataset is split into training, validation and tests sets of 10000, 2000 and 50000 samples, respectively. We replace rotation equivariant layers in p4-CNN (Cohen & Welling, 2016), DREN and DRENMaxPooling (Li et al., 2018) with co-attentive ones. Our results show that co-attentive equivariant networks consistently outperform conventional ones (see Table 1).
CIFAR-10. The CIFAR-10 dataset (Krizhevsky et al., 2009) consists of 60000 real-world 32x32 RGB images uniformly drawn from 10 classes. Contrarily to the rotated MNIST dataset, this dataset does not exhibit rotation symmetry. The dataset is split into training, validation and tests sets of 40000, 10000 and 10000 samples, respectively. We replace equivariant layers in the p4 and p4m variations of the All-CNN (Springenberg et al., 2014) and the ResNet44 (He et al., 2016) proposed by Cohen & Welling (2016) with co-attentive ones. Likewise, we modify the r x4-variations of the NIN (Lin et al., 2013) and ResNet20 (He et al., 2016) models proposed by Li et al. (2018) in the same manner. Our results show that co-attentive equivariant networks consistently outperform conventional ones in this setting as well (see Table 1).
Training convergence of equivariant networks. Li et al. (2018) reported that adding too many rotational equivariant (isotonic) layers decreased the performance of their models on CIFAR-10. As a consequence, they did not report results on fully rotational equivariant networks for this setting and attributed this behaviour to the non-symmetricity of the data. We noticed that, with equal initialization strategies, rotational equivariant CNNs were much more prone to divergence than ordinary CNNs. This behaviour can be traced back to the additional feedback resulting from roto-translational convolutions (Eq. 3) compared to ordinary ones (Eq. 2). After further analysis, we noticed that the data preprocessing strategy utilized by Li et al. (2018) leaves some very large outlier values in the data (|x| >100), which strongly contribute to the behaviour outlined before. In order to evaluate the relative contribution of co-attentive equivariant neural networks we constructed fully equivariant DREN architectures based on their implementation. Although the obtained results were much worse than those originally reported in Li et al. (2018), we were able to stabilize
training by clipping input values to the 99 percentile of the data (|x| ≤2.3) and reducing the learning rate to 0.01, such that the same hyperparameters could be used across all network types. The obtained results (see Table 1) signalize that DREN networks are comparatively better than CNNs both in fully and partially rotational settings, contradictorily to the conclusions drawn in Li et al. (2018).
This behaviour elucidates that although the inclusion of equivariance to larger transformation groups is beneficial both in terms of accuracy and parameter efficiency, one must be aware that such benefits are directly associated with an increase of the network susceptibility to divergence during training.

heading: 5 DISCUSSION AND FUTURE WORK
text: Our results show that co-attentive equivariant feature mappings can be utilized to enhance conventional equivariant ones. Interestingly, co-attentive equivariant mappings are beneficial both in partially and fully rotational settings. We attribute this to the fact that a set of co-occurring orientations between patterns can be easily defined (and exploited) in both settings. It is important to note that we utilized attention independently over each spatial position u on the codomain of the corresponding group convolution. Resultantly, we were restricted to mappings of the form xA, which, in turn, constraint our attention mechanism to have a circulant structure in order to preserve equivariance (since group actions acting in the codomain of the group convolution involve cyclic permutations and cyclic self-attention is applied in the codomain of the group convolution).
In future work, we want to extend the idea presented here to act on the entire group simultaneously (i.e., along u as well). By doing so, we lift our current restriction to mappings of the form xA and therefore, may be able to develop attention instances with enhanced descriptive power. Following the same line of though, we want to explore incorporating attention in the convolution operation itself. Resultantly, one is not restricted to act exclusively on the codomain of the convolution, but instead, is able to impose structure in the domain of the mapping as well. Naturally, such an approach could lead to enhanced descriptiveness of the incorporated attention mechanism. Moreover, we want to utilize and extend more complex attention strategies (e.g., Bahdanau et al. (2014); Luong et al. (2015); Vaswani et al. (2017); Mishra et al. (2017)) such that they can be applied to large transformation groups without disrupting equivariance. As outlined earlier in Section 3.1, this becomes very challenging from a computational perspective as well, as it requires extensive usage of the corresponding attention mechanism. Resultantly, an efficient implementation thereof is mandatory. Furthermore, we want to extend co-attentive equivariant feature mappings to continuous (e.g., Worrall et al. (2017)) and 3D space (e.g., Cohen et al. (2018); Worrall & Brostow (2018); Cohen et al. (2019)) groups, and for applications other than visual data (e.g., speech recognition).
Finally, we believe that our approach could be refined and extended to a first step towards dealing with the enumeration problem of large groups (Gens & Domingos, 2014), such that functions acting on the group (e.g., group convolution) are approximated by evaluating them on the set of cooccurring transformations as opposed to on the entire group. Such approximations are expected to be very accurate, as non-co-occurrent transformations are rare. This could be though of as sharping up co-occurrent attention to co-occurrent restriction.

heading: 6 CONCLUSION
text: We have introduced the concept of co-attentive equivariant feature mapping and applied it in the context of equivariant neural networks. By attending to the co-occurrence envelope of the data, we are able to improve the performance of conventional equivariant ones on fully (rotated MNIST) and partially (CIFAR-10) rotational settings. We developed cyclic equivariant self-attention, an attention mechanism able to attend to the co-occurrence envelope of the data without disrupting equivariance to a large set of transformation groups (i.e., all transformation groups G, whose action in the codomain of a G-equivariant feature mapping produce cyclic permutations). Our obtained results support the proposed co-occurrence envelope hypothesis.

heading: ACKNOWLEDGMENTS
text: We gratefully acknowledge Jan Klein, Emile van Krieken, Jakub Tomczak and our anonymous reviewers for their helpful and valuable commentaries. This work is part of the Efficient Deep Learning (EDL) programme (grant number P16-25), which is partly founded by the Dutch Research Council (NWO) and Semiotic Labs.

heading: A OBTAINING CO-OCCURRENT ATTENTION VIA EQUATION 5
text: In this section, we provide a meticulous description on how co-occurrent attention is obtained via the method presented in the paper. Intuitively, a direct approach to address the problem illustrated in the introduction (Section 1) and Figure 2 requires an attention mechanism that acts simultaneously on r and λ (see Eq. 3). However, we illustrate how the depicted problem can be simplified such that attention along r is sufficient by taking advantage of the equivariance property of the network.
Let p be the input of a roto-translational convolution fR : Z2×Θ×Λ0 → Z2×Θ×Λ1 as defined in Eq. 3, and Θ be the set of rotations by θr degrees: Θ = {θr = r 2πrmax } rmax r=1. Let fR(p)(u) ∈ Rrmax×Λ1 be the matrix consisting of the rmax oriented responses for each λ ∈ Λ1 learned representation at a certain position u. Since the vectors fR(p)(u, λ) ∈ Rrmax , λ ∈ Λ1 permute cyclically as a result of the rotation equviariance property of fR, it is mandatory to ensure equivariance to cyclic permutations for each fR(p)(u, λ) during the course of the attention procedure (see Section 3).
At first sight, one is inclined to think that there is no connection between multiple vectors fR(p)(u, λ) in fR(p)(u), and, therefore, in order to exploit co-occurences, one must impose additional constraints along the λ axis. However, there is indeed an implicit restriction in fR(p)(u) along λ resulting from the rotation equivariance property of the mapping fR, which we can take advantage from to simplify the problem at hand. Consider, for instance, the input θip, a θi-rotated version of p. By virtue of the equivariance property of fR, we have (locally) that fR(θip) = Pi(fR(p)). Furthermore, we know that this property must hold for all the learned feature representations fR(p)(u, λ),∀λ ∈ Λ1. Resultantly, we have that:
fR(θip)(u, r, λ) = Pi(fR(p)(u, r, λ)) , ∀λ ∈ Λ1 (12) In other words, if one of the learned mappings fR(p)(u, r, λ) experiences a permutation Pi along r, all the learned representations fR(p)(u, r, λ), ∀λ ∈ Λ1 must experience the exact same permutation Pi as well. Resultantly, the equivariance property of the mapping fR ensures that all the Λ1 learned feature representations fR(p)(u, λ) “move synchronously” as a function of input rotation θi.
Likewise, if we apply a cyclic equivariant attention mechanism ACλ independently on top of each λ learned representation fR(p)(u, λ), we obtain that the relation
ACλ(fR(θip))(u, r, λ) = Pi(ACλ(fR(p))(u, r, λ)) , ∀λ ∈ Λ1 (13) must hold as well. Similarly to the case illustrated in Eq. 12 and given that ACλ is equivariant to cyclic permutations on the domain, we obtain that all the Λ1 learned attention masks ACλ “move synchronously” as a function of input rotation θi as well (see Fig. 4).
From Eq. 13 and Figure 4, one can clearly see that by utilizingACλ independently along r and taking advantage from the fact that all Λ1 learned feature representations are tied with one another via fR, one is able to prioritize learning of feature representations that co-occur together as opposed to the much looser formulation in Eq. 12, where feedback is obtained from all orientations.

Review summaries:
This paper describes an approach to applying attention in equivariant image classification CNNs so that the same transformation (rotation+mirroring) is selected for each kernel. For example, if the image is of an upright face, the upright eyes will be selected along with the upright nose, as opposed to allowing the rotation of each to be independent. Applying this approach to several different models on rotated MNIST and CIFAR-10 lead to smaller test errors in all cases.