Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring in Data
Authors: David W. Romero
Mark Hoogendoorn
Venue:  (2020)
Keywords: Equivariant Neural Networks
Attention Mechanisms
Deep Learning

Abstract: Equivariance is a nice property to have as it produces much more parameter efficient neural architectures and preserves the structure of the input through the feature mapping. Even though some combinations of transformations might never appear (e.g. an upright face with a horizontal nose), current equivariant architectures consider the set of all possible transformations in a transformation group when learning feature representations. Contrarily, the human visual system is able to attend to the set of relevant transformations occurring in the environment and utilizes this information to assist and improve object recognition. Based on this observation, we modify conventional equivariant feature mappings such that they are able to attend to the set of co-occurring transformations in data and generalize this notion to act on groups consisting of multiple symmetries. We show that our proposed co-attentive equivariant neural networks consistently outperform conventional rotation equivariant and rotation & reflection equivariant neural networks on rotated MNIST and CIFAR-10.

Published as a conference paper at ICLR 2020
CO-ATTENTIVE EQUIVARIANT NEURAL NETWORKS: FOCUSING EQUIVARIANCE ON TRANSFORMATIONS CO-OCCURRING IN DATA
David W. Romero
Mark Hoogendoorn
Vrije Universiteit Amsterdam d. w. romeroguzman@-vu n1
Vrije Universiteit Amsterdam m. www.hhoddooo@@@uu m.l
ABSTRACT
Equivariance is a nice property to have as it produces much more parameter ef- ficient neural architectures and preserves the structure of the input through the feature mapping. Even though some combinations of transformations might never appear (e.g., an upright face with horizontal nose), current equivariant archi- tectures consider the set of all possible transformations in a transformation group when learning feature representations. Contrarily, the human visual system is able to attend to the set of relevant transformations occurring in the environment and utilizes this information observation, we modify conventional equivariant feature mappings such that they are able attend to the set of co-occurring transformations in data and general- ize this notion to act on groups consisting of multiple symmetries. We show that our proposed co-attentive equivariant neural networks consistently outperform conventional rotation equivariant and rotation & reflection equivariant neural net- works on rotated MNIST and CIFAR-10.
assist and improve object recognition. Based on this
INTRODUCTION
Thorough experimentation in the fields of psychology and neuroscience has provided support to the intuition that our visual perception and cogmition systems are able to identify familiar objects despite modifications in size, location, background, viewpoint and lighting (Bruce & Humphreys, 1994). Interestingly, we are not just able to recognize such modified objects, but are able to characterize which modifications have been applied to them as well. As an example, when we see picture of a cat, we are not just able to tell that there is a cat in it, but also its position, its size, facts about the lighting conditions of the picture, and so forth. Such observations suggest that the human visual system is equivariant to a large transformation group containing translation, rotation, scaling, among others. In other words, the mental representation obtained by seeing transformed version of an object, is equivalent to that of seeing the original object and transforming it mentally next.
These fascinating abilities exhibited by biological visual systems have inspired large field of re- search towards the development of neural architectures able to replicate them. Among these, the most popular and successful approach is the Convolutional Neural Network (CNN) (LeCun al., 1989), which incorporates equivariance to translation via convolution. Unfortunately, in counterpart to the human visual system, CNNs do not exhibit equivariance other transformations encountered in visual data (e.g., rotations). Interestingly, however, if an ordinary CNN happens to learn rotated copies of the same filter, the stack of feature maps becomes equivariant rotations even though individual feature maps are not (Cohen & Welling, 2016). Since ordinary CNNs must learn such rotated copies independently, they effectively utilize an important number of network parameters suboptimally to this end (see Fig. 3 in Krizhevsky et al. (2012)). Based on the idea that equivari- ance CNNs can be extended to larger transformation groups by stacking convolutional feature maps, several approaches have emerged to extend equivariance to, e.g., planar rotations (Dieleman et al., 2016; Marcos et al., 2017; Weiler et al., 2018; Li et al., 2018), spherical rotations (Cohen et al., 2018; Worrall & Brostow, 2018; Cohen et al., 2019), scaling (Marcos et al., 2018; Worrall & Welling, 2019) and general transformation groups (Cohen & Welling, 2016), such that transformed copies of single entity are not required to be learned independently.

---

Published as a conference paper at ICLR 2020
Figure I: Our visual system infers object identities according to their size, location and orientation in a scene. In this blurred picture, observers describe the scene as containing a car and a pedestrian in the street. However, the pedestrian in fact the same shape as the car, except for 90 rotation. The atypicality of this orientation for a car within the context defined by the street scene causes the car to be recognized pedestrian. Extracted from Oliva & Torralba (2007).
Although incorporating equivariance to arbitrary transformation groups is conceptually and theo- retically similar', evidence from real-world experiences motivating their integration might strongly differ. Several studies in neuroscience and psychology have shown that our visual system does not react equally to all transformations we encounter in visual data. Take, for instance, translation and rotation. Although we easily recognize objects independently of their position of appearance, large corpus of experimental research has shown that this is not always the case for in-plane rotations. Yin (1969) showed that mono-orientee objects, i.e., complex objects such as faces which are customarily seen in one orientation, are much more difficult to be accurately recognized when presented upside- down. This behaviour has been reproduced. among others, for magazine covers (Dallett et al., 1968), symbols (Henle, 1942) and even familiar faces (e.g.. from classmates) (Brooks & Goldstein, 1963). Intriguingly, Schwarzer (2000) found that this effect exacerbates with age (adults suffer from this effect much more than children), but, adults much faster and accurate in detecting moie-oriented objects in usual orientations. Based on these studies, we draw the following conclusions:
The human visual system does not perform (fully) equivariant feature transformations to visual data. Consequently, it does not react equally to all possible input transformations encountered in visual data, even if they belong to the same transformation group (e.g.. in-plane rotations).
The human visual system does just encode familiarity to objects but seems to learn through experience the poses in which these objects customarily appear in the environment to assist and improve object recognition (Freire al., 2000; Riesenhuber al., 2004; Sinha et al., 2006).
Complementary studies (Tarr& & Pinker, 1989; Oliva & Torralba, 2007) suggest that our visual system encodes orientation atypicality relative the context rather than an absolute manner (Fig- 1).
Motivated by the aforementioned observations we state the co-occurrence envelope hypothesis:
The Co-occurrence Envelope Hypothesis. By allowing equivariant feature mappings to detect transformations that co-occur in the data and focus learning on the set formed by these co-occurrent transformations (i.e., the co-occurrence envelope of the data), one able to induce learning of more representative feature representations of the data, and, resultantly, enhance the descriptive power of neural networks utilizing them. We refer to one such feature mapping as co-attentive equivariant.
Identifying the co-occurrence envelope. Consider a rotation equivariant network receiving two copies of the same face (Fig. 2a). A conventional rotation equivariant network required to perform inference and learning on the set of all possible orientations of the visual patterns constituting face regardless of the input orientation (Fig. 2b). However, by virtue of rotation equivariance, is able to recognize rotated faces even if it is trained on upright faces only. A possible strategy to simplify the task at hand could be to restrict the network to react exclusively to upright faces (Fig. 2c). In this case, the set of relevant visual pattern orientations becomes much smaller, the expense of disrupting equivariance to the rotation group. Resultantly, the network would risk becoming unable to detect faces in any other orientation than those it is trained A better strategy results from restricting the set of relevant pattern orientations by defining them relative to one another
It is achieved by developing feature mappings that utilize the transformation group in the feature mapping itself (e.g., translating filter in the course of feature transformation is used to obtain translation equivariance).
2

---

Published as a conference paper at ICLR 2020
(a)
(c)
(d)
Figure 2: Effect of multiple attention strategies for the prioritization of relevant pattern orientations in rotation equivariant networks for the task of face recognition. Given that all attention strategies are learned exclusively from upright faces, we show the set of relevant directions for the recognition of faces in two orientations (Fig. 2a) obtained by: no attention (Fig. 2b), attending to the pattern orientations of appearance independently (Fig. 2c) and, attending to the pattern orientations of appearance relative to one another (Fig. 2d). Built upon Figure 1 from Schwarzer (2000).
(e.g., mouth orientation relative the eyes) as opposed to absolutely (e.g., upright mouth) (Fig- 2d). In such a way, we are able to exploit information about orientation co-occurrences in the data without disrupting equivariance. The set of co-occurrent orientations in Fig. 2d corresponds to the co-occurrence envelope of the samples in Fig, 2a for the transformation group defined by rotations.
In this work, we introduce co-attentive equivariant feature mappings and apply them on existing equivariant neural architectures. To this end, we leverage the concept of attention (Bahdanau al., 2014) to modify existing mathematical frameworks for equivariance, such that co-occurrent trans- formations be detected. It is critical not to disrupt equivariance in the attention procedure to preserve it across the entire network. To this end, we introduce cyclic equivariant self-attention, a novel attention mechanism able preserve equivariance cyclic groups.
Experiments and results. We explore the effects of co-attentive equivariant feature mappings for single and multiple symmetry groups. Specifically, we replace conventional rotation equivariant mappings in p4-CNNs (Cohen & Welling, 2016) and DRENs (Li et al., 2018) with co-attentive ones. We show that co-attentive rotation equivariant neural networks consistently outperform their conventional counterparts in fully (rotated MNIST) and partially (CIFAR-10) rotational settings. Subsequently, we generalize cyclic equivariant self-attention to multiple similarity groups and apply it on DAm-CCNN (Cohen & Welling, 2016) (equivariant to rotation and mirror reflections). Our results are line with those obtained for single symmetry groups and support our stated hypothesis.
Contributions.
We propose the co-occurrence envelope hypothesis and demonstrate that conventional equiv- ariant mappings are consistently outperformed by our proposed co-attentive equivariant ones.
We generalize co-attentive equivariant mappings to multiple symmetry groups and provide, to the best of our knowledge, the first attention mechanism acting generally on symmetry groups
2 PRELIMINARIES
Equivariance. We say that feature mapping f X + Y is equivariant to (transformation) group G (or G-equivariant) if commutes with actions of the group G acting on its domain and codomain:
f(T (z)) IT (f(z)) Ag G
(I)
where 2 denotes a group action in the corresponding space. In other words, the ordering in which we apply group action T, and the feature mapping f is inconsequential. There are multiple reasons as of why equivariant feature representations are advantageous for learning systems. Since group actions T produce predictable and interpretable transformations TY in the feature space, the hypothesis space of the model is reduced (Weiler et al., 2018) and the learning process simplified (Worrall al., 2017). Moreover, equivariance allows the construction of T--layered networks by
3

---

Published as a conference paper at ICLR 2020
stacking several equivariant feature mappings {f(), .... f(.,.))) ff...... ((.)) such that the input structure as regarded by the group G is preserved (c.g., CNNs and imput translations). As result, an arbitrary intermediate network representation (f(() o ... (x), structure of well. Invariance is special case of equivariance in which Ty Idy, the identity, and thus all group actions in the imput space are mapped to the same feature representation.
I,, is able to take advantage of the
Equivariant neural networks. In neural networks, the integration of equivariance to arbitrary groups G has been achieved by developing feature mappings f that utilize the actions of the group G' in the feature mapping itself. Interestingly, equivariant feature mappings encode equivariance as parameter sharing with respect to G, i.e., the same weights are reused for all E G. This makes the inclusion of larger groups extremely appealing in the context of parameter efficient networks.
Conventionally, the I-th layer of neural network receives signal 2z(1) (u, X) (where E 2² is the spatial position and E A₁ is the unstructured channel index, e.g., RGB channels in a color image), and applies a feature mapping (()) 7 Ar - 72 x A1+1 to generate the feature representation (((++)((, (u, A). In CNNs, the feature mapping f()) := f1 is defined by convolution (*g²) between
the input signal x( and a learnable convolutional filter W X Ar, À E A4+1:
x(+1) A) (x xga (
W = Σ Ou uuu N)W
By sliding ( X across u, CNNs are able to preserve the spatial structure of the input x through the feature mapping ft and successfully provide equivariance to the translation group T (22,+))
The underlying idea for the extension of equivariance to larger groups in CNNs conceptually equivalent to the strategy utilized by LeCun et al. (1989) for translation equivariance. Consider, for instance, the inclusion of equivariance to the set of rotations by 0, degrees": : = {0, = = 2m Yan
Yim r==
To this end, we modify the feature mapping f() := ( 2² x x A + 22 x θ x A to include the rotations defined by O Let (et.::(((u (u, A) and w
W (u, be the input and the convolutional filter with an affixed index r for rotation. The roto-translational convolution (+R) (!l) is defined as: (x(+)) (u. A) = (xr xe W(l)) (,,
()) Σ ( r', X)W X (0, - (3)
Since 1 produces (dim(O) mmax times more output feature maps than fT need to learn much smaller convolutional filters w to produce the same number of output feature channels.
Learning equivariant neural networks. Consider the change of variables g u. G = Z², G and g = (u, r). G = Z x O E G in Eq. and Eq. 3, respectively. In general, neural networks are learned via backpropagation (LeCun al., 1989) by iteratively applying the chain rule of derivation to update the network parameters. Intuitively, the networks outlined in Eq. 2 and Eq. obtain feedback from all g E G and, resultantly, are inclined learn feature representations that perform optimally on the entire group G. However, as outlined in Fig. 2 and Section 1, several of those feature combinations are not likely to simultaneously appear. Resultantly, the hypothesis space of the model as defined by Weiler et al. (2018) might be further reduced.
Note that this reasoning tightly related existing explanations for the large success of spatial (Xu et al., 2015; Woo et al., 2018; Zhang et al., 2018) 8) and temporal (Luong et al., 2015; Vaswani et al., 2017; Mishra et al., 2017: Zhang et al., 2018) attention in deep learning architectures.
CO-ATTENTIVE EQUIVARIANT NEURAL NETWORKS
In this section we define co-attentive feature mappings and apply them in the context of equivariant neural networks (Figure 3). To this end, we introduce cyclie equivariant self-attention and utilize it to construct co-attentive rotation equivariant neural networks. Subsequently, we show that cyclic cquivariant self-attention extendable to larger symmetry groups and make use of this fact to con- struct co-attentive neural networks equivariant to rotations and mirror reflections.
Formally it is as a correlation. However, we hold on to the standard deep learning terminology. 'The reader may easily verify that θ (and hence z² θ with (x) the semi-direct product) forms group.
4

---

Published as a conference paper at ICLR 2020
A
A
Figure 3: Co-attentive equivariant feature mappings acting on the groups p4 (top) and p4ma (bottom). In order to learn co-attentive equivariant representations, cyclic equivariant self-attention Ae is applicd on top of the output of a conventional equivariant feature mapping (here p4 and p4m group convolutions, respectively). Resultantly, the group convolution responses modulated based on their assessed relevance. For multiple symmetry groups, the group convolution responses must be rearranged in vector structure so that the permutation laws of AAC correspond those of the composing group symmetries. Same colors in .A denote equal weights. The circulans (block) structure of A ensures that equivariance to the corresponding group is preserved through the course of attention. Consequently, if the input is rotated (or mirrored in p4m), the attention mask shown here is transformed accordingly. Built upon Figures and from Cohen & Welling (2016).
3.1 CO-ATTENTIVE ROTATION EQUIVARIANT NEURAL NETWORKS
To allow rotation equivariant networks to utilize and learn co-attentive equivariant representationss we introduce an attention operator A on top of the att-ttranlltional convolution f(u)) with which discernment along the rotation axis r of the generated feature responses z (u, Formally, our co-altentive rotation equivariant feature mapping p()) is defined as follows:
r, A) is possible.
110 A" - (x)))) (0 *e w( xa))
(4)
Theoretically, A could be defined globally over (1)(1(()) (i.e., simultancously along u, r. A) as depicted in Eq. 4. However, we apply attention locally to: (1) grant the algorithm enough flexibility to attend locally to the co-occurrence envelope of feature representations and, (2) utilize attention exclusively along the rotation axis r, such that our contributions are clearly separated from those possibly emerging from spatial attention. To this end, we apply attention pixel-wise on top of f/ () (Eq. 5). Furthermore, we assign a single attention instance A to each learned feature representation and utilize it across the spatial dimension of the output feature maps*:
,((+)((u,, 10 ([z (u, A))
(5)
Attention and sell-attention. Consider source vector (x ...,..... 2n)) and a target vector y (y1...., ). In general, an attention operator A leverages information from the source vector x (or multiple feature mappings thereof) to estimate an attention matrix A [0, 1]axm such that: (1) the element Ar provides an importance assessment of the source element I; reference to the target element Y/ and (2) the sum of importance over all x; equal to one: Σ, AL, 1. Subsequently, the matrix A is utilized to modulate the original source vector
to altend to a subset of relevant )T @ x (where @ is the Hadamard product). special
source positions with regard to yy: 3 = (A.,,)" case of attention is that of self-attention (Cheng et al., 2016), in which the target and the source vectors are equal (y := x². In other words, the attention mechanism estimates the influence of the sequence x the element x, for its weighting.
*For more meticulous discussion how Eq. 5 attains co-occurrent attention, see Appendix A.
5

---

Published as a conference paper at ICLR 2020
In general, the attention matrix A E [0, [0.1 is constructed via nonlinear space transforma- tions fA : R" - Raxm of the source vector z, on top of which the softmax function is applied: This ensures that the properties previously mentioned hold. Typically, the mappings SA found in literature take feature transformation pairs of x as input (e.g.. {s, H} in RNNs (Luong et al., 2015). (Q, K] in self-attention networks (Vaswani al., 2017)), and perform (non)-linear mappings on top of it, ranging from multiple feed-forward layers (Bahdanau et al., 2014) to several operations between the transformed pairs (Luong et al., 2015; Vaswani al., 2017: Mishra et al., 2017; Zhang et al., 2018). Due to the computational complexity of these approaches and the fact that we do extensive pixel-wise usage of fA on every network layer, their direct integra- tion in our framework is computationally prohibitíve. To circumvent this problem, we modify the usual self-attention formulation as to enhance its descriptive power in a much more compact setting.
A =ofttmx(ff<()).))
Compact local self-attention. Initially, we relax the range of values of A from [0, 1| to R This allows us to encode much richer relationships between element pairs (z) at the cost of less interpretability. Subsequently, we define A = x² @ Å, where À E R is a matrix of learnable parameters. Furthermore, instead of directly applying softmax on the columns of A, we first sum over the contributions of each element 2, to obtain a vector a = {Ei P which then passed to the softmax function. Following Vaswani et al. (2017), we prevent the softmax function from reaching regions of low gradient by scaling its argument by ( (v dim(AA) ( = (1 / n): à soft (1/ / n.) a). Lastly, we counteract the contractive behaviour of the softmax function by normalizing á before weighting x as to preserve the magnitude range of its argument. This allows us to use A in deep architectures. Our compuct self-atlention mechunism is summarized as follows:
a = Ac, = Ex" ① A)1, = xA
(6)
à = softmax((11 / n) a) a A(z) (a mmax(a)) Ox
(7)
(8)
r, A)) the vector of re- The cyclic equivariant self-attention operator AC. Consider [u(u, sponses generated by a roto-translational convolution Sn stacked along the rotation axis r. By applying self-attention along r,, we are able to generate an importance matrix E RTrax XTa relat- ing all pairs of (f), 9, )-otated responses in the rotational group Q at certain position. We refer to this attention mechanism as full self-attention (A"). Although AT is able to encode arbitrary linear target relationships for each target position, it is not restricted to conserve equivariance to O. Resultantly, we risk incurring into the behavior outlined in Fig. 2c. Before we further elaborate on this issue, we introduce the cyclic permutation operator Pi, which induces a cyclic shift of i positions on its argument: o² (3r))= = ((dddm)))( Vaz E x.
Consider a full self-attention operator A acting on top of a roto-translational convolution Sa- Let p be an input pattern to which fR only produces a strong activation in the feature map z(f) fF(P)(F), f E {r {r) Intuitively, during learning, only the corresponding attention coefficients A. in A would be significantly increased. Now, consider the presence of the input pattern 0,p, 0.-rotated variant of p. By virtue of the rotational equivariance property of the feature mapping fR. obtain (locally) an exactly equal response to that of p up to cyclic permutation of i positions on r, and thus, we obtain strong activation in the feature map Pi (x((()))= = 2(a²' (f)). We encounter two problems in this setting: AA is not be able to detect that p and 0&p correspond to the exact same input pattern and, as each but the attention coefficients À. is small, the network might considerably damp the response generated by 0.p. As a result, the network might (1) squander important feedback information during learning and (2) induce learning of repeated versions of the same pattern for different orientations. In other words, A² does not behave equivariantly as function of 0,-
Interestingly. we are able to introduce prior-knowledge into the attention model by restricting the structure of A. By leveraging the idea of equivariance to the cyclic group C,, we are able to solve the problems exhibited by A and simultaneously reduce the number of additional parameters required by the self-attention mechanism (from 2 to mmmx). Consider again the input patterns p and Osp. We incorporate the intuition that p and 6Gp are one and the same entity, and thus, fR (locally) generates pt (fR(P)). Consequently,
the same output feature map up to cyclic permutation pt: fR (0.p)) the attention mechanism should produce the exact same output for both and 6.p up to the same cyclic permutation Pt. In other words, A (and thus Á) should be equivariant to cyclic permutations.
Technically, each column of A is restricted a simplex and hence lives a subspace of [0. 1) mm

---

Published as a conference paper at ICLR 2020
A well-known fact in mathematics is that matrix A is equivariant with respect to cyclic permuta- tions of the domain if and only if it is circulant (Alkarni, 2001; Ahlander & Munthe- Kaahe 2005). We make use of this certitude and leverage the concept of circulant matrices to impose cyclic equiv- ariance to the structure of Å. Formally, circulant matrix C E R=x* is composed of eyclic permutations of its defining vector = {<>) such that its j-th column is a cyclic permutation of
j - positions of c: C.,- = (c)" We construct our cyclic equivariant self-attention operator A by defining À as a circulant matrix specified by a learnable attention vector d = ={a JE
À A= (p) ))²)))
and subsequently applying Eqs. 6 8. Resultantly, .AC is able to assign the responses generated by fR for rotated versions of an input pattern p to a unique entity: fR((.p) = P' (fR())), and dy- namically adjust its output to the angle of appearance 0., such that the attention operation does not disrupt its propagation downstream the network: A (fr((.p)) = (f)()) Consequently, the attention weights at are updated equally regardless of specific values of 0g Due to these prop-
erties, A does not incur in any of the problems outlined earlier in this section. Conclusively, our co-attentive rotation equivariant feature mapping f is defined as follows:
11+11) (u, r, A) ( (z) (u. A) = A( (Ix!))
(1)
* W (,,),,
(10)
Note that a co-attentive equivariant feature mapping SR. is approximately equal (up to a normalized softmax operation (Eq. 8)) to a conventional equivariant one fn. if Á aT for any a E TR.
3.2 EXTENDING A To MULTIPLE SYMMETRY GROUPS
The self-attention mechanisms outlined in the previous section are easily extendable to larger groups consisting of multiple symmetrics. Consider, for instance, the group 8. of rotations by 8, degrees and mirror reflections defined analogously to the group påm in Cohen & Welling (2016). Let p(u, r, m, A) be an input signal with an affixed index m E {Ma. mı} for mirror reflections (m1 indicates mirrored) and fe.m be a group convolution (Cohen & Welling, 2016) on the 6, group. The group convolution fa.m produces two times as many output channels (2rmax Mo + (maaa) as those generated by the roto-translational convolution JR (Eq. 3, Fig. 3).
Full self-attention AF can be integrated directly by modulating the output of Jo.m as depicted in Sec. 3.1 with A E 12 X2rms, Here, AA relates each of the group convolution responses with one another. However, just as for fR. A= disrupts the equivariance property of fe.m to the f, group.
Similarly, the cyclic equivariant self-attention operator AC can be extended to multiple symmetry groups as well. Before we continue, we introduce the cyclic permutation operator pit which induces a cyclic shift of i positions on its argument along the transformation axis t. Consider the input patterns p and 0.p outlined in the previous section and mp, a mirrored instance of p. Let (u,, r. m, X) fe.m (p)m()((u, (u, r, m, A) be the response of the group convolution fe.m for the input pattern p. By virtue of the rotation equivariance property of fe.m. the generated re- sponse for Opp is equivalent to that of p up to cyclic permutation of positions along the ro- I(u, o (r), m, A). Sim-
tation axis r: fo.m (O,p)(u,,, m, A) ilarly, by virtue of the mirror equivariance property of fa.m. the response generated by mp is equivalent to that of p up to a cyclic permutation of one position along the mirroring axis m: mm. (u, r, m, A) = pl. pl.""(ff.m (p).((p)))u, r, m, = æ(u, r, aPt (m), A). Note that if we take two elements from a group g. h, their composition (gh) is also element of the group. Resultantly, fo.m m(mm")) 1p) (u, r, m, >) = (pl.m o (for) (P))((())))))))))))))))))) r, m, = pl." (r), (r),PP' (m),
pir (fe.. (P)) r, m,
r, m, A)
pI. (l. p (r), A) x(u, o pop
In other words, in order to extend AC to the 0,m group, necessary to restrict the structure of À such that it respects the permutation laws imposed by the equivariant mapping fa,m- Let us rewrite z(u, r, m. X) as x(u,g) 9, X), (mr) E {mo. m } x (f) In this case, we must impose circulant block matrix structure on Á such that: (1) the composing blocks permute internally as defined by pir and (2) the blocks themselves permute with one another as defined by pl.m. Formally, À is defined as:
A²
À
(11)
A
Á
7

---

Published as a conference paper at ICLR 2020
where {A, R X ), E (1,2) are circulant matrices (Eq. 9). Importantly, the ordering of the permutation laws in À is interchangeable if the input vector is modified accordingly, i.e., 9 (rm)-
Conclusively, cyclic equivariant self-attention A is directly extendable to act on any G-equivariant feature mapping fa. and for any symmetry group G, if the group actions Tу produce eyclic permu- tations on the codomain of fG. To this end, one must restrict the structure of Ā to that of a circulant block matrix, such that all permutation laws of T hold: T5 (Ac()) = (TT (fa)), Vg e G.
4 EXPERIMENTS
Experimental Setup. We validate our approach by exploring the effects of co-attentive equivari- ant feature mappings for single and multiple symmetry groups existing equivariant architectures. Specifically, we replace conventional rotation equivariant mappings in p4-CNNs (Cohen Welling, 2016) and DRENs (Li et al., 2018) with co-attentive equivariant ones and evaluate their effects in fully (rotated MNIST) and partially (CIFAR-10) rotational settings. Similarly, we evaluate co- attentive equivariant maps acting on multiple similarity groups by replacing equivariant mappings in D4m-CNNs (Cohen & Welling, 2016) (equivariant to rotation and mirror reflections) likewise. Unless otherwise specified, we replicate as close as possible the same data processing, initialization strategies, hyperparameter values and evaluation strategies utilized by the baselines in our exper- iments. Note that the goal of this paper is study and evaluate the relative effects obtained by co-attentive equivariant networks with regard to their conventional counterparts. Accordingly, we do not perform any additional tuning relative to the baselines. We believe that improvements on our reported results are feasible by performing further parameter tuning (e.g.. on the network structure or the used hyperparameters) on the proposed co-attentive equivariant networks.
The additional learnable parameters, i.e., those associated to the cyclic self-attention operator (A) are initialized identically to the rest of the layer. Subsequently, we replace the values of Á along the diagonal by 1 (i.e., (iii(A 1) such that Ami approximately resembles the identity I and, hence, co-attentive equivariant layers are initially approximately equal to equivariant ones.
Rotated MNIST. The rotated MNIST dataset (Larochelle et al., 2007) contains 62000 gray-scale 28x28 handwritten digits uniformly rotated on the entire circle [0, 2x). The dataset is split into training. validation and tests sets of 10000, 2000 and 50000 samples, respectively. We replace rotation equivariant layers in p4-CNN (Cohen & Welling, 2016), DREN and DRENMaxPooling (Li et al., 2018) with co-attentive ones. Our results show that co-attentive equivariant networks consistently outperform conventional ones (see Table 1).
CIFAR-10. The CIFAR-IO 10 dataset (Krizhevsky et al., 2009) consists of 60000 real-world 32x32 RGB images uniformly drawn from 10 classes. Contrarily to the rotated MNIST dataset, this dataset does not exhibit rotation symmetry. The dataset is split into training, validation and tests sets of 40000, 10000 and 10000 samples, respectively. We replace equivariant layers in the p4 and p4m variations of the All-CNN (Springenberg et al., 2014) and the ResNet44 (He et al., 2016) proposed by Cohen & Welling (2016) with co-attentive ones. Likewise, we modify the r_x4-varaaiions of the NIN (Lin et al., 2013) and ResNet20 (He et al., 2016) models proposed by Li et al. (2018) in the same manner. Our results show that co-attentive equivariant networks consistently outperform conventional ones in this setting as well (see Table 1).
Training convergence of equivariant networks. Li et al. (2018) reported that adding too many ro- tational equivariant (isotonic) layers decreased the performance of their models on CIFAR-10. As consequence, they did not report results on fully rotational equivariant networks for this setting and attributed this behaviour to the non-symmetricity of the data. We noticed that, with equal initial- ization strategies, rotational equivariant CNNs were much more prone to divergence than ordinary CNNs. This behaviour can be traced back to the additional feedback resulting from roto-transe noto-translational convolutions (Eq. 3) compared to ordinary ones (Eq. 2). After further analysis, we noticed that the data preprocessing strategy utilized by Li et al. (2018) leaves some very large outlier values in the data (|0l > 100), which strongly contribute to the behaviour outlined before.
In order evaluate the relative contribution of co-attentive equivariant neural networks con- structed fully equivariant DREN architectures based on their implementation. Although the obtained results were much worse than those originally reported in Li et al. (2018), were able stabilize
8

---

Published as a conference paper at ICLR 2020
Table 1: Comparison of conventional equivariant and co-attentive equivariant neural networks.
Values between parenthesis correspond to relevant results obtained from our own experiments.
Rotated MNIST
CIFAR-10
Network
Test Error (%)
Param.
Network
Test Error (%)
Param.
Z2CNN
5.03 1 0.002
21.75k
AlI-CNN
9.44
1.372M
p4-CNN
2.28 + 0.0004
19.88k
P4-AII-CNN
8.84
1.371M
a-p4-CNN
2.06 + 0.0429
20.76k
a-p4-All-CNN
7.68
1.373M
DREN
1.78 (1.99)
19.88k
p4m-All-CNN
7.59
1.219M
o-DREN
1.674
20.76k
d-pAm-AlI-CNN
6.92
1.223M
PRENM.xPool.
1.56 (1.60)
24.68k
ResNer44
9,45 (9.85)
2.639M
a- o-DRENMaxPool.
1.34
25.68k
3Am-ResNet44
6,46 (9,47)
2.623M
4esee44
9.12
2.632M
NIN
10.41 (15.92)
0,967M
r-NINx4
14.96
0,958M
a-r-NINx4
13.67
0.968M
ResNet20
9.00 (12.32)
0.335M
Nee25e120
12.31
0.333M
a-r- ResNeeeet2
11.32
0.339M
training by clipping input values to the 99 percentile of the data (|x) <2.3) and reducing the learning rate to 0.01, such that the same hyperparameters could be used across all network types. The ob- tained results (see Table 1) signalize that DREN networks comparatively better than CNNs both in fully and partially rotational settings, contradictorily to the conclusions drawn in Li et al. (2018).
This behaviour elucidates that although the inclusion of equivariance to larger transformation groups is beneficial both in terms of accuracy and parameter efficiency, one must be aware that such benefits are directly associated with an increase of the network susceptibility divergence during training.
DISCUSSION AND FUTURE WORK
Our results show that co-attentive equivariant feature mappings can be utilized enhance conven- tional equivariant ones. Interestingly, co-attentive equivariant mappings are beneficial both in par- tially and fully rotational settings. We attribute this to the fact that set of co-occurring orientations between patterns can be easily defined (and exploited) in both settings. It is important to note that we utilized attention independently over each spatial position u on the codomain of the correspond- ing group convolution. Resultantly, we were restricted to mappings of the form AAA which, in turn, constraint our attention mechanism to have a circulant structure order to preserve equivariance (since group actions acting in the codomain of the group convolution involve cyclic permutations and cyclic self-attention is applied in the codomain of the group convolution).
In future work, we want to extend the idea presented here to act on the entire group simultancously (i.e., along u as well). By doing so, we lift our current restriction to mappings of the form æA and therefore, may be able to develop attention instances with enhanced descriptive power. Following the same line of though, we want to explore incorporating attention in the convolution operation itself. Resultantly, one is not restricted act exclusively instead, able to impose structure in the domain of the mapping as well. Naturally, such an ap- proach could lead enhanced descriptiveness of the incorporated attention mechanism. Moreover, we want to utilize and extend more complex attention strategies (c.g., Bahdanau et al. (2014); Lu- ong et al. (2015); Vaswani al. (2017); Mishra et al. (2017)) such that they can be applied to large transformation groups without disrupting equivariance. As outlined earlier in Section 3.1, this be- comes very challenging from a computational perspective as well, as requires extensive usage of the corresponding attention mechanism. Resultantly, an efficient implementation thereof is manda- tory. Furthermore, we want to extend co-attentive equivariant feature mappings to continuous (e.g.. Worrall et al. (2017)) and 3D space (e.g.. Cohen et al. (2018); Worrall & Brostow (2018); Cohen et al. (2019)) groups, and for applications other than visual data (c.g., speech recognition).
the codomain of the convolution, but

---

Published as a conference paper at ICLR 2020
Finally, we believe that our approach could be refined and extended to first step towards deal- ing with the enumeration problem of large groups (Gens & Domingos, 2014), such that functions acting the group (e.g.. group convolution) are approximated by evaluating them on the set of co- occurring transformations as opposed to on the entire group. Such approximations are expected to be very accurate, non-co-occurrent transformations rare. This could be though of as sharping up co-occurrent attention to co-occurrent restriction.
CONCLUSION
We have introduced the concept of co-attentive equivariant feature mapping and applied it in the context of equivariant neural networks. By attending to the co-occurrence envelope of the data, we are able to improve the performance of conventional equivariant ones on fully (rotated MNIST) and partially (CIFAR-10) rotational settings. We developed cyclic equivariant self-attention, an attention mechanism able to attend to the co-occurrence envelope of the data without disrupting equivariance large set of transformation groups (i.e., all transformation groups G, whose action in the codomain of G-equivariant feature mapping produce cyclic permutations). Our obtained results support the proposed co-occurrence envelope hypothesis.
ACKNOWLEDGMENTS
We gratefully acknowledge Jan Klein, Emile van Krieken, Jakub Tomezak and anonymous reviewers for their helpful and valuable commentaries. This work is part of the Efficient Deep Learning (EDL) programme (grant number P16-25), which is partly founded by the Dutch Research Council (NWO) and Semiotic Labs.
REFERENCES
Krister Åhlander and Hans Munthe-Kast Applications of the generalized fourier transform in nu- merical linear algebra. BIT Numerical Mathematics, 45(4))819-88) 850, 2005.
SH Alkarni. Statistical applications for equivariant matrices. International Journal of Mathematics and Mathematical Sciences, 25(1):53-66) 61, 2001.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0477 2014.
Richard M Brooks and Alvin G Goldstein. Recognition by children of inverted photos of faces. Child Development, 1963.
Vicki Bruce and Glyn W Humphreys. Recognizing objects and faces. Visual cognition, 1(2-3): 141-180, 1994.
Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv: 1601, 06733,, 2016.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-29999 100002299,000 2016.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical cnns. abs/1801.10030, 2018 URL http: org/ abs/ 18011 1111111110000 10130.
CoRR,
Taco S Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolu- tional networks and the icosahedral cnn. arXiv preprint arXiv:1902.04615. 2019.
Kent Dallett, Sandra G Wilcox, and Lester D'andrea. Picture memory experiments. Journal of Experimental Psychology, 76(2p1):312, 1968.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convo- 2016.
lutional neural networks. arXiv preprint
Alejo Freire, Kang Lee, and Lawrence A Symons. The face-inversion effect as a deficit in the encoding of configural information: Direct evidence. Perception, 199-):15--170. 2000.
10

---

Published as a conference paper at ICLR 2020
Robert Gens and Pedro M Domingos. Deep symmetry networks. In Advances in neural information processing systems, pp. 2537-2545, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Mary Henle. An experimental investigation of past experience as a determinant of visual form perception. Journal of Experimental Psychology, 30(1):1, 1942.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empir- ical evaluation of deep architectures problems with many factors of variation. In Proceedings of the 24th international conference on Machine learning, pp. 4734444400000 ACM, 2007.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Junying Li, Zichen Yang, Haifeng Liu, and Deng Cai. Deep rotation equivariant network. Neuro- computing, 290:26-33 33, 2018.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. arxiv preprint 1rXivv/508.000 04025, 2015.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5048- 5057, 2017.
Diego Marcos, Benjamin Kellenberger, Sylvain Lobry, and Devis Tuia. Scale equivariance in enns with vector fields. arXiv preprint 1807.11788 2018.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbeel. A simple neural attentive meta- learner. arXiv preprint 17Xiv:170..03141, 2017.
Aude Oliva and Antonio Torralba. The role of context in object recognition. Trendy in cognitive sciences, 11(12)5220-221 2007.
Maximilian Riesenhuber, Izzat Jarudi, Sharon Gilad, and Pawan Sinha. Face processing in humans is compatible with a simple shspe model of vision. Proceedings of the Royal Society of London. Series B: Biological Sciences, 271 (suppl 6):$448-S455 2004.
Gudrun Schwarzer. Development of face processing: The effect of face inversion. Child develop- ment, 71(2):391-401, 2000.
Pawan Sinha, Benjamin Balas, Yuri Ostrovsky, and Richard Russell. Face recognition by humans: Nineteen results all computer vision researchers should know about. Proceedings of the IEEF, 94 (11):1948-1962. 2006.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6800 2014.
Michael Tarr and Steven Pinker. Mental rotation and diieetatiiondependdenn in shape recognition. Cognitive psychology. 21(2):233-28) 1989.
11

---

Published as a conference paper at ICLR 2020
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp.
2017.
Maurice Weiler, Fred Hamprecht, and Martin Storath. Learning steerable filters for rotation equiv- ariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849 858, 2018.
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 3-19, 2018.
Daniel Worrall and Gabriel Brostow. Cubenet: Equivariance to 3d rotation and translation. In
Proceedings of the European Conference on Computer Vision (ECCV), pp. 567-584, 2018.
Daniel E Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. arXiv preprint
arXiv: 11697, 2019.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2048-2057, 2015.
Robert K Yin. Looking at upside-down faces. Journal of experimental psycholoyy, 81(1):141, 1969.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv: 2018.
12

---

Published as a conference paper at ICLR 2020
A OBTAINING CO-OCCURRENT ATTENTION VIA EQUATION 5
Mashh
A
fn
Figure 4: Synchronous movement of feature mappings and attention masks function of input rotation in the group p4 (Fmax = 4).
In this section, we provide a meticulous description on how co-occurrent attention is obtained via the method presented in the paper. Intuitively, direct approach to address the problem illustrated in the introduction (Section 1) and Figure requires an attention mechanism that acts simultaneously on and À (see Eq. 3). However, illustrate how the depicted problem be simplified such that attention along r is sufficient by taking advantage of the equivariance property the network.
Let p be the input of ooto-translational convolution fR 2 θ Ag + 2² θ A₁ defined in Eq. 3, and θ be the set of rotations by 0, degrees: θ = [0, = r 2x Vaux J Let (n( E TR" XA1
be the matrix consisting of the Tmax oriented responses for each À E A1 learned representation certain position result of the rotation equviariance property of J'. is mandatory to ensure equivariance to cyclic permutations for each (R(P) (u, A) during the course of the attention procedure (see Section 3).
Since the vectors f'r(p) A) R À A permute cyclically
At first sight, one is inclined to think that there is no connection between multiple vectors fR(P)(u, X) in (R)) and, therefore, in order exploit co-occurences, one tional constraints along the axis. However, there is indeed an implicit restriction in fR(P)()) along resulting from the rotation equivariance property of the mapping fR. which we can take advantage from to simplify the problem at hand. Consider, for instance, the input 0pp, 0--rotated version of p. By virtue of the equivariance property of fn, we have (locally) that ['z(O.p) = ('()r())) Furthermore, we know that this property must hold for all the learned feature representations fR())(u, A), e Aq. Resultantly, we have that:
impose addi-
fn(O,p)(u, r, A) ('(f(()((u,,,))) A
(12)
In other words, if one of the learned mappings (u, experiences a permutation Pi along r, all the learned representations fn(()( r, A), Aλ E A₁ must experience the exact same permutation p as well. Resultantly, the equivariance property of the mapping fn ensures that all the A learned feature representations fR( X) "move synchronously" as a function input rotation 0,.
Likewise, if we apply a cyclic equivariant attention mechanism A independently top of each À learned representation (", A), we obtain that the relation
(f) = ""A(
AA
(13)
must hold as well. Similarly to the case illustrated in Eq. 12 and given that A is equivariant to cyclic permutations on the domain, we obtain that all the A₁ learned attention masks Aç "move synchronously"
function of input rotation 0g as well (see Fig. 4).
From Eq. 13 and Figure 4, one can clearly see that by utilizing Af independently along and taking advantage from the fact that all A₃ learned feature representations are tied with one another via fn- one is able to prioritize learning of feature representations that co-occur together as opposed to the much looser formulation in Eq. 12, where feedback is obtained from all orientations.
13


Reviews:
[{"review_id": "ryguPqudYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Post-rebuttal update]\n\nHaving read the rebuttals and seen the new draft, the authors have answered a lot of my concerns. I am still unsatisfied about the experimental contribution, but I guess producing a paper full of theory and good experiments is a tall ask. Having also read through the concerns of the other reviews and the rebuttal to them, I have decided to upgrade my review to a 6.\n\n*Paper summary*\n\nThe paper combines attention with group equivariance, specifically looking at the p4m group of rotations, translations, and flips. The basic premise is to use a group equivariant CNN of, say, Cohen and Welling (2016), and use self-attention on top. The authors derive a form of self-attention that does not destroy the equivariance property.\n\n\n*Paper decision*\n\nI have decided that the paper be given a weak reject. The method seems sound and I think this in itself is a great achievement., But the experiments lack focus. Just showing that you get better accuracy results does not actually test why attention helps in an equivariant setting. That said, I feel the lack of clarity in the writing is actually the main drawback. The maths is poorly explained and the technical jargon is quite confusing. I think this can be improved in a camera-ready version or in submission to a later conference, should overall acceptance not be met.\n\n\n*Supporting arguments*\n\nI enjoyed the motivation and discussion on equivariance from a neuroscientific perspective. This is something I have not seen much of in the recent literature (which is more mathematical in nature) and serves as a refreshing take on the matter. There was a good review of the neuroscientific literature and I felt that the conclusions, which were draw (of approximate equivariance, and learned canonical transformations) were well motivated by these paper.\n\nThe paper is well structured. That said, I found the clarity of the technical language at times quite difficult to follow because terms were not defined. By way of example, I still have trouble understanding terms like “co-occurence” or “dynamically learn”. In the co-occurence envelope hypothesis, for instance, what does it mean for a learned feature representation to be “optimal in the set of transformations that co-occur”. Against what metric exactly would a representation be optimal? This is not defined.\n\nThat said, I feel that the content and conclusions of the paper are technically sound, having followed the maths, because the text was too confusing.\n\n\n*Questions/notes for the authors*\n\n- I would like to know whether the co-occurence envelope hypothesis is the authors’ own contribution. This was not apparent to me from the text.\n- I’m not sure what exactly the co-occurence envelope is. It does not seem to be defined very precisely. What is it in layman’s terms?\n- I found the section “Identifying the co-occurence envelope” very confusing. I’m not sure what the authors are trying to explain here. Is it that a good feature representation of a face would use the *relevant* offsets/rotations/etc. of visual features from different parts of the face, independent of global rotation?\n- Is Figure 1 supposed to be blurry?\n- At the end of paragraph 1 you have written: sdfgsdfg asdfasdf. Please delete this.\n- I believe equation 4 is a roto-translational convolution since it is equivariant to rotation *and translation*. Furthermore, it is not exactly equivariant due to the fact that you are defining input on a 2D square grid, but that is a minor detail in the context of this work.\n- Now that we have automatic differentiation, is the section on how to work out the gradients in Equations 5-7 really necessary?\n- In equation 8 (second equality), you have said f_R^l(F^l) = A(f_R^l(F^l)). How can this be true if A is not the identity? Giving the benefit of the doubt, this could just be a typo.\n- Please define \\odot (I think it’s element-wise multiplication).\n- Are you using row-vector convention? That would resolve some of my confusion with the maths.\n- You define the matrix A as in the space [0,1]^{n x m}. While sort of true, it is more precise to note that each column is actually restricted to a simplex, so A lives in a subspace of [0,1]^{n x m}.\n- I think it would have been easier just say that you are using a roto-translation or p4m equivariant CNN with attention after each convolution. Then you could derive the constraint on the attention matrices to maintain equivariance. It would be easier to follow and make easy connections with existing literature on the topic.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eJfdke5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an approach to applying attention in equivariant image classification CNNs so that the same transformation (rotation+mirroring) is selected for each kernel. For example, if the image is of an upright face, the upright eyes will be selected along with the upright nose, as opposed to allowing the rotation of each to be independent. Applying this approach to several different models on rotated MNIST and CIFAR-10 lead to smaller test errors in all cases.\n\nOverall, this is a good idea that appears to be well implemented and well evaluated.  It includes an extensive and detailed bibliography of relevant work. The approach seems to be widely applicable. It could be applied to any deep learning-based image classification system. It can be applied to additional transformations beyond rotation and mirroring.\n\nThe one shortcoming of the paper is that it takes a simple idea and makes it somewhat difficult to follow through cumbersome notation and over-mathmaticization. The ideas presented would be much clearer as an algorithm or more code-like representation as opposed to as equations. Even verbal descriptions could suffice. The paper is also relatively long, going onto the 10th page. In order to save space, some of the mathematical exposition can be condensed.\n\nIn addition, as another issue with clarity, the algorithm has one main additional hyperparameter, r_max, but the description of the experiments does not appear to mention the value of this hyperparameter. It also states that the rotated MNIST dataset is rotated on the entire circle, but not how many fractions of the circle are allowed, which is equivalent to r_max.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bygx8wipFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Update after rebuttal period]\n\nWhile I still find the paper somewhat hard to parse, the revision and responses have addressed most of my concerns. I think this paper should be accepted, because it presents a novel and non-trivial concept (rotation-equivariant self attention).\n\n\n[Original review]\n\nThe authors propose a self-attention mechanism for rotation-equivariant neural nets. They show that introduction of this attention mechanisms improves classification performance over regular rotation-equivariant nets on a fully rotational dataset (rotated MNIST) and a regular non-rotational dataset (CIFAR-10).\n\nStrengths:\n+ States a clear hypothesis that is well motivated by Figs. 1 & 2\n+ Appears to accomplish what it claims as contributions\n+ Demonstrates a rotation-equivariant attention mechanism\n+ Shows that its introduction improves performance on some tasks\n\nWeaknesses:\n- Unclear how the proposed attention mechanism accomplishes the goal outlined in Fig. 2d\n- Performance of the authors' evaluations of the baselines is lower than reported in the original papers, casting some doubt on the performance evaluation\n- The notation is somewhat confusing and cumbersome, making it hard to understand what exactly the authors are doing\n- No visualisation or insights into the attention mechanism are provided\n\nThere are three main issues detailed below that I'd like to see addressed in the authors' response and/or a revised version of the paper. If the authors can address these concerns, I am willing to increase my score.\n\n1. The motivation for the attention mechanism (as discussed in the introduction and illustrated in Fig. 2) seems to be to find patterns of features which commonly get activated together (or often co-occur in the training set). However, according to Eq. (9), attention is applied separately to orientations of the same feature ($A_i$ is indexed by i, the channel dimension), and not across different features. Since the attention is applied at each spatial location separately, such mechanism only allows to detect patterns of relative orientations of the same feature appearing at the same spatial location. The motivation and utility of such formulation is unclear, as it appears to be unable to solve the toy problem laid out in Fig. 2. Please clarify how the proposed mechanism would solve the toy example in Fig. 2.\n\n2. The only real argument that the proposed mechanism is useful are the numbers in Table 1. However, the experimental results for CIFAR-10 are hard to compare to the baselines because of differences in reported and reproduced results. I would appreciate a clarification about the code used (was it published by the authors of other papers?) and discussion of why the relative improvement achieved by the proposed method is not an artefact of implementation or optimisation issues. \n\n3. The exposition and notation in section 3.1 is very hard to follow and requires substantial improvement. For instance, the sections \"Attention and self attention\" and \"Compact local self attention\" seem to abstract from the specific case and use x and y, but it is unclear to me what x and y map to specifically. Maybe also provide a visualization of how exactly attention is applied.\n\n\nMinor comments/questions:\n\n- If the attention is applied over the orientations of the same feature, why does it improve the performance on Rotated MNIST (which is rotation invariant)?\n\n- I assume the attention matrix $A_i$ is different for each layer, because the features in different layers are different and require different attention mechanisms. However, unlike F and K, A is not indexed by layer l.\n\n- It would be good to provide the standard deviation for the reported results on CIFAR-10 to see if the improvement is significant.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an attention mechanism for equivariant neural networks towards the goal of attending to co-occurring features. It instantiates the approach with rotation and reflection transformations, and reports results on rotated MNIST and CIFAR-10. All reviewers have found the idea of using self-attention on top of equivariant feature maps technically novel and sound. There were some concerns about readability which the authors should try to address in the final version. 
Decision: Accept (Poster)