user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring in Data

Paper content:
Equivariance is a nice property to have as it produces much more parameter efficient neural architectures and preserves the structure of the input through the feature mapping.
Even though some combinations of transformations might never appear (e.g. an upright face with a horizontal nose), current equivariant architectures consider the set of all possible transformations in a transformation group when learning feature representations.
Contrarily, the human visual system is able to attend to the set of relevant transformations occurring in the environment and utilizes this information to assist and improve object recognition.
Based on this observation, we modify conventional equivariant feature mappings such that they are able to attend to the set of co-occurring transformations in data and generalize this notion to act on groups consisting of multiple symmetries.
We show that our proposed co-attentive equivariant neural networks consistently outperform conventional rotation equivariant and rotation & reflection equivariant neural networks on rotated MNIST and CIFAR-10.
Thorough experimentation in the fields of psychology and neuroscience has provided support to the intuition that our visual perception and cognition systems are able to identify familiar objects despite modifications in size, location, background, viewpoint and lighting (Bruce & Humphreys, 1994) .
Interestingly, we are not just able to recognize such modified objects, but are able to characterize which modifications have been applied to them as well.
As an example, when we see a picture of a cat, we are not just able to tell that there is a cat in it, but also its position, its size, facts about the lighting conditions of the picture, and so forth.
Such observations suggest that the human visual system is equivariant to a large transformation group containing translation, rotation, scaling, among others.
In other words, the mental representation obtained by seeing a transformed version of an object, is equivalent to that of seeing the original object and transforming it mentally next.
These fascinating abilities exhibited by biological visual systems have inspired a large field of research towards the development of neural architectures able to replicate them.
Among these, the most popular and successful approach is the Convolutional Neural Network (CNN) (LeCun et al., 1989) , which incorporates equivariance to translation via convolution.
Unfortunately, in counterpart to the human visual system, CNNs do not exhibit equivariance to other transformations encountered in visual data (e.g. rotations).
Interestingly, however, if an ordinary CNN happens to learn rotated copies of the same filter, the stack of feature maps becomes equivariant to rotations even though individual feature maps are not (Cohen & Welling, 2016) .
Since ordinary CNNs must learn such rotated copies independently, they effectively utilize an important number of network parameters suboptimally to this end (see Fig. 3 in Krizhevsky et al. (2012) ).
Based on the idea that equivariance in CNNs can be extended to larger transformation groups by stacking convolutional feature maps, several approaches have emerged to extend equivariance to, e.g. planar rotations (Dieleman et al., 2016; Marcos et al., 2017; Weiler et al., 2018; Li et al., 2018) , spherical rotations (Cohen et al., 2018; Worrall & Brostow, 2018; Cohen et al., 2019) , scaling (Marcos et al., 2018; Worrall & Welling, 2019) and general transformation groups (Cohen & Welling, 2016) , such that transformed copies of a single entity are not required to be learned independently.
Figure 1: Our visual system infers object identities according to their size, location and orientation in a scene.
In this blurred picture, observers describe the scene as containing a car and a pedestrian in the street.
However, the pedestrian is in fact the same shape as the car, except for a 90
• rotation.
The atypicality of this orientation for a car within the context defined by the street scene causes the car to be recognized as a pedestrian.
Extracted from Oliva & Torralba (2007) .
Although incorporating equivariance to arbitrary transformation groups is conceptually and theoretically similar 1 , evidence from real-world experiences motivating their integration might strongly differ.
Several studies in neuroscience and psychology have shown that our visual system does not react equally to all transformations we encounter in visual data.
Take, for instance, translation and rotation.
Although we easily recognize objects independently of their position of appearance, a large corpus of experimental research has shown that this is not always the case for in-plane rotations.
Yin (1969) showed that mono-oriented objects, i.e. complex objects such as faces which are customarily seen in one orientation, are much more difficult to be accurately recognized when presented upsidedown.
This behaviour has been reproduced, among others, for magazine covers (Dallett et al., 1968) , symbols (Henle, 1942) and even familiar faces (e.g. from classmates) (Brooks & Goldstein, 1963) .
Intriguingly, Schwarzer (2000) found that this effect exacerbates with age (adults suffer from this effect much more than children), but, adults are much faster and accurate in detecting mono-oriented objects in usual orientations.
Based on these studies, we draw the following conclusions:
• The human visual system does not perform (fully) equivariant feature transformations to visual data.
Consequently, it does not react equally to all possible input transformations encountered in visual data, even if they belong to the same transformation group (e.g. in-plane rotations).
• The human visual system does not just encode familiarity to objects but seems to learn through experience the poses in which these objects customarily appear in the environment to assist and improve object recognition (Freire et al., 2000; Riesenhuber et al., 2004; Sinha et al., 2006) .
Complementary studies (Tarr & Pinker, 1989; Oliva & Torralba, 2007) suggest that our visual system encodes orientation atypicality relative to the context rather than on an absolute manner (Fig. 1) .
Motivated by the aforementioned observations we state the co-occurrence envelope hypothesis:
The Co-occurrence Envelope Hypothesis.
By allowing equivariant feature mappings to detect transformations that co-occur in the data and focus learning on the set formed by these co-occurrent transformations (i.e. the co-occurrence envelope of the data), one is able to induce learning of more representative feature representations of the data, and, resultantly, enhance the descriptive power of neural networks utilizing them.
We refer to one such feature mapping as co-attentive equivariant.
Identifying the co-occurrence envelope.
Consider a rotation equivariant network receiving two copies of the same face (Fig. 2a) .
A conventional rotation equivariant network is required to perform inference and learning on the set of all possible orientations of the visual patterns constituting a face regardless of the input orientation (Fig. 2b) .
However, by virtue of its rotation equivariance, it is able to recognize rotated faces even if it is trained on upright faces only.
A possible strategy to simplify the task at hand could be to restrict the network to react exclusively to upright faces (Fig. 2c) .
In this case, the set of relevant visual pattern orientations becomes much smaller, at the expense of disrupting equivariance to the rotation group.
Resultantly, the network would risk becoming unable to detect faces in any other orientation than those it is trained on.
A better strategy results from restricting the set of relevant pattern orientations by defining them relative to one another (e.g. mouth
Figure 2: Effect of multiple attention strategies for the prioritization of relevant pattern orientations in rotation equivariant networks for the task of face recognition.
Given that all attention strategies are learned exclusively from upright faces, we show the set of relevant directions for the recognition of faces in two orientations (Fig. 2a) obtained by: no attention (Fig. 2b) , attending to the pattern orientations of appearance independently (Fig. 2c) and, attending to the pattern orientations of appearance relative to one another (Fig. 2d ).
Built upon Figure 1 from Schwarzer (2000) .
orientation w.r.t. the eyes) as opposed to absolutely (e.g. upright mouth) (Fig. 2d ).
In such a way, we are able to exploit information about orientation co-occurrences in the data without disrupting equivariance.
The set of co-occurrent orientations in Fig. 2d corresponds to the co-occurrence envelope of the samples in Fig. 2a for the transformation group defined by rotations.
In this work, we introduce co-attentive equivariant feature mappings and apply them on existing equivariant neural architectures.
To this end, we leverage the concept of attention (Bahdanau et al., 2014) and modify existing mathematical frameworks for equivariance, such that co-occurrent transformations can be detected.
It is critical not to disrupt equivariance in the attention procedure as to preserve it across the entire network.
To this end, we introduce cyclic equivariant self-attention, a novel attention mechanism able to preserve equivariance to a large set of transformation groups.
Experiments and results.
We explore the effects of co-attentive equivariant feature mappings for single and multiple symmetry groups.
Specifically, we replace conventional rotation equivariant mappings in p4-CNNs (Cohen & Welling, 2016) and DRENs (Li et al., 2018) with co-attentive ones.
We show that co-attentive rotation equivariant neural networks consistently outperform their conventional counterparts in fully (rotated MNIST) and partially (CIFAR-10) rotational settings.
Subsequently, we generalize cyclic equivariant self-attention to multiple similarity groups and apply it on p4m-CNNs (Cohen & Welling, 2016 ) (equivariant to rotation and mirror reflections).
Our results are in line with those obtained for single symmetry groups and support our stated hypothesis.
Our results show that co-attentive equivariant feature mappings can be utilized to enhance conventional equivariant ones.
Interestingly, co-attentive equivariant mappings are beneficial both in partially and fully rotational settings.
We attribute this to the fact that a set of co-occurring orientations between patterns can be easily defined (and exploited) in both settings.
It is important to note that we utilized attention independently over each spatial position u on the codomain of the corresponding group convolution.
Resultantly, we were restricted to mappings of the form xA, which, in turn, constraint our attention mechanism to have a circulant structure in order to preserve equivariance (since group actions acting in the codomain of the group convolution involve cyclic permutations and cyclic self-attention is applied in the codomain of the group convolution).
In future work, we want to extend the idea presented here to act on the entire group simultaneously (i.e. along u as well).
By doing so, we lift our current restriction to mappings of the form xA and therefore, may be able to develop attention instances with enhanced descriptive power.
Following the same line of though, we want to explore incorporating attention in the convolution operation itself.
Resultantly, one is not restricted to act exclusively on the codomain of the convolution, but instead, is able to impose structure in the domain of the mapping as well.
Naturally, such an approach could lead to enhanced descriptiveness of the incorporated attention mechanism.
Moreover, we want to utilize and extend more complex attention strategies (e.g. Bahdanau et al. (2014); Luong et al. (2015) ; Vaswani et al. (2017) ; Mishra et al. (2017) ) such that they can be applied to large transformation groups without disrupting equivariance.
As outlined earlier in Section 3.1, this becomes very challenging from a computational perspective as well, as it requires extensive usage of the corresponding attention mechanism.
Resultantly, an efficient implementation thereof is mandatory.
Furthermore, we want to extend co-attentive equivariant feature mappings to continuous (e.g. Worrall et al. (2017) ) and 3D space (e.g. Cohen et al. (2018) ; Worrall & Brostow (2018) ; Cohen et al. (2019) ) groups, and for applications other than visual data (e.g. speech recognition).
Published as a conference paper at ICLR 2020
Finally, we believe that our approach could be refined and extended to a first step towards dealing with the enumeration problem of large groups (Gens & Domingos, 2014) , such that functions acting on the group (e.g. group convolution) are approximated by evaluating them on the set of cooccurring transformations as opposed to on the entire group.
Such approximations are expected to be very accurate, as non-co-occurrent transformations are rare.
This could be though of as sharping up co-occurrent attention to co-occurrent restriction.
We have introduced the concept of co-attentive equivariant feature mapping and applied it in the context of equivariant neural networks.
By attending to the co-occurrence envelope of the data, we are able to improve the performance of conventional equivariant ones on fully (rotated MNIST) and partially (CIFAR-10) rotational settings.
We developed cyclic equivariant self-attention, an attention mechanism able to attend to the co-occurrence envelope of the data without disrupting equivariance to a large set of transformation groups (i.e. all transformation groups G, whose action in the codomain of a G-equivariant feature mapping produce cyclic permutations).
Our obtained results support the proposed co-occurrence envelope hypothesis.
A OBTAINING CO-OCCURRENT ATTENTION VIA EQUATION 5
Figure 4: Synchronous movement of feature mappings and attention masks as a function of input rotation in the group p4 (r max = 4).
In this section, we provide a meticulous description on how co-occurrent attention is obtained via the method presented in the paper.
Intuitively, a direct approach to address the problem illustrated in the introduction (Section 1) and Figure 2 requires an attention mechanism that acts simultaneously on r and λ (see Eq. 3).
However, we illustrate how the depicted problem can be simplified such that attention along r is sufficient by taking advantage of the equivariance property of the network.
Let p be the input of a roto-translational convolution f R : Z 2 × Θ × Λ 0 → Z 2 × Θ × Λ 1 as defined in Eq. 3, and Θ be the set of rotations by θ r degrees: Θ = {θ r = r 2π rmax } rmax r=1 .
Let f R (p)(u) ∈ R rmax×Λ1 be the matrix consisting of the r max oriented responses for each λ ∈ Λ 1 learned representation at a certain position u.
Since the vectors f R (p)(u, λ) ∈ R rmax , λ ∈ Λ 1 permute cyclically as a result of the rotation equviariance property of f R , it is mandatory to ensure equivariance to cyclic permutations for each f R (p)(u, λ) during the course of the attention procedure (see Section 3).
At first sight, one is inclined to think that there is no connection between multiple vectors f R (p)(u, λ) in f R (p)(u), and, therefore, in order to exploit co-occurences, one must impose additional constraints along the λ axis.
However, there is indeed an implicit restriction in f R (p)(u) along λ resulting from the rotation equivariance property of the mapping f R , which we can take advantage from to simplify the problem at hand.
Consider, for instance, the input θ i p, a θ i -rotated version of p.
By virtue of the equivariance property of f R , we have (locally) that f R (θ i p) = P i (f R (p)).
Furthermore, we know that this property must hold for all the learned feature representations f R (p)(u, λ),∀λ ∈ Λ 1 .
Resultantly, we have that:
In other words, if one of the learned mappings f R (p)(u, r, λ) experiences a permutation P i along r, all the learned representations f R (p)(u, r, λ), ∀λ ∈ Λ 1 must experience the exact same permutation P i as well.
Resultantly, the equivariance property of the mapping f R ensures that all the Λ 1 learned feature representations f R (p)(u, λ) "move synchronously" as a function of input rotation θ i .
Likewise, if we apply a cyclic equivariant attention mechanism A C λ independently on top of each λ learned representation f R (p)(u, λ), we obtain that the relation A C λ (f R (θ i p))(u, r, λ) = P i (A C λ (f R (p))(u, r, λ)) ∀λ ∈ Λ 1 (13) must hold as well.
Similarly to the case illustrated in Eq. 12 and given that A C λ is equivariant to cyclic permutations on the domain, we obtain that all the Λ 1 learned attention masks A C λ "move synchronously" as a function of input rotation θ i as well (see Fig. 4 ).
From Eq. 13 and Figure 4 , one can clearly see that by utilizing A C λ independently along r and taking advantage from the fact that all Λ 1 learned feature representations are tied with one another via f R , one is able to prioritize learning of feature representations that co-occur together as opposed to the much looser formulation in Eq. 12, where feedback is obtained from all orientations.

assistant: text: This paper combines attention with group equivariance, specifically looking at the p4m group of rotations, translations, and flips, and derives a form of self-attention that doesn't destroy the equivariance property.