Conjugation Invariant Learning with Neural Networks

1 INTRODUCTION . Learning under group actions . Learning under group actions has emerged as a topic of key interest in recent years . The natural motivation for this paradigm comes from the many applications in which there is a canonical action of a group of symmetries , often arising as isometries or invariances of the model under consideration ( Jebara & Kondor , 2008 ; Wein , 2018 ) . Instances of learning models that admit natural group actions are manifold . Examples range from natural image data that exhibit symmetries accorded by rotations and translations ( Kondor , 2007 ; Kakarala , 2012 ) ; data arising in tomography ( Vasco , 2007 ; Moroder et al. , 2012 ) ; network based data ( Giridhar & Kumar , 2006 ; Herzig et al. , 2018 ) ; synchronization problems in robotics ( Rosen et al. , 2020 ) ; computer vision ( Agrawal et al. , 2006 ) ; problems like multi object tracking where many natural questions are invariant under the rearrangements of object labels ( Kondor et al. , 2007 ) ; atomic physics ( Seko et al. , 2019 ) ; learning permutations ( Huang et al. , 2009 ) ; Gaussian mixture models under group actions ( Brunel , 2019 ) , among many others ( Bandeira et al. , 2015b ) . Significant recent trends include neural networks that are equivariant under group actions on the data ( Kondor & Trivedi , 2018 ; Worrall et al. , 2016 ; Weiler et al. , 2018a ; Thiede et al. , 2021 ) and related convolutional neural networks ( CNNs ) ( Kondor et al. , 2018 ; Cohen et al. , 2018b ; Niepert et al. , 2016 ; Defferrard et al. , 2016 ; Cohen & Welling , 2016 ; Cohen et al. , 2018a ; Esteves et al. , 2018 ; Weiler et al. , 2018b ) ; multi-reference alignment ( MRA ) ( Bandeira et al. , 2014 ; Perry et al. , 2019 ; Bendory et al. , 2017 ; Sorzano et al. , 2010 ; Abbe et al. , 2018 ; Fan et al. , 2020 ) and cryo electron microscopy ( cryo EM ) under the action of the relevant Euclidean isometries ( Singer et al. , 2018 ; Hadani & Singer , 2011 ; Bendory et al. , 2020 ; Bandeira et al. , 2015a ) ; and so forth . Invariant functionals . A significant class of observables ( i.e. , functionals of the data ) that attract natural interest in this setting consist of those that are invariant with respect to the group action under consideration ; e.g. , mutual distances under the action of a group of isometries , or the relative order of two objects under the permutation of their labels . In this work , we focus on a fundamental category of such observables , called class functions , that are invariant under the conjugation action of a group . The conjugation action . Group action by conjugation is one of the foundational ways in which a group can act , and usually corresponds to a change of basis , or relabelling of objects or indices . A simple example is accorded by data in the form of matrices , where conjugation by an invertible matrix translates into a change of Euclidean basis , leading to the presentation of the information in a different co-ordinate system . In this setting , any function of the spectrum of the data matrix remains invariant under the change of basis ( i.e. , the conjugation action ) , and is clearly a significant feature to study . Yet another ready example is accorded by the permutation of the node labels in a network , which leaves invariant important network properties , such as diameter , subgraph counts and other aspects of network geometry , as well as the spectral properties of the network . Symmetries , group representations and Fourier analysis . Fourier analysis is a central tool in fundamental machine learning tasks , entailing a vast collection of concepts including bandlimited functions and low pass filters , productively focussing on the low frequency part of the Fourier spectrum of a signal as the most significant component . At a conceptual level , much of this originates from the actions of the translational group of symmetries acting canonically on signals defined on Euclidean spaces ; in particular , the structural simplicity is largely a consequence of the fact that these groups are abelian or commutative . However , natural learning scenarios where the canonical symmetries form a non-abelian group ( i.e. , non-commutative ) are numerous ; even on Euclidean spaces , the translational symmetries when augmented with rotations constitute a ready example . In such settings , group representation theory ( Simon , 1996 ) accords an extension of many of the Fourier analytic concepts ; for instance the expansion of a function in terms of matrix-valued Fourier coefficients indexed by the so-called irreducible representations of the group . For a more detailed discussion on the fundamental synergies between Fourier analysis and group representations , we refer the reader to Sec . E in the appendix . A non-linear conjugation invariant learning paradigm with neural networks . In the present work , we are focussed on learning conjugation invariant functions ( i.e. , class functions ) , which happen to accord a much simpler version of the non-commutative Fourier analysis discussed above . This is realised via an orthogonal linear basis of the so-called irreducible characters , which are essentially traces of certain minimal matrix representations of the group , and are well-understood in many concrete applications ( c.f . Sec . E in appendix for details ) . It turns out that the space of irreducible characters exhibit a much richer structure in the form an algebra , which entails that it is closed under polynomial combinations . As an upshot , using polynomials of a few irreducible characters of relatively low order , one is able to express much more complicated characters of substantially higher order ; leading to an approximation paradigm for general class functions via non-linear , polynomial combinations of a small alphabet of irreducible characters . In implementing our programme , a fundamental role is played by neural networks , which allow us to effectively implement this non-linear approximation scheme . Our approach , in essence , leads to a global co-ordinate system for learning conjugation invariant functions on groups . Viewed from the perspective of function space approximation , our paradigm may be seen as a kind of non-linear dimension reduction tool for class functions . Groups acting on homogeneous spaces . In some applications , the group of symmetries G acts on a set X of objects ( e.g. , rotations acting on a sphere ) ; such a set ( under a mild transitivity assumption ) is referred to as a homogeneous space of the group . Our setup may be extended to cover such settings , using a canonical ‘ lift ’ of any function f : X 7→ R to a function f̃ : G 7→ R that satisfies f̃ = f ◦ π , for a standard quotient map π : G 7→ X ( Dummit & Foote , 2004 ) . Stipulating that f̃ is conjugation-invariant is equivalent to f being [ G , G ] -invariant on X , where [ G , G ] is the so-called commutator subgroup of G ( c.f . Lemma C.1 in appendix ) . This covers the setting of functions f that are constant on the orbits of [ G , G ] acting on X . Such functions include binary classifiers on the permutation group Sn , classification problems on the dihedral group D2n , and so on . For a detailed discussion , we refer the reader to Sec . C in the appendix . Further applications of our paradigm to groups acting on homogeneous spaces can be obtained via a general averaging paradigm , e.g . in the context of image alignment problems ( c.f . Proposition 2.1 ) . Related literature . Class functions have appeared in various forms as useful methodological tools in the machine learning literature . These include the study of bi-invariant kernels in the context of RKHS on groups and homogeneous spaces ( Jebara & Kondor , 2008 ) ; applications to problems of multi-object tracking Kondor et al . ( 2007 ) ; learning permutations ( Jebara & Kondor , 2008 ) ; diffusion kernels on graphs Kondor & Lafferty ( 2002 ) ; Saloff-Coste ( 2001 ) , and so on . Yarotsky ( 2021 ) examines an invariant theory based approach to learning under general symmetries via two layer neural nets with rather complex polynomial inputs ; in contrast our approach inputs the ubiquitous group characters and approximates polynomials of high degree , via a few-layer network architecture that exploits the compositional aspect of neural nets in a non-trivial way . To the best of our knowledge , our approach to efficient learning of conjugation invariant functions through the lens of group character theory appears to be rather novel , and prior literature in a similar vein appears to be highly limited . 2 CONJUGATION INVARIANT FUNCTIONS AND THEIR APPLICATIONS . Our interest in this paper would be focussed on the so-called class functions , so we begin with a concrete definition . Definition . Let G be a group . Two elements x , y ∈ G are said to be conjugate if ∃t ∈ G such that txt−1 = y . A function f : G 7→ S for any set S is called a class function if f ( txt−1 ) = f ( x ) ∀t ∈ G. Conjugacy is an equivalence relation , partitioning G into equivalence classes which are referred to as conjugacy classes . It follows from the definition that conjugation-invariant functions are constant on these conjugacy classes , hence the name class functions . Class functions on SO ( 3 ) and Sn . Elements of the 3-D rotation group SO ( 3 ) are precisely 3 × 3 orthogonal matrices with determinant equal to 1 . Two elements of SO ( 3 ) are conjugate if and only if they are similar matrices . This means that class functions on SO ( 3 ) only depend on the matrix spectrum of the elements of SO ( 3 ) . In fact , every R ∈ SO ( 3 ) is similar ( via conjugation ) to a block diagonal matrix of the form ( cos θ − sin θ 0 sin θ cos θ 0 0 0 1 ) . In geometric terms , the conjugacy classes of SO ( 3 ) are the sets of rotations by the same angle θ about different axes . The trace of these matrices is an example of a class function ; tr ( R ) = 1 + 2 cos θ . In the case of the symmetric group Sn , two permutations lie in the same conjugacy class if and only if they have the same cycle shape . Since cycle shapes are conveniently labelled using partitions , the conjugacy classes of Sn are most commonly labelled using partitions of n. Class functions arise in many natural settings in machine learning . We provide below several such scenarios where class functions are of key importance . A general averaging paradigm . We first provide a general setting where class functions arise naturally through a process of averaging over the group ; conjugation invariant functionals often emerge in practical problems through such a procedure . Let G be a group acting on a space X ; that is for every g ∈ G , we may associate a map Tg : X → X such that Te = idX is the identity map on the space X ( where e is the identity element in the group G ) , and Tgh = Tg ◦ Th ∀g , h ∈ G. Let µ be the left Haar measure on the group G , i.e. , dµ ( g · h ) = dµ ( h ) for all g , h ∈ G. Then we may state : Proposition 2.1 . Let f : X ×X 7→ R be such that f ( g ( x ) , g ( y ) ) = f ( x , y ) ∀x , y ∈ X , g ∈ G. For any pair x , y ∈ X and T ∈ G , define f̃x , y ( T ) : = ∫ G f ( Tg ( x ) , g ( y ) ) dµ ( g ) . Then f̃x , y is a class function . Functions f as in Proposition 2.1 that are invariant under the diagonal action of G naturally occur in a wide array of settings ; prominent among them being in the form of distances ( or angles ) on a Euclidean space X under the action of a group of isometries G. For a concrete use-case , we refer to the image alignment application below . Proof of Proposition 2.1 . Given any h , T ∈ G , we have f̃x , y ( hTh −1 ) = ∫ G f ( hTh−1g ( x ) , g ( y ) ) dµ ( g ) = ∫ G f ( Th−1g ( x ) , h−1g ( y ) ) dµ ( g ) = ∫ G f ( Tg′ ( x ) , g′ ( y ) ) dµ ( g′ ) = f̃x , y ( T ) . The third equality is due to left-invariance of µ and setting g′ = h−1g . Image Alignment . Image alignment problems have received considerable attention in recent years , and furnish a natural example of class functions . To set the stage , we consider spherical images , which are well-motivated by projections of planar images onto the northern sphere ( Kondor et al. , 2018 ; Jebara & Kondor , 2008 ) , images in the form of photos taken using a 360◦ camera , and so on . In the setting of Proposition 2.1 , we set G = SO ( 3 ) , X = S2 and f ( x , y ) to be the Euclidean distance between x and y , which would naturally be rotation invariant . Then f̃x , y ( T ) is a measure of how well an image centered at x is aligned with an image centered at y after applying the rotation operator T . The quantityMx , y = minT f̃x , y ( T ) captures the maximal alignment between the images , and can be used e.g . for classification tasks ( e.g. , if x is a prototype and y is a new image , then we classify y to the category of x ifMx , y smaller than a threshold ) . Proposition 2.1 demonstrates that f̃x , y is a class function on SO ( 3 ) . Quadratic Assignment Problem . Let Pn denote the group of n× n permutation matrices ; that is matrices having exactly one entry equal to 1 in each row and column , and the remaining entries being 0 . The quadratic assignment problem ( QAP ) is a fundamental problem in combinatorial optimization Koopmans & Beckmann ( 1957 ) , where the goal is to find argmin X∈Pn tr ( AXBXT ) ; we will consider this problem in the setting where A ∈ Pn and B is a general n× n matrix . Proposition 2.2 . Let f : Pn 7→ R be given by f ( A ) : = min X∈Pn tr ( AXBXT ) , and B be an n × n matrix . Then f is a class function . Proof . We verify that f is a class function . Fix any Q , A ∈ Pn . Note that QT = Q−1 . f ( QAQ−1 ) = min X∈Pn tr ( QAQTXBXT ) = min X∈Pn tr ( QT ( QAQTXBXT ) Q ) = min X∈Pn tr ( AQTXBXTQ ) = min X∈Pn tr ( A ( QTX ) B ( QTX ) T ) = f ( A ) . This gives us a stopping algorithm ( or target ) for QAP in the following sense . Once we learn the function f , we could obtain or approximate the objective minimum f ( A ) for the QAP . We can then use some optimization algorithm ( such as gradient descent ) to update the matrixX in order to decrease the objective tr ( AXBXT ) , and terminate the optimization process once tr ( AXBXT ) − f ( A ) < for some acceptable error tolerance threshold > 0 . Testing on comparative rankings . Denote { 1 , 2 , . . . , n } by [ n ] . Let Sn be the symmetric group on [ n ] . Suppose that we have two real-valued functions f1 , f2 : [ n ] → R. Then , f1 and f2 would induce orderings σ1 and σ2 respectively of [ n ] . For instance , we could have n individuals and f1 ( k ) , f2 ( k ) are resp . the height and weight of individual k , leading to their ranking by heights or weights ( denoted resp . by σ1 , σ2 ) . A natural question is the comparative analysis of such rankings , e.g . testing whether they are approximately similar . In such considerations , of natural interest are statistics like the number of individuals having the same rank , given by Ln : = 1n ∑ k∈ [ n ] 1 [ σ1 ( k ) =σ2 ( k ) ] = 1 n ∑ k∈ [ n ] 1 [ σ−12 σ1 ( k ) =k ] , where 1 is the indicator function . Let f : Sn → R be the function f ( σ ) = 1n ∑ k∈ [ n ] 1 [ σ ( k ) =k ] for all σ ∈ Sn ; notice that Ln = f ( σ−12 σ1 ) . For any π , σ ∈ Sn , we may compute f ( πσπ−1 ) = 1n ∑ k∈ [ n ] 1 [ πσπ−1 ( k ) =k ] = 1 n ∑ k∈ [ n ] 1 [ σπ−1 ( k ) =π−1 ( k ) ] = f ( σ ) . This demonstrates that the important statistic Ln = f is a class function . Multi-object tracking . In the multi-object tracking problem , there are k objects with labels { 1 , . . . , k } , and we observe their arrangement which evolves with time ; e.g . the evolution of the formation of jets flying in an air show . The data is succinctly described in terms of a permutation σ ( t ) mapping each label to its position in the arrangement at time t. Most of the fundamental questions about this model are actually independent of the labels ; e.g. , whether the starting and the final arrangements are the same , or the first time t = t0 at which σ ( t ) undergoes a change . etc . Any change in object labels amounts to a conjugation σ ( t ) 7→ πσ ( t ) π−1 for some permutation π , leading to the fact that the answer to the above natural questions pertaining to relative order are all class functions .

Summary: In this manuscript, the authors develop a DNN paradigm that learns functions on groups as polynomials using canonical tracial class functions as an orthogonal basis for class functions called irreducible representations. Since the basis constitutes a nonlinear coordinate system, it has the potential to efficiently handle complex symmetries. The paper shows analyticaly and numericaly that for simple systems such as SO(3) and SU(3), the proposed paradigm has high expressive power with few bases.