Conjugation Invariant Learning with Neural Networks

1 INTRODUCTION . Learning under group actions . Learning under group actions has emerged as a topic of key interest in recent years . The natural motivation for this paradigm comes from the many applications in which there is a canonical action of a group of symmetries , often arising as isometries or invariances of the model under consideration ( Jebara & Kondor , 2008 ; Wein , 2018 ) . Instances of learning models that admit natural group actions are manifold . Examples range from natural image data that exhibit symmetries accorded by rotations and translations ( Kondor , 2007 ; Kakarala , 2012 ) ; data arising in tomography ( Vasco , 2007 ; Moroder et al. , 2012 ) ; network based data ( Giridhar & Kumar , 2006 ; Herzig et al. , 2018 ) ; synchronization problems in robotics ( Rosen et al. , 2020 ) ; computer vision ( Agrawal et al. , 2006 ) ; problems like multi object tracking where many natural questions are invariant under the rearrangements of object labels ( Kondor et al. , 2007 ) ; atomic physics ( Seko et al. , 2019 ) ; learning permutations ( Huang et al. , 2009 ) ; Gaussian mixture models under group actions ( Brunel , 2019 ) , among many others ( Bandeira et al. , 2015b ) . Significant recent trends include neural networks that are equivariant under group actions on the data ( Kondor & Trivedi , 2018 ; Worrall et al. , 2016 ; Weiler et al. , 2018a ; Thiede et al. , 2021 ) and related convolutional neural networks ( CNNs ) ( Kondor et al. , 2018 ; Cohen et al. , 2018b ; Niepert et al. , 2016 ; Defferrard et al. , 2016 ; Cohen & Welling , 2016 ; Cohen et al. , 2018a ; Esteves et al. , 2018 ; Weiler et al. , 2018b ) ; multi-reference alignment ( MRA ) ( Bandeira et al. , 2014 ; Perry et al. , 2019 ; Bendory et al. , 2017 ; Sorzano et al. , 2010 ; Abbe et al. , 2018 ; Fan et al. , 2020 ) and cryo electron microscopy ( cryo EM ) under the action of the relevant Euclidean isometries ( Singer et al. , 2018 ; Hadani & Singer , 2011 ; Bendory et al. , 2020 ; Bandeira et al. , 2015a ) ; and so forth . Invariant functionals . A significant class of observables ( i.e. , functionals of the data ) that attract natural interest in this setting consist of those that are invariant with respect to the group action under consideration ; e.g. , mutual distances under the action of a group of isometries , or the relative order of two objects under the permutation of their labels . In this work , we focus on a fundamental category of such observables , called class functions , that are invariant under the conjugation action of a group . The conjugation action . Group action by conjugation is one of the foundational ways in which a group can act , and usually corresponds to a change of basis , or relabelling of objects or indices . A simple example is accorded by data in the form of matrices , where conjugation by an invertible matrix translates into a change of Euclidean basis , leading to the presentation of the information in a different co-ordinate system . In this setting , any function of the spectrum of the data matrix remains invariant under the change of basis ( i.e. , the conjugation action ) , and is clearly a significant feature to study . Yet another ready example is accorded by the permutation of the node labels in a network , which leaves invariant important network properties , such as diameter , subgraph counts and other aspects of network geometry , as well as the spectral properties of the network . Symmetries , group representations and Fourier analysis . Fourier analysis is a central tool in fundamental machine learning tasks , entailing a vast collection of concepts including bandlimited functions and low pass filters , productively focussing on the low frequency part of the Fourier spectrum of a signal as the most significant component . At a conceptual level , much of this originates from the actions of the translational group of symmetries acting canonically on signals defined on Euclidean spaces ; in particular , the structural simplicity is largely a consequence of the fact that these groups are abelian or commutative . However , natural learning scenarios where the canonical symmetries form a non-abelian group ( i.e. , non-commutative ) are numerous ; even on Euclidean spaces , the translational symmetries when augmented with rotations constitute a ready example . In such settings , group representation theory ( Simon , 1996 ) accords an extension of many of the Fourier analytic concepts ; for instance the expansion of a function in terms of matrix-valued Fourier coefficients indexed by the so-called irreducible representations of the group . For a more detailed discussion on the fundamental synergies between Fourier analysis and group representations , we refer the reader to Sec . E in the appendix . A non-linear conjugation invariant learning paradigm with neural networks . In the present work , we are focussed on learning conjugation invariant functions ( i.e. , class functions ) , which happen to accord a much simpler version of the non-commutative Fourier analysis discussed above . This is realised via an orthogonal linear basis of the so-called irreducible characters , which are essentially traces of certain minimal matrix representations of the group , and are well-understood in many concrete applications ( c.f . Sec . E in appendix for details ) . It turns out that the space of irreducible characters exhibit a much richer structure in the form an algebra , which entails that it is closed under polynomial combinations . As an upshot , using polynomials of a few irreducible characters of relatively low order , one is able to express much more complicated characters of substantially higher order ; leading to an approximation paradigm for general class functions via non-linear , polynomial combinations of a small alphabet of irreducible characters . In implementing our programme , a fundamental role is played by neural networks , which allow us to effectively implement this non-linear approximation scheme . Our approach , in essence , leads to a global co-ordinate system for learning conjugation invariant functions on groups . Viewed from the perspective of function space approximation , our paradigm may be seen as a kind of non-linear dimension reduction tool for class functions . Groups acting on homogeneous spaces . In some applications , the group of symmetries G acts on a set X of objects ( e.g. , rotations acting on a sphere ) ; such a set ( under a mild transitivity assumption ) is referred to as a homogeneous space of the group . Our setup may be extended to cover such settings , using a canonical ‘ lift ’ of any function f : X 7→ R to a function f̃ : G 7→ R that satisfies f̃ = f ◦ π , for a standard quotient map π : G 7→ X ( Dummit & Foote , 2004 ) . Stipulating that f̃ is conjugation-invariant is equivalent to f being [ G , G ] -invariant on X , where [ G , G ] is the so-called commutator subgroup of G ( c.f . Lemma C.1 in appendix ) . This covers the setting of functions f that are constant on the orbits of [ G , G ] acting on X . Such functions include binary classifiers on the permutation group Sn , classification problems on the dihedral group D2n , and so on . For a detailed discussion , we refer the reader to Sec . C in the appendix . Further applications of our paradigm to groups acting on homogeneous spaces can be obtained via a general averaging paradigm , e.g . in the context of image alignment problems ( c.f . Proposition 2.1 ) . Related literature . Class functions have appeared in various forms as useful methodological tools in the machine learning literature . These include the study of bi-invariant kernels in the context of RKHS on groups and homogeneous spaces ( Jebara & Kondor , 2008 ) ; applications to problems of multi-object tracking Kondor et al . ( 2007 ) ; learning permutations ( Jebara & Kondor , 2008 ) ; diffusion kernels on graphs Kondor & Lafferty ( 2002 ) ; Saloff-Coste ( 2001 ) , and so on . Yarotsky ( 2021 ) examines an invariant theory based approach to learning under general symmetries via two layer neural nets with rather complex polynomial inputs ; in contrast our approach inputs the ubiquitous group characters and approximates polynomials of high degree , via a few-layer network architecture that exploits the compositional aspect of neural nets in a non-trivial way . To the best of our knowledge , our approach to efficient learning of conjugation invariant functions through the lens of group character theory appears to be rather novel , and prior literature in a similar vein appears to be highly limited . 2 CONJUGATION INVARIANT FUNCTIONS AND THEIR APPLICATIONS . Our interest in this paper would be focussed on the so-called class functions , so we begin with a concrete definition . Definition . Let G be a group . Two elements x , y ∈ G are said to be conjugate if ∃t ∈ G such that txt−1 = y . A function f : G 7→ S for any set S is called a class function if f ( txt−1 ) = f ( x ) ∀t ∈ G. Conjugacy is an equivalence relation , partitioning G into equivalence classes which are referred to as conjugacy classes . It follows from the definition that conjugation-invariant functions are constant on these conjugacy classes , hence the name class functions . Class functions on SO ( 3 ) and Sn . Elements of the 3-D rotation group SO ( 3 ) are precisely 3 × 3 orthogonal matrices with determinant equal to 1 . Two elements of SO ( 3 ) are conjugate if and only if they are similar matrices . This means that class functions on SO ( 3 ) only depend on the matrix spectrum of the elements of SO ( 3 ) . In fact , every R ∈ SO ( 3 ) is similar ( via conjugation ) to a block diagonal matrix of the form ( cos θ − sin θ 0 sin θ cos θ 0 0 0 1 ) . In geometric terms , the conjugacy classes of SO ( 3 ) are the sets of rotations by the same angle θ about different axes . The trace of these matrices is an example of a class function ; tr ( R ) = 1 + 2 cos θ . In the case of the symmetric group Sn , two permutations lie in the same conjugacy class if and only if they have the same cycle shape . Since cycle shapes are conveniently labelled using partitions , the conjugacy classes of Sn are most commonly labelled using partitions of n. Class functions arise in many natural settings in machine learning . We provide below several such scenarios where class functions are of key importance . A general averaging paradigm . We first provide a general setting where class functions arise naturally through a process of averaging over the group ; conjugation invariant functionals often emerge in practical problems through such a procedure . Let G be a group acting on a space X ; that is for every g ∈ G , we may associate a map Tg : X → X such that Te = idX is the identity map on the space X ( where e is the identity element in the group G ) , and Tgh = Tg ◦ Th ∀g , h ∈ G. Let µ be the left Haar measure on the group G , i.e. , dµ ( g · h ) = dµ ( h ) for all g , h ∈ G. Then we may state : Proposition 2.1 . Let f : X ×X 7→ R be such that f ( g ( x ) , g ( y ) ) = f ( x , y ) ∀x , y ∈ X , g ∈ G. For any pair x , y ∈ X and T ∈ G , define f̃x , y ( T ) : = ∫ G f ( Tg ( x ) , g ( y ) ) dµ ( g ) . Then f̃x , y is a class function . Functions f as in Proposition 2.1 that are invariant under the diagonal action of G naturally occur in a wide array of settings ; prominent among them being in the form of distances ( or angles ) on a Euclidean space X under the action of a group of isometries G. For a concrete use-case , we refer to the image alignment application below . Proof of Proposition 2.1 . Given any h , T ∈ G , we have f̃x , y ( hTh −1 ) = ∫ G f ( hTh−1g ( x ) , g ( y ) ) dµ ( g ) = ∫ G f ( Th−1g ( x ) , h−1g ( y ) ) dµ ( g ) = ∫ G f ( Tg′ ( x ) , g′ ( y ) ) dµ ( g′ ) = f̃x , y ( T ) . The third equality is due to left-invariance of µ and setting g′ = h−1g . Image Alignment . Image alignment problems have received considerable attention in recent years , and furnish a natural example of class functions . To set the stage , we consider spherical images , which are well-motivated by projections of planar images onto the northern sphere ( Kondor et al. , 2018 ; Jebara & Kondor , 2008 ) , images in the form of photos taken using a 360◦ camera , and so on . In the setting of Proposition 2.1 , we set G = SO ( 3 ) , X = S2 and f ( x , y ) to be the Euclidean distance between x and y , which would naturally be rotation invariant . Then f̃x , y ( T ) is a measure of how well an image centered at x is aligned with an image centered at y after applying the rotation operator T . The quantityMx , y = minT f̃x , y ( T ) captures the maximal alignment between the images , and can be used e.g . for classification tasks ( e.g. , if x is a prototype and y is a new image , then we classify y to the category of x ifMx , y smaller than a threshold ) . Proposition 2.1 demonstrates that f̃x , y is a class function on SO ( 3 ) . Quadratic Assignment Problem . Let Pn denote the group of n× n permutation matrices ; that is matrices having exactly one entry equal to 1 in each row and column , and the remaining entries being 0 . The quadratic assignment problem ( QAP ) is a fundamental problem in combinatorial optimization Koopmans & Beckmann ( 1957 ) , where the goal is to find argmin X∈Pn tr ( AXBXT ) ; we will consider this problem in the setting where A ∈ Pn and B is a general n× n matrix . Proposition 2.2 . Let f : Pn 7→ R be given by f ( A ) : = min X∈Pn tr ( AXBXT ) , and B be an n × n matrix . Then f is a class function . Proof . We verify that f is a class function . Fix any Q , A ∈ Pn . Note that QT = Q−1 . f ( QAQ−1 ) = min X∈Pn tr ( QAQTXBXT ) = min X∈Pn tr ( QT ( QAQTXBXT ) Q ) = min X∈Pn tr ( AQTXBXTQ ) = min X∈Pn tr ( A ( QTX ) B ( QTX ) T ) = f ( A ) . This gives us a stopping algorithm ( or target ) for QAP in the following sense . Once we learn the function f , we could obtain or approximate the objective minimum f ( A ) for the QAP . We can then use some optimization algorithm ( such as gradient descent ) to update the matrixX in order to decrease the objective tr ( AXBXT ) , and terminate the optimization process once tr ( AXBXT ) − f ( A ) < for some acceptable error tolerance threshold > 0 . Testing on comparative rankings . Denote { 1 , 2 , . . . , n } by [ n ] . Let Sn be the symmetric group on [ n ] . Suppose that we have two real-valued functions f1 , f2 : [ n ] → R. Then , f1 and f2 would induce orderings σ1 and σ2 respectively of [ n ] . For instance , we could have n individuals and f1 ( k ) , f2 ( k ) are resp . the height and weight of individual k , leading to their ranking by heights or weights ( denoted resp . by σ1 , σ2 ) . A natural question is the comparative analysis of such rankings , e.g . testing whether they are approximately similar . In such considerations , of natural interest are statistics like the number of individuals having the same rank , given by Ln : = 1n ∑ k∈ [ n ] 1 [ σ1 ( k ) =σ2 ( k ) ] = 1 n ∑ k∈ [ n ] 1 [ σ−12 σ1 ( k ) =k ] , where 1 is the indicator function . Let f : Sn → R be the function f ( σ ) = 1n ∑ k∈ [ n ] 1 [ σ ( k ) =k ] for all σ ∈ Sn ; notice that Ln = f ( σ−12 σ1 ) . For any π , σ ∈ Sn , we may compute f ( πσπ−1 ) = 1n ∑ k∈ [ n ] 1 [ πσπ−1 ( k ) =k ] = 1 n ∑ k∈ [ n ] 1 [ σπ−1 ( k ) =π−1 ( k ) ] = f ( σ ) . This demonstrates that the important statistic Ln = f is a class function . Multi-object tracking . In the multi-object tracking problem , there are k objects with labels { 1 , . . . , k } , and we observe their arrangement which evolves with time ; e.g . the evolution of the formation of jets flying in an air show . The data is succinctly described in terms of a permutation σ ( t ) mapping each label to its position in the arrangement at time t. Most of the fundamental questions about this model are actually independent of the labels ; e.g. , whether the starting and the final arrangements are the same , or the first time t = t0 at which σ ( t ) undergoes a change . etc . Any change in object labels amounts to a conjugation σ ( t ) 7→ πσ ( t ) π−1 for some permutation π , leading to the fact that the answer to the above natural questions pertaining to relative order are all class functions .

Summary: The paper first makes the case that learning conjugation invariant functions is of interest in many machine learning problems, stating in a general form in which they appear. It then presents a simple method, using a MLP, for approximating such functions efficiently using a small number of irreducible characters. This capitalizes on the notion that irreducible characters are closed under polynomial combinations. Thus, using polynomials of a few irreducible characters (of lower order) one can express characters of considerably higher order. It is shown that the method proposed can approximate class functions of interest well numerically.