Title: MatrixNet: Learning over symmetry groups using learned group representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Group Theory

Representation Theory

Feature learning

geometric deep learning

homomorphic
Abstract: Group theory has been used in machine learning to provide a theoretically grounded approach for incorporating known symmetry transformations in tasks from robotics to protein modeling. In these applications, equivariant neural networks use known
symmetry groups with predefined representations to learn over geometric input data. We propose MatrixNet, a neural network architecture that learns matrix representations of group element inputs instead of using predefined representations. MatrixNet achieves higher sample efficiency and generalization over several standard baselines in prediction tasks over the several finite groups and the Artin braid group. We also show that MatrixNet respects group relations allowing generalization to group elements of greater word length than in the training set. Our code is available at https://github.com/lucas-laird/MatrixNet.

MatrixNet: Learning over symmetry groups using learned group representations

Abstract

Group theory has been used in machine learning to provide a theoretically grounded approach for incorporating known symmetry transformations in tasks from robotics to protein modeling. In these applications, equivariant neural networks use known symmetry groups with predefined representations to learn over geometric input data. We propose MatrixNet, a neural network architecture that learns matrix representations of group element inputs instead of using predefined representations. MatrixNet achieves higher sample efficiency and generalization over several standard baselines in prediction tasks over the several finite groups and the Artin braid group. We also show that MatrixNet respects group relations allowing generalization to group elements of greater word length than in the training set. Our code is available at https://github.com/lucas-laird/MatrixNet .

1 Introduction

The choice of representation for input features is a key design aspect for deep learning. While a wide variety of data types are considered in learning tasks, for example, sequences, sets, images, time series, and graphs, neural network architectures admit only tensors as input. Various methods exist for mapping different input types to tensors such as one-hot embeddings, discretization of continuous signals, learned token embeddings of image patches or words [1, 2], adjacency matrices [3], positional encodings [1], or spectral embeddings [4].

In this paper we consider the question of what feature representations to use for learning tasks with inputs coming from a symmetry group. There are many examples of tasks defined over symmetry groups, such as policy learning in robotics [5], reinforcement learning [6], pose estimation in computer vision [7], sampling states of quantum systems [8], inference over orderings of a set [9], and grouptheoretic invariants in pure mathematics. Past work has typically employed fixed representations chosen from among the known representation theory of the group. Representation theory is the branch of mathematics concerned with classifying the set of representations of a group G which, in this context, refers to homomorphic realizations of a group in terms of n × n matrices. For groups with well understood representation theory, for example, the symmetric group S n or SO( n ) , this provides a ready set of embeddings for converting group elements into tensors for use in downstream

models. Trial and error or topological analysis has shown that the choice of group representation is critical for learning [10, 11, 12, 13]. 1

Instead of using predefined group element representations, we propose to learn feature representations for each group element using a learned group representation. That is, we learn to map group elements to invertible n × n matrices which respect the symmetry group structure. There are several advantages to this strategy. First, unlike predefined group representations, learned representations allow the model to adapt to the given task and capture relevant information for the learning task. Second, learned representations provide reasonable correlations between the learned features for different group elements since they incorporate algebraic structure into the model. This structure is encouraged using free group generators and group relation regularization. Third, relative to learned vector embeddings, learned matrix representations are very parameter efficient for encoding different group elements, reducing the problem to learning only representations of group generators; using sequence encoding, our method is able to generalize to combinatorially large or even infinite groups. Fourth, the learned representation admits analysis in terms of the irreducible subspaces of the generators, giving insight into the model's understanding of the task.

We integrate our learned group representation into a specialized architecture, MatrixNet, adapted for learning mappings related to open problems in representation theory. We compare against several more general baselines on order prediction over finite groups and estimating sizes of categorical objects under an action of the Artin braid group. Through our experiments we observe that our approach achieves higher sample efficiency and performance than the baselines. We additionally show that MatrixNet's constraints allow for it to generalize to unseen group elements automatically without the need for additional data augmentation.

Our contributions include:

Formulation of the problem of learning over groups as a sequence learning problem in terms of group generators and invariance to group axioms and relations,

The MatrixNet architecture, which utilizes learned group representations for flexible and efficient feature learning over symmetry groups,

The matrix block method for constraining the network to respect the group axioms and an additional loss term for learning group relations,

Empirical validation showing MatrixNet outperforms baseline models on a task over the symmetric group and a task over the braid group related to open problems in mathematics.

2 Related Works

Mathematically Constrained Networks Many deep learning methods incorporate mathematical constraints to better model underlying data. For instance, the application of graph neural networks [3, 14] to problems with an underlying graph structure has led to state of the art performance in many domains. Deep learning models have also been designed for tasks with known mathematical formulations by parameterizing components of algorithmic solutions as neural networks and leveraging their structures for more efficient optimization [15, 16]. More broadly the field of geometric deep learning [17] advocates for building neural networks which reflect the geometric structure of data in their latent features. When symmetries are present in data, group equivariant neural networks [18, 19, 20, 21] can enable improved generalization and data efficiency by incorporating known symmetries into the model architecture using the representation theory of groups. Our method also utilizes group representations, but unlike equivariant neural networks, we use learned as opposed to fixed representations. We also focus on modeling functions defined on the group as opposed to between representations of the group.

Learning Structured Representations Instead of using predefined representations as inputs, many methods seek to learn mathematically structured representations from data. This idea has been applied in physics [22, 23], robotics [24], world models [25], self-supervised learning [26, 27], and unsupervised disentanglement [28]. Park et al. [29], for example, use a combination of

learned symmetry and equivariant constraints to map images to group elements or vectors in group representations. Similar techniques are used in symmetry discovery where the underlying group symmetry is not known a priori [30, 31, 32]. Yang et al. [32] use a generative model to learn latent representations with a linear group action in order to find unknown group symmetries in data. Here, in contrast, we start with a known group and learn a matrix representation. Hajij et al. [33] propose algebraically-informed neural networks which learn a non-linear group action from a group presentation. We also learn a group action but consider linear group actions and the learning signal comes not only from the group presentation but also a downstream task.

Deep Learning for Math Recent work has shown deep learning can be useful for providing examples, insight, and proofs related to open problems in mathematics. One approach is the application of language models to mathematics [34], which has the benefit of flexibility in how the model is prompted yet is difficult to interpret and prone to errors. Meanwhile significant work has been done in the area of symbolic regression and automated theorem proving [35]. Other work applies deep learning to the direct modeling of partial differential equations [36, 37]. These methods can perform exceptionally well on real-world data [38], but suffer when trying to interpret predictions made for the purposes of mathematical research. Another avenue involves training graph neural networks on mathematical data such as group or knot invariants and analyzing the learned representations to see which features are significant as a way to provide intuition to mathematicians [39] . Our method also uses structured inputs and learned features, but uses a sequence model and learned group representation instead of a graph neural network with learned node attributes.

3 Background

3.1 Group Theory

Groups encode systems of symmetry and have been used in machine learning to build invariance into neural networks to various transformations [18]. Formally, a group G is a set equipped with an associative binary operation ◦ : G × G → G which satisfies two axioms: (1) there exists an identity element 1 ∈ G , such that g ◦ 1 = 1 ◦ g = g , (2) for each g ∈ G , there exists an inverse g -1 ∈ G such that g ◦ g -1 = g -1 ◦ g = 1 . Examples of groups include finite groups such as the dihedral group D 4 which gives the symmetries of the square, SO(3) , the continuous group of 3D rotations, or ( Z , +) the infinite discrete group of integer shifts.

Since groups may be combinatorially large or infinite, it is essential to encode their elements and compositional structure in a succinct way. For many discrete groups, generators and relations provide such a description. A set of elements S = { g 1 , ..., g n } ⊆ G are called generators if every element of G can be written as a composition of g 1 , g -1 1 , ..., g n , g -1 n . In general, each element of G may be written in many different ways in terms of the generators; this non-uniqueness is encoded using a set of relations. A set R = { r 1 , . . . , r m } of words in the generators S are relations for G if each word r i is equal to the identity in G and if R generates the entire set of words equal to identity under composition and conjugation. The generators and relations of a group taken together are called a presentation and denoted G = ⟨ g 1 , ..., g n | r 1 , ..., r m ⟩ . For example D 4 = ⟨ r, f | r 4 = f 2 = frfr ⟩ . Due to relations, group elements do not have a unique word representation. For example frf = r 3 = r 7 all represent the same group element. By convention relations are sometimes stated as equalities instead of single group elements, for example frf = r 3 . The free group F S = ⟨ g 1 , . . . , g n ⟩ is defined to have no relations except for those coming from the two group axioms above.

An important notion for our discussion is the order of an element. If g ∈ G then the order of g , denoted | g | , is the smallest k such that g k = e . (For non-finite groups, k may be infinity). The order of the group | G | is simply the number of elements in the group. For any g , Lagrange's theorem implies that | g | is a divisor of | G | [40], which restricts the possible orders an element may take.

3.2 Representation Theory

Abstract group presentations are difficult to work with in many settings. Group representations map group elements to invertible matrices such that composition of group elements corresponds to matrix multiplication. This gives the group a natural action on vector spaces and allow for analysis of the group using linear algebra. Formally, a representation of a group G is a group homomorphism

Φ : G → GL( n ) to the group of invertible n × n matrices. That is Φ( g 1 · g 2 ) = Φ( g 1 ) · Φ( g 2 ) . A property of Φ is that Φ(1) = I n × n and Φ( g -1 ) = Φ( g ) -1 . Due to the homomorphism property, it is sufficient to define Φ for generators of the group G . For example, a 2 × 2 representation of D 4 is given by mapping r to a π/ 2 -rotation matrix and f to a reflection over the x -axis.

The representations of many groups are well classified. This provides a ready source of tensor representations for group elements to use as inputs for neural networks. For example, for finite groups, by Maschke's Theorem [41], representations may be decomposed into irreducible representations . That is, there exists a basis such that the representation matrices are all block diagonal with the same block sizes and these blocks cannot be further subdivided. The irreducible representations may then be further classified by computing character tables.

3.3 Symmetric and Braid Group

The braid group on n strands B n has presentation

The braid group intuitively represents all possible ways to braid a set of n strands. The generators σ i correspond to twisting strand i over i +1 and σ -1 i is the reverse, twisting strand i +1 over i . It is defined topologically as equivalence classes up to ambient isotopy of n non-intersecting curves in R 3 connecting two sets of n fixed points. The braid group is infinite and though some representations are known, they are not fully classified [42]. The braid group has important connections to knot theory, mathematical physics, representation theory, and category theory.

The symmetric group on n elements, denoted S n , is defined as the set of bijections from { 1 , . . . , n } to itself. It is also a quotient of the braid group B n and has a presentation similar to Eqn. 1 but with additional relations σ 2 i = 1 for 1 ≤ i ≤ n -1 . Here σ i is the transposition ( i i +1) . The symmetric group has finite order | S n | = n ! . Representations of the symmetric group are well understood. The irreducible representations are parameterized by partitions of n . For more details, see [43].

3.4 Categorical Braid Actions

One current active research problem in mathematics concerns actions of braid groups on categories. A category is an abstract mathematical structure that has objects and maps or morphisms between objects, satisfying several coherence axioms. For example, the category Vect R has objects which are real vector spaces and morphisms which are linear maps. Functors are maps between categories that take objects to objects and morphisms to morphisms between the corresponding objects, satisfying several compatibility conditions. The action of a group G on a category C means that each group element g ∈ G is associated with an invertible functor F g : C → C , such that any relation x = y in the group implies that the corresponding functors F x and F y are naturally isomorphic.

Given a category on which a braid group acts, mathematicians are interested in measuring how objects grow under repeated applications of elements of the braid group. The 'size' of an object in a category may be measured using a tool called Jordan-Hölder filtrations. For example, Bapat et al. [44] attempt to measure growth rates of objects in a specific category C n under repeated applications of certain twist functors σ P i , which define an action of the braid group B n on C n . Each object in the category has a Jordan-Hölder filtration giving a unique vector in Z n ≥ 0 of Jordan-Hölder multiplicities. For more details see [44] and Appendix A.

However, they are only able to compute the action in certain cases and a simple formula is elusive. A complete description of the Jordan-Hölder multiplicities after applying combinations of σ P i to one of the generating objects is only known for n = 3 ; that is, the case of the 3-strand braid group B 3 . Understanding how these multiplicities evolve under repeated application of braids is a challenging open research problem in mathematics.

4 Methods

We formulate the problem of learning a function on a symmetry group as a sequence learning problem using a presentation of the group in terms of generators and relations. We propose MatrixNet which predicts the label using a learned matrix representation for the group. The homomorphism property of the representation is enforced through a combination of model design and an auxiliary loss term.

4.1 Problem Formulation

Weconsider task functions of the form f : G → R c where G is a finite or discrete group and the output space R c may represent either a regression target or class label. While such tasks appear in computer vision, robotics, and protein modeling, we are particularly interested in problems in mathematics where neural models may lend additional examples and insight towards proving theorems.

To efficiently represent group elements in infinite or large groups, we consider a presentation of G = ⟨ S | R ⟩ in terms of generators S = { g 1 , . . . , g n } and relations R = { r 1 , . . . , r m } . Model inputs g ∈ G are represented by sequences of generators ( g i 1 , . . . , g i ℓ ) where g = g i 1 ◦ . . . ◦ g i ℓ is of arbitrary length ℓ ≥ 0 and 1 ≤ i j ≤ n . For convenience, we can include the identity g 0 = e among the generators to pad sequences without changing the group element.

Since a single group element may be represented by different sequences, it is critical for the model f θ to be invariant to both the group axioms and relations. That is, we desire

4.2 MatrixNet

g = 010201 E Sn

Lce(g, y)

Sequential

MatrixMultiplication

Matrix Block

Product

Flattened

01 → W, →

Matrix

Matrix

Task

Ecp

M

Model

2 → W2→

We propose MatrixNet (see Figure 1), a neural sequence model which models functions on a group G by taking as input sequences of generators for G . It achieves invariance to group axioms and relations through a combination of built in constraints and an auxiliary loss term. The key part of the MatrixNet architecture is the matrix block which takes a group generator g i as input and outputs an invertible square matrix representation M g i . The matrix representation M g for an arbitrary group element g is defined as the product of matrix representations of generators needed to generate g . This matrix representation is then flattened and passed to a downstream task model (such as an MLP) which computes the label. In what follows, we give a more detailed description of the matrix block and some variations on the architecture.

Signed one-hot encoding We define the signed one-hot encoding , a modified version of the traditional one-hot encoding, for encoding group generators used as input to MatrixNet. Let ( g ϵ 1 i 1 , g ϵ 2 i 2 , ..., g ϵ ℓ i ℓ ) be a sequence of generators where 0 ≤ i k ≤ n and ϵ k ∈ {± 1 } . The signed one-hot encoding encodes each generator g ϵ k i k as a vector v g i k = ϵe i = [0 , . . . , 0 , ϵ k , 0 , . . . , 0] ∈ R n which is 1 or -1 in the i th entry. The identity element g 0 = 1 is mapped to the zero vector v 1 = 0 ∈ R n . The signed one-hot encoding is chosen since it intuitively relates a generator and its inverse as v g -1 = -v g .

4.2.1 Matrix Blocks and Learned Matrix Representations

The matrix block is designed as a parameterized representation of the free group F S , that is, a homomorphism Φ : F S → GL ( n ) which satisfies 3 properties: (1) the matrix Φ( g ) = M g is invertible, (2) Φ(1) = I n × n , and (3) if Φ( g ) = M g then Φ( g -1 ) = M -1 g . In what follows v g i k is

the signed one-hot encoding of the generator g i k and W is a learnable parameter matrix. For a group element g = g i 1 . . . g i n , the matrix block is defined

where Reshape reshapes a vector in R n 2 into a square n × n matrix.

Proposition 1. Matrix Block defines a representation of the free group.

Proof. Property (1) is satisfied since the outputs of a matrix exponential are invertible. Properties (2) and (3) follow from v g -1 i k = -v g i k , v 1 = 0 , and properties of the matrix exponential,

4.2.2 Variations of Matrix Block

We now present some variations of this simple design that satisfy the group homomorphism properties.

Linear Network (MatrixNet-LN) The first variant replaces the single parameter matrix W with a linear network that has two parameter matrices W 1 , W 2 . The linear network matrix block changes the computation of the intermediate A k matrix to:

This is still a linear function of v g i k meaning Proposition 1 still holds by the same argument.

While this variation does not give increased expressivity over the original formulation of the matrix block, the linear network can change the optimization landscape leading to different performance in practice. This variation is called MatrixNet-LN.

Non-Linearity (MatrixNet-NL) The second variation introduces an element-wise odd non-linear function f . That is f ( -x ) = -f ( x ) as with tanh . The non-linear matrix block modifies

Proposition 2. Non-linear matrix block defines a representation of the free group.

Proof. Property (1) is satisfied by the same argument in Proposition 1. Since f is an odd function, f ( W 1 v 1 ) = f ( 0 ) and Reshape( f ( W 1 v g -1 i k )) = Reshape( f ( -W 1 v g i k )) = -A k . Therefore Properties (2) and (3) are satisfied by the same argument in Proposition 1.

This variation can be combined with the first as Proposition 2 holds for linear transformations of A k . That is, A k = Reshape( W 2 f ( W 1 v g i k )) . Unless otherwise noted we set f = tanh .

Block-Diagonal (MatrixNet-MC) The third variation is inspired by the decomposition of representations into irreducible representations. For certain classes of groups such as finite groups, every representation decomposes such that the matrices M g have a consistent block diagonal structure in some basis. Thus to learn an arbitrary representation of the group G , it suffices to learn a block diagonal representation assuming the blocks are large enough.

This variation learns ℓ intermediate n j × n j matrices A k j which are combined to form a blockdiagonal n × n matrix A k where n = ∑ ℓ j =1 n j . The block-diagonal matrix block is defined with

Note that MatrixExp and matrix multiplication preserve the block structure. If the sizes n j are fixed to all be equal, this formulation can be implemented as a multi-channel matrix block where both A k and M g i k are ℓ × n j × n j tensors with ℓ channels. Each A k j is calculated identically to A k in the original matrix block formulation, and BlockDiag is linear, so Proposition 1 still holds. This variation is also compatible with the previous two variations. In our experiments, we implement a 3-block version called MatrixNet-MC with a single linear layer and no non-linearity.

4.3 Enforcing group relation invariances

The matrix block is constrained to learn a representation of the free group F S . As a consequence MatrixNet will satisfy (2) and (3) as desired. However, most groups have relations which cannot be enforced through a simple weight-sharing scheme used in equivariant architectures [19]. We propose to learn the relations through a secondary loss which measures how closely the representation respects the group relations. More concretely, let G = ⟨ S | R ⟩ be a group with relations r i ∈ R . The loss is:

Since MatrixNet learns a representation that is invariant to group axioms, it is sufficient to sum over only { r i } i m =1 and not all compositions of relations. For architectures which do not respect the free group structure, the relations r i alone may not guarantee that all equivalent words have identical feature representations, requiring potentially combinatorial amounts of data augmentation. This allows MatrixNet to both efficiently learn group relation invariance and simply verify this invariance without any data augmentation.

5 Experiments

We use two learning tasks to evaluate the four variants of MatrixNet and compare our approach against several baseline models. We use several finite groups on a well understood task as an initial test to validate our approach and then move on to an infinite group, the braid group B 3 , on a task related to open problems. As baselines, we compare to an MLP for fixed maximum sequence length and LSTM and Transformer models on longer sequences. Baseline model parameters were chosen so all of the models have approximately the same number of trainable parameters.

5.1 Order Prediction in Finite Groups

The first task is to predict the order of group elements in finite groups. Elements of finite groups are input as finite sequences of generators as described in Section 3.3. The typical efficient algorithm for computing the order involves disjoint cycle decomposition, making order classification a non-trivial task. See Appendix B.1 for more details on the sampling method and data splits.

Models Compared We compare MatrixNet variants and three baselines for order prediction in S 10 : (1) MLP with 3 layers with hidden dimension 256 and SiLU activations, (2) Fixed representation input to a 2-layer classifier MLP with 256 hidden dimensions and SiLU activations, (3) LSTM input to a 2-layer LSTM with 256 hidden dimensions with a subsequent MLP classifier using SiLU activations, (4) MatrixNet-LN with a 2-layer 256 hidden dimension matrix block and classifier network with SiLU activations. (5) MatrixNet-MC with a 2 x 2 matrix block size over 5 matrix channels and classifier network with SiLU activations. (6) MatrixNet-NL with a 2-layer 256 hidden dimension matrix block with SiLU activations and classifier network with SiLU activations. The precomputed representation is an ablated version of MatrixNet using a fixed representation of S 10 instead of a learned one. For the fixed representation, we use the standard 10 × 10 representation given by the permutation matrices corresponding to the group element. In MatrixNet-LN, the activation between layers of the matrix block is set to a linear passthrough while in MatrixNet-NL the activation is specified to be SiLU. MatrixNet-MC enforces a 2 x 2 block diagonal structure on the learned representations corresponding to the 2-dimensional irreps of S 10 .

We also note the use of SiLU activation in our S 10 MatrixNet model. Due to the generator self-inverse property we need not consider separate generator inverses, and so the odd function requirement given in Proposition 2 is not applicable.

Model Comparison Results Results of the experiments are summarized in Table 1. All variants of MatrixNet achieve a classification accuracy of at least 99% across multiple independent trials. Of note is the inferior performance of the precomputed representation baseline compared to the MLP and MatrixNet on both loss and accuracy metrics, suggesting that there is an advantage to a learnable representation. These results on S 10 order classification validate that group representation learning can aid learning of tasks defined over groups.

Order Prediction over Different Groups In order to demonstrate the flexibility of MatrixNet, we show that MatrixNet can be used to predict order across several different sizes and types of groups. In addition to S 10 , we evaluate MatrixNet on a larger symmetric group S 12 an Abelian group C 11 × C 12 × C 13 × C 14 × C 15 and a product S 5 × S 5 × S 5 × S 5 . These product groups provide a more complex group structure which MatrixNet must learn for successful generalization, with varying representation structure. The results for these experiments are summarized in Table 2.

MatrixNet achieves a high classification accuracy across all additional groups tested. However, accuracy for the Abelian group is lower than the accuracies for other groups tested ( 87% vs 99% ). One explanation for this decrease in accuracy is due to the large number of valid orders of the group. Additionally, due to the structure of finite Abelian groups, many element orders will be underrepresented in random sampling.

5.2 Categorical Braid Action Prediction

In our second experiment, we train models to predict the Jordan-Hölder multiplicities from braid words in the braid group B 3 (see Section 3.4). The task is formulated as a regression task with a mean-squared error (MSE) loss function. The Jordan-Hölder multiplicities are integers, so we evaluate accuracy by rounding the vector entries to the nearest integer. This accuracy is reported as an average accuracy over the three entries of the Jordan-Hölder multiplicities vector. Elements of B 3 are generated by two generators σ 1 , σ 2 and their inverses and are encoded using a signed one-hot encoding. For more details on the dataset generation process and data splits see Appendix B.2.

We additionally performed an experiment to evaluate how well MatrixNet generalizes to unseen braid words longer than those seen in training. For this experiment, we compare against the MLP and LSTM since these were the highest performing baselines.

Baseline Comparison Results We trained all of the models for 100 epochs as all of the models except the Transformer converged within 100 epochs. Despite performance converging much faster than 100 epochs for most MatrixNet variants, we found that additional epochs of training improved the model's invariance to group relations with minimal variations in performance. Table 3 shows the performance of the baseline models and MatrixNet variations at 50 and 100 epochs of training

averaged over 5 runs. The simple MatrixNet model was the worst performing MatrixNet variant slightly outperforming the baseline models at 100 epochs. All other variants of MatrixNet vastly outperform baselines with both MatrixNet-LN and MatrixNet-NL achieving MSE far below the baselines and perfect or near perfect accuracy across all runs. These results confirm the results from the order prediction experiments and demonstrate the advantage of MatrixNet for learning over groups.

Length Extrapolation Results The results in Figure 2 show how the MSE and average accuracy change as input length increases averaged over 10 runs. A single run was omitted from the results for MatrixNet due to training instability. We observe explosive MSE growth for MatrixNet and MatrixNet-MC, but both maintain higher average accuracy than the baselines. The high variance in MSE suggests that both variants are capable of extrapolating despite struggling compared to the other two variants. MatrixNet-LN and MatrixNet-NL both maintain near-zero average MSE as length increases and consequently achieve near perfect average accuracy as length increases.

The discrepancy in extrapolation performance of the MatrixNet variations suggest that MatrixNet-LN and MatrixNet-NL learn better representations than MatrixNet and MatrixNet-MC. To measure this, we compare the relational error of the four MatrixNet variations in Table 4. The group B 3 has only the braid relation σ 1 σ 2 σ 1 = σ 2 σ 1 σ 2 . We calculate the relational error as the Fröbenius norm of the difference || M σ 1 M σ 2 M σ 1 = M σ 2 M σ 1 M σ 2 || . We also compute this distance between two non-equivalent braids σ 1 σ 1 σ 2 , σ 2 σ 2 σ 1 for reference under Non-Relational Difference in Table 4.

The relational error results in Table 4 mirrors the extrapolation performance confirming that representation quality is important for effective generalization. High relational error compounds over longer word lengths hindering generalization whereas low relational error allows MatrixNet to automatically generalize to longer word lengths through invariance to group relations. These results show that MatrixNet, particularly the MatrixNet-LN and MatrixNet-NL variants, is able to learn group representations invariant to the group relations allowing for effective generalization to longer unseen group words.

5.3 Visualizing the Learned Representations

We present some visualizations of the learned representations of the braid group from the highest performing variant, MatrixNet-NL. Figure 2 shows visual plots of the learned representations. In

Extrapolation MSE

Extrapolation Multiplicity Accuracy

:Multiplicity Accuracy

1.0

10

0.8

Average MSE

E

8

0.6

6

4

0.4

2

Averag

0.2

0

0.0

7

8

9

10

7

8

9

10

Word Length

WordLength

MatrixNet

MatrixNet-LN

MatrixNet-MC

MatrixNet-NL

LSTM

MLP

the last two plots, the learned representation for two equivalent words are approximately equal even though this relation is not among the relations r j used in the loss L rel . This shows MatrixNet does indeed generalize over relations, allowing it to generate nearly identical representations for equivalent words.

g1 rep

g2 rep

g191g291g2rep

g2919292g2rep

6 Conclusion

In this paper we have presented MatrixNet, a novel neural network architecture designed for learning tasks with group element inputs. We developed 3 variations of our approach which structurally enforce group axioms and a regularization approach for enforcing invariance to relations. We evaluate MatrixNet on two group learning problems over several finite groups and B 3 and demonstrate our model's performance and ability to automatically generalize to unseen group elements. In future work we plan to develop interpretability analysis methods based on group representation theory to better understand the structure of MatrixNet's learned representations. Understanding the learned representations may provide valuable insights and explanations of the model outputs assisting with generating new conjectures for open mathematical research problems.

Limitations The current work relies on the assumption that the studied group is finitely presented which limits us to discrete groups. However, learned group representations may also be useful for learning over Lie groups. In such case, extending our method will require working with infinitesimal Lie algebra generators. Additionally, while the group axioms are strictly enforced by the model structure, the fact the relations are enforced using auxiliary loss terms means the homomorphism property is not exact. Future work may explore methods of reducing this error.

Acknowledgements This work is supported in part by NSF 2134178. Lucas Laird is supported by the MIT Lincoln Laboratory Lincoln Scholars program. Circe Hsu is supported by a Northeastern University Undergraduate Research and Fellowships PEAK Experiences Award. The authors would like to thank Mustafa Hajij and Paul Hand for helpful discussions.

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 , 2013.

Dian Wang, Robin Walters, and Robert Platt. SO(2) -equivariant reinforcement learning. arXiv preprint arXiv:2203.04439 , 2022.

Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the rubik's cube with deep reinforcement learning and search. Nature Machine Intelligence , 1(8):356-363, 07 2019. doi: 10.1038/s42256-019-0070-z.

Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicitpdf: Non-parametric representation of probability distributions on the rotation manifold. arXiv preprint arXiv:2106.05965 , 2021.

Denis Boyda, Gurtej Kanwar, Sébastien Racanière, Danilo Jimenez Rezende, Michael S Albergo, Kyle Cranmer, Daniel C Hackett, and Phiala E Shanahan. Sampling using su (n) gauge equivariant flows. Physical Review D , 103(7):074504, 2021.

Jonathan Huang, Carlos Guestrin, and Leonidas Guibas. Fourier theoretic probabilistic inference over permutations. Journal of machine learning research , 10(5), 2009.

Luca Falorsi, Pim De Haan, Tim R Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S Cohen. Explorations in homeomorphic variational auto-encoding. arXiv preprint arXiv:1807.04689 , 2018.

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5745-5753, 2019.

Romain Brégier. Deep regression on manifolds: a 3d rotation case study. In 2021 International Conference on 3D Vision (3DV) , pages 166-174. IEEE, 2021.

A René Geist, Jonas Frey, Mikel Zobro, Anna Levina, and Georg Martius. Learning with 3d rotations, a hitchhiker's guide to so (3). arXiv preprint arXiv:2404.11735 , 2024.

Shiang Fang, Mario Geiger, Joseph G. Checkelsky, and Tess Smidt. Phonon predictions with e(3)equivariant graph neural networks, 2024.

Adeel Pervez, Phillip Lippe, and Efstratios Gavves. Differentiable mathematical programming for objectcentric representation learning. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=1J-ZTr7aypY .

Adeel Pervez, Francesco Locatello, and Stratis Gavves. Mechanistic neural networks for scientific machine learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 40484-40501. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/pervez24a.html .

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016.

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International conference on machine learning , pages 2747-2755. PMLR, 2018.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219 , 2018.

Arnab Kumar Mondal, Siba Smarak Panigrahi, Oumar Kaba, Sai Rajeswar Mudumba, and Siamak Ravanbakhsh. Equivariant adaptation of large pretrained models. Advances in Neural Information Processing Systems , 36, 2024.

Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications , 14(1):579, 2023.

Michael Douglas, Subramanian Lakshminarasimhan, and Yidi Qi. Numerical calabi-yau metrics from holomorphic networks. In Mathematical and Scientific Machine Learning , pages 223-252. PMLR, 2022.

Hyunwoo Ryu, Hong in Lee, Jeong-Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: Se(3)equivariant energy-based models for end-to-end visual robotic manipulation learning, 2023.

Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations , 2019.

Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. In The Fortieth International Conference on Machine Learning , 2023.

Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljaˇ ci´ c. Equivariant self-supervised learning: Encouraging equivariance in representations. In International Conference on Learning Representations , 2022.

Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems , 33: 19727-19737, 2020.

Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. In International Conference on Machine Learning , 2022.

Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algebra convolutional network. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 2503-2515. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/ paper/2021/file/148148d62be67e0916a833931bd32b26-Paper.pdf .

Alex Gabel, Victoria Klein, Riccardo Valperga, Jeroen S. W. Lamb, Kevin Webster, Rick Quax, and Efstratios Gavves. Learning lie group symmetry transformations with neural networks. In Timothy Doster, Tegan Emerson, Henry Kvinge, Nina Miolane, Mathilde Papillon, Bastian Rieck, and Sophia Sanborn, editors, Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML) , volume 221 of Proceedings of Machine Learning Research , pages 50-59. PMLR, 28 Jul 2023. URL https://proceedings.mlr.press/v221/gabel23a.html .

Jianke Yang, Nima Dehmamy, Robin Walters, and Rose Yu. Latent space symmetry discovery. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 56047-56070. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/yang24g.html .

Mustafa Hajij, Ghada Zamzmi, Matthew Dawson, and Greg Muller. Algebraically-informed deep networks (aidn): A deep learning approach to represent algebraic structures, 2021.

Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models, 2023.

Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature , 625(7995):476-482, Jan 2024. doi: 10.1038/s41586-023-06747-5.

Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations, 2017.

Joel Janek Dabrowski, YiFan Zhang, and Ashfaqur Rahman. ForecastNet: A Time-Variant Deep FeedForward Neural Network Architecture for Multi-step-Ahead Time-Series Forecasting , page 579-591. Springer International Publishing, 2020. ISBN 9783030638368. doi: 10.1007/978-3-030-63836-8_48. URL http://dx.doi.org/10.1007/978-3-030-63836-8_48 .

Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators, 2022.

Alex Davies, Petar Veliˇ ckovi´ c, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, and et al. Advancing mathematics by guiding human intuition with ai. Nature , 600(7887):70-74, Dec 2021. doi: 10.1038/s41586-021-04086-x.

Michael Artin. Algebra . Prentice Hall, 2011.

Heinrich Maschke. Ueber den arithmetischen charakter der coefficienten der substitutionen endlicher linearer substitutionsgruppen. Mathematische Annalen , 50(4):492-498, 1898.

Camilo Arias Abad. Introduction to representations of braid groups. arXiv preprint arXiv:1404.0724 , 2014.

William Fulton and Joe Harris. Representation theory: a first course , volume 129. Springer Science & Business Media, 2013.

Asilata Bapat, Anand Deopurkar, and Anthony M. Licata. A Thurston compactification of the space of stability conditions. 2020. http://arxiv.org/abs/2011.07908 .

Paul Seidel and Richard Thomas. Braid group actions on derived categories of coherent sheaves. Duke Math. J. , 108(1):37-108, 2001. ISSN 0012-7094. doi: 10.1215/S0012-7094-01-10812-0.

Raphaël Rouquier and Alexander Zimmermann. Picard groups for derived module categories. Proc. London Math. Soc. (3) , 87(1):197-225, 2003. ISSN 0024-6115. doi: 10.1112/S0024611503014059.

Asilata Bapat, Louis Becker, and Anthony M. Licata. q-deformed rational numbers and the 2-calabi-yau category of type. Forum of Mathematics, Sigma , 11, 2023. ISSN 2050-5094. doi: 10.1017/fms.2023.32. URL http://dx.doi.org/10.1017/fms.2023.32 .

Mikhail Khovanov and Paul Seidel. Quivers, Floer cohomology, and braid group actions. J. Amer. Math. Soc. , 15(1):203-271, 2002. ISSN 0894-0347. doi: 10.1090/S0894-0347-01-00374-5.

James E. Humphreys. Reflection groups and Coxeter groups . Cambridge University Press, 2000.

Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondˇ rej ˇ Certík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Štˇ epán Rouˇ cka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science , 3:e103, January 2017. ISSN 2376-5992. doi: 10.7717/peerj-cs.103. URL https: //doi.org/10.7717/peerj-cs.103 .

A More details about the categorical braid group action

The aim of this appendix is to provide a few more details about the particular categorical braid group action that we use in our experiments.

A.1 Sketch of the construction of the category

The category C n we consider is the 2-Calabi-Yau triangulated category associated to the Dynkin graph of type A n . This is an undirected graph with n vertices and n -1 edges arranged in a line, as shown in Figure 4.

•

•

· · ·

•

•

0

1

n

-

1

n

Let Γ n be the Dynkin graph of type A n . Let Γ dbl n be its doubled quiver , which is a directed graph in which each undirected edge of Γ is replaced by a pair of oppositely oriented directed edges, as shown in Figure 5.

•

•

· · ·

•

•

0

1

n

-

1

n

Recall that the path algebra of a directed graph (or quiver) Q over some field k is generated as a free vector space by all possible paths in Q , including the trivial paths at each vertex. The product in kQ is given as follows: let q : a → b be a path and p : c → d be a path. The product pq is equal to zero unless b = c . If b = c , then the product pq is simply the composite path a q - → b p - → d . The path algebra of Q is denoted kQ .

We are interested in a quotient of the path algebra of Γ dbl n called the zig-zag algebra, and denoted Z n . To obtain Z n , we impose the following relations on k Γ dbl n . In what follows, ( i | i ± 1) represents the unique arrow i → i ± 1 .

For each i , set

For each i , set

Consequently, in the zig-zag algebra, any path of length at least 3 is automatically zero, and the only surviving paths of length 2 are back-and-forth loops starting from any vertex (and all possible such loops are set to be equal). The paths of length 0 and 1 remain as-is.

Let Z n -proj be the category of (graded) projective modules over Z n . The category C n is constructed as a differential graded version of the bounded homotopy category of complexes of projective modules over Z n , in which we identify the internal grading shift with the homological grading shift, and consequently also the triangulated shift. For more details, see, e.g. [44, Section 6].

A.2 Important properties of the category

In this section, we record some important properties of the category C n . First, the category C n is triangulated : there is a a triangulated shift functor [1]: C n →C n which is an equivalence. An n -fold composition of [1] is denoted [ n ] .

Denote by Hom( A,B ) the set of morphisms in C n from an object A to an object B . Sometimes we write Hom( A,B ) as Hom 0 ( A,B ) , and further write Hom n ( A,B ) to mean Hom( A,B [ n ]) for any integer n .

The category C n is generated as a triangulated category by the objects P 1 , . . . , P n . That is, the smallest triangulated subcategory of C n (closed under isomorphisms) that contains the objects P 1 , . . . , P n is C n itself. These objects correspond to the indecomposable projective modules of the zig-zag algebra Z n .

Each object P i is spherical . This means that

Remark A.1. The reason for the notation is that the ring of endomorphisms of P i of all possible degrees is isomorphic to the cohomology ring of a sphere (in this case a 2 -sphere).

It is a general fact that any spherical object X of a triangulated category gives an associated functor σ X , called the spherical twist in X (see Seidel-Thomas [45] for more details.) The functor σ X is an auto-equivalence; that is, it has an inverse equivalence σ -1 X such that compositions in both directions are isomorphic to the identity functor.

In particular, we obtain equivalences σ P i : C n →C n .

A.3 The Jordan-Hölder filtration

The category C n has a bounded t -structure. This means that there is an abelian subcategory A n ⊂ C n , such that every object X ∈ C n has a unique finite filtration whose factors lie in A [ i ] for decreasing values of i . This filtration is called the cohomology filtration . In fact, this abelian subcategory A n is also generated by the objects P i : it is the extension-closure of the objects P i . Moreover, the objects P i are simple objects of A n .

First consider any object X ∈ A n . The category A n is a finite-length abelian category. It is a standard fact that X has a Jordan-Hölder filtration whose factors are simple objects in A n , namely the objects P i .

Now consider a general object X ∈ C n . We first consider the cohomology filtration of X , with factors Y j ∈ A [ j ] . For each Y j ∈ A [ j ] , we consider its (shifted) Jordan-Hölder filtration, which breaks Y j up further into copies of P i [ j ] . Putting these two together, we obtain a finer filtration of the object X , which we also call the Jordan-Hölder filtration of X .

The factors of this Jordan-Hölder filtration are shifted copies of P i for all i . Thus we can count the number of occurrences of each P i in the Jordan-Hölder filtration of X , and it is well-known that these counts do not depend on the specific choice of the Jordan-Hölder filtration.

A.4 The action of the braid group

Recall the spherical twist functors σ P i : C n →C n . A remarkable observation of Seidel-Thomas [45] is that these functors (weakly) obey the relations of the n -strand braid group. That is, we have isomorphisms of functors

̸

whenever | i -j | = 1 , and whenever | i -j | = 1 .

Since these are precisely the relations of the group B n , we obtain an action of B n on the objects of the category C n .

A.5 Open problems and future directions

Consider the following broad question. Given an object X of C n and a braid g ∈ B n , can we relate the Jordan-Hölder multiplicities of β ( X ) to the Jordan-Hölder multiplicities of X ? Stated more simply, can we compute the Jordan-Hölder multiplicities of β ( P i ) for any i and any β ?

Answers to this question are known in some cases, and remain open in others. For instance, a complete answer was obtained by Rouquier-Zimmermann [46] for the 3 -strand braid group B 3 acting on C 3 . This answer was rediscovered and refined in terms of more general filtrations (Harder-Narasimhan filtrations) in [44] and [47].

It is also known that if β = σ ℓ P j for some ℓ , then the limit as ℓ →∞ of the counts of β ( X ) can be obtained, up to a common scaling factor, by computing the sum of the dimensions of Hom m ( P j , X ) for all m [47].

However, for the vast majority of values of n and most of the elements of the braid groups B n , we do not have a good answer to this question. While there is vast potential for future work, we write down a few specific open problems.

Generalise the Rouquier-Zimmermann theorem (and its corresponding versions in [44] and [47]) to larger values of n .

We can compute a finer version of Jordan-Hölder multiplicities: split up the number of occurrences of each P i by degree shift. That is, record the number of occurrences of P i [ d ] separately for every possible d . This information can be encoded in a polynomial in one variable in q ± 1 , in which the coefficient of q d is the multiplicity of P i [ d ] .

Generalise the Rouquier-Zimmermann theorem in this setting to larger values of n .

By using a more refined version of Jordan-Hölder multiplicities, known as HarderNarasimhan multiplicities, we observe that the possible Harder-Narasimhan factors of any object of the form β ( P i ) are highly constrained, and satisfy some very nice combinatorial properties.

This constraint can be explicitly described for any B n via a geometric model (due to Khovanov-Seidel [48]) for objects in the category C n . Nevertheless, the relationship of these constrained sets with the action of B n is mysterious. For example, given an object X , is there an algorithm to write down a braid that will send X to an object with a desired set of Harder-Narasimhan filtration factors?

Can we use the combinatorial structure mentioned above to algorithmically write down combinatorial actions of braid groups on simpler sets? What properties do these actions satisfy?

All of the categorical constructions described in this paper also go through for more general versions of braid groups, known as Artin-Tits braid groups. All of the questions above remain open for all but the simplest cases of Artin-Tits groups.

B Dataset and Model Parameter Details

B.1 Symmetric Group Dataset

We generated a dataset of 500 , 000 samples consisting of words of the free group F 10 , and labels corresponding to their order as elements of S 10 . The first step of dataset generation was to fix a maximum word length chosen such that it is possible to sample every element of S 10 . For a generating set corresponding to adjacent transpositions of elements in S n , this longest word will be of length n ( n -1) 2 [49], and for S 10 choose our maximum length to be 64. We define a uniform distribution on the set of generators σ 0 , σ 1 , ..., σ n -1 where σ 0 = 1 and all other σ i = ( i i + 1) . Informally, our generating set consists of adjacent transpositions of the form ( i i +1) along with an identity generator. The presence of the identity generator adds variability to word length while enforcing identity invariance. Sample order labels in S 10 are computed using the SymPy package [50].

While we do not strictly enforce a separation of elements between training, test, and validation sets, it is statistically unlikely to have any significant overlap between the splits. Recall that | S 10 | = 10! = 3 , 628 , 800 . Our dataset of 500 , 000 samples therefore covers at most 13 . 7% of S 10 , implying the likelihood of significant overlap between partitions upon reduction is very low. Moreover, because we are sampling unreduced words from F 10 of length 64, there are 10 64 possible words we could sample from, making the probability of direct overlap between partitions effectively zero.

B.2 Categorical Braid Action Experiment Details

Dataset Generation An initial dataset of Jordan-Hölder multiplicities for braid words up to length 6 was provided. We implemented a state automaton algorithm from [47] to generate additional examples for longer braid words. This method was compared against the Jordan-Hölder multiplicities of the initial dataset to verify correctness.

Baseline Comparison Experiment Details The baseline comparison dataset consists of 47,831 examples with braid words up to length 8. The data was split into 60% training data, 20% validation data, and 20% test which were fixed for all models. All of the models trained using an Adam optimizer with a learning rate of 1 e4 and a batch size of 128. The chosen parameters for the models are:

MatrixNet: Single channel 14 × 14 matrix size

MatrixNet-LN: Single channel 10 × 10 matrix size, 128 dimensions for linear network in the matrix block

MatrixNet-MC: 3-channel 8 × 8 matrix size

MatrixNet-NL: Single channel 10 × 10 matrix size, 128 hidden dimensions and a tanh non-linearity between linear layers of matrix block

MLP: 3-layer MLP with 128 hidden dimensions for each layer and ReLU activation functions followed by a single linear layer output.

LSTM: 6 LSTM layers with 16 dimensional input embeddings and 32 hidden dimensions followed by a 2-layer MLP classifier with 64 hidden dimensions and ReLU activation.

Transformer: 3 transformer layers with 4 attention heads, 16 dimensional embeddings and 32 hidden dimensions. Used mean pooling and a single linear layer output.

All of the MatrixNet architectures used a 2-layer MLP with 128 hidden dimensions and ReLU activation to compute output after the matrix block.

Length Extrapolation Experiment For the generalization experiment we generated a dataset of all braid words up to length 7 which was split into 80% training and 20% validation. We also generated three separate test datasets of 10,000 examples each with braid words of length 8, 9, and 10 to evaluate how performance degrades over increasing length. The models were trained for 100 epochs on the training and validation sets and then tested on the three test sets.

Regularization Details All MatrixNet architectures were trained using the regularization loss defined in Section 4.3. We chose the Frobenius norm for the norm and used the braid relation σ 1 σ 2 σ 2 = σ 2 σ 1 σ 2 and the inverse σ -1 1 σ -1 2 σ -1 1 = σ -1 2 σ -1 1 σ -1 2 . The regularization term was added to the loss every 10 training batches.

Hardware Details All of the categorical braid action experiments were run on a machine with a single Nvidia RTX 2080 ti GPU.

B.3 Length Interpolation Results

We also performed a length interpolation generalization experiment to test how well each model generalizes to braid words that are shorter than the maximum length seen during training. The results are not presented in the main body of the paper as all models generalize to shorter braid words.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide bulleted claims and contributions in the introduction. We provide experimental results and proofs in the methods and experiments sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The Limitations section details the assumptions of the model and the limitations of regularization as a method for enforcing group relation invariance.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theoretical results are stated and proven in the Methods section as propositions and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our architecture and regularization method are described in the Methods section. We also provide a summary of relevant theoretical background in background section and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We plan to open source our code upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe the model parameters, dataset generation and splits, and optimizer in the appendix. We also give context for the experiments in the experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We have not performed these exact experiments enough times to produce error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The hardware used is reported in the appendix. Our model and data do not require significant compute and are small enough to be run on most machines.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper focuses on deep learning in a mathematical context. There were no human participants or personal data involved in this research.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work focuses entirely on mathematical problems with no broader societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not use any pretrained models and generate our own data. The data is not sensitive or private in nature.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all packages used and acknowledge compute resources used during this research. We do not use any preexisting assets in this research.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We are not releasing new assets as part of this submission. We will provide documentation and code upon publication.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not crowdsource data or experiments in this research. There are no human participants involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: There are no human subjects involved in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors study the question of what feature representations to use for learning tasks with inputs coming from a symmetry group. They propose MatrixNet, a neural network architecture that learns matrix representations of group element inputs instead of using predefined representations.\n\nThe main contributions are as follows:\n1. Formulation of the problem of learning over groups as a sequence learning problem in terms of group generators with invariance to group axioms and relations.\n2. The proposed neural network architecture, MatrixNet, achieves higher sample efficiency and generalization over several more general baselines in prediction tasks over the symmetric group and Artin braid group.\n3. The matrix block method for constraining the network to respect the group axioms and an additional loss term for learning group relations.\n\nSTRENGTHS:\nThe strength of the paper is building up links among group reprsentation theory, learning tasks and neural networks, more specifically:\n1. Formulation of the problem of learning over groups as a sequence learning problem in terms of group generators with invariance to group axioms and relations.\n2. The matrix block method for constraining the network to respect the group axioms and an additional loss term for learning group relations.\n\nWEAKNESSES:\nThe motivation is not so clear to me. The applications of group theory are everywhere in the real-world. Thus, how to choose a good representation for the group which is associated with the learning task is more important. In this paper, authors formulate this problem but do not give a clear answer (see my questions below), and authors focus on solving a more abstract group theory question instead which is already studied a lot by mathematician. \n\nI didn't see any insight behind the paper. For the experiment, I think it only solved a mathematical problem at simple setting which is far from what we expect. For example, how the model works when the group's order become larger?\n\nQUESTIONS:\n1. The motivation is to solve a mathematical problem or real-world learning task? I hope it can help me understand the paper clearly and inprove the writing.\n\n2. In subsection 4.1, authors formulate the problem. In line 166, can $f$ be seen as a representation of $G$ or not? If it is a representation, can you write a explicite expression as an example? Since if it is a representation, the target domain $R^{c}$ may not be a linear space which makes it unclear for me.\n\n3. Can you show more experiment results? Such as larger symmetric group? Also I didn't understand the meaning of predict the order of the group element? I think we should determine it instead of estimating it? Maybe I did't get your point, but I am open to discuss.\n\nLIMITATIONS:\nsee the weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes MatrixNet, a method to learn group representations such that they are optimized for a certain task of interest using a neural network. , The neural network takes in a group element in the form of a sequence of generators that compose to form the group element and forms an intermediate output that contain the matrices that are the group representations for each of the generators. The generators can them be multiplied to form the group representation which is then fed to further neural layers that are task-specific. The network is trained so as to (approximately) satisfy the constraints of group representations using architectural constraints and the loss function. Experiments on predicting the order of a element of the symmetric group and Jordan-Holder multiplicities for the 3-strand braid group.\n\nSTRENGTHS:\n1. The paper present new ideas for learning on groups. Design of architectures is clever and done well with good mathematical justification. The architectures provably output group representations.\n\n2. Experiments show that the proposed architecture leads to learning group representations that lead to better results compared to just using \n\n3. The paper also does a good study of the possible variations of MatrixNet and presents experimental results for them.\n\nWEAKNESSES:\n1. One of the weaknesses is that the parts about the braid group are very difficult to follow and probably need a lot more background for readers and attendees for this conference. This includes myself and I was not able to fully follow the details or why that experiment is important.\n\n2. The experiments are also a little weak in my opinion. It was not clear to me how the learned representations were different from precomputed ones and why they were better for a given task. Also, what happens when the size of the group representations is larger. Perhaps for the first experiment, it would be nice to see the results as a function of the size of the group representation. I am not sure if the proposed architecture and learning mechanism can learn good representations when the size is large. This is important to address, in my opinion.\n\nQUESTIONS:\nNo additional questions.\n\nLIMITATIONS:\nYes, authors have listed the limitations of the current submission and suggested directions for future research.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies feature representations of a group element for supervised learning. It considers a regression task where an input is a group element g of a finite group and a target is some label. Firstly, g is decomposed into a sequence of generators (g_1, ..., g_n) such that g = g_1 \\circ ... \\circ g_n. Next, each generator is mapped by a trainable matrix embedding W: Gen(G) --> R^{n \\times n} followed by the matrix exponential so that we can get a group representation M_i = exp(W(g_i)) ∈ GL(n). Then their product M_1 \\circ ... \\circ M_n becomes the feature. Similar variants are also proposed. The performance is evaluated on two synthetic tasks: the order prediction of the symmetric group S_10 and the action prediction of the Braid group B_3.\n\nSTRENGTHS:\n1. The paper is well written. Technical details are clear enough. \n1. The idea to construct a feature representation of a group element as a learnable parameter is interesting and it is a possibly promising direction.\n\nWEAKNESSES:\n1. Limited applicability to real tasks. To the best of my knowledge, regression (or classification) of group elements is practically important for continuous groups such as SO(3) (e.g., pose estimation). However, the current approach is only applicable to finite groups. Also, I think the tasks conducted in the experiments are not directly bridged to real problems, and I feel uncertain about how the proposed method is practically valuable. \n1. It is unclear which component significantly contributed to the final performance gain. The proposed method consists of (at least) two parts: the decomposition of g into generators and a trainable representation of a generator. The current experiments are not designed to evaluate them separately. \n1. The experiments are not convincing enough—they are relatively small scale, use synthetic tasks only, and have less variety (two tasks).\n\nQUESTIONS:\n1. Given g, is its decomposition into the generators always uniquely determined?\n1. In the experiment of 5.1, what will happen when we employ the group decomposition while using the fixed representation? I mean, the feature of g is given as concat[\\rho(g_1), ..., \\rho(g_n)] where g = g_1 \\circ ... \\circ g_n and \\rho(g_i) is some representation of g_i (e.g. irreducible rep).\n\nTypo: \n* One of M_{g_ik^-1} would be {M_{g_ik}}^{-1} in the equation below line 203.\n\nLIMITATIONS:\nLimitations are adressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The authors use neural networks to learn group representations. A group element is represented by its generators formatted as a sequence of learned matrix representations. These generators are mapped to a single matrix representation of the group element via the Matrix Block which enforces group axioms. The resulting feature is used for downstream tasks on groups. The authors show that the proposed architecture MatrixNet successfully predicts group element orders for S_10 and Jordan-Holder multiplicities for braid group B_3. Further analysis on word length extrapolation and visualization demonstrates the superiority and usefulness of the approach.

STRENGTHS:
1. In Table 1, the faster convergence of MatrixNet potentially indicates that the proper architectural constraints provide good inductive bias for learning group representations and solving downstream tasks over groups. It’s good to see how one can explicitly build in these constraints in the problem of group representations and that it works better than a less constraining MLP without domain-specific inductive bias.
2. The paper is well-written.

WEAKNESSES:
1. It’s unclear why naive MatrixNet and MatrixNet-MC cannot extrapolate to longer word length. 
2. Empirical evaluations are a bit limited. It might be helpful to evaluate groups with different properties (see questions).

QUESTIONS:
In the matrix block, is it possible to introduce commutative operations between different matrix representations of group generators if the input group element comes from an abelian group. It seems possible to introduce further architectural constraints for specific groups.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11