MatrixNet: Learning over symmetry groups using learned group representations
Authors: Lucas Laird
Circe Hsu
Asilata Bapat
Robin Walters
Venue: NeurIPS 2024 poster (2024)
Keywords: Group Theory
Representation Theory
Feature learning
geometric deep learning
homomorphic
Primary area: deep_learning_architectures
TL;DR: When learning over symmetry groups, a learned group representation of group elements works well as a feature  representation.

Abstract: Group theory has been used in machine learning to provide a theoretically grounded approach for incorporating known symmetry transformations in tasks from robotics to protein modeling. In these applications, equivariant neural networks use known
symmetry groups with predefined representations to learn over geometric input data. We propose MatrixNet, a neural network architecture that learns matrix representations of group element inputs instead of using predefined representations. MatrixNet achieves higher sample efficiency and generalization over several standard baselines in prediction tasks over the several finite groups and the Artin braid group. We also show that MatrixNet respects group relations allowing generalization to group elements of greater word length than in the training set. Our code is available at https://github.com/lucas-laird/MatrixNet.

MatrixNet: Learning over symmetry groups using learned group representations
Lucas Laird
Circe Hsu
Khoury College of Computer Sciences Northeastern University
Department of Mathematics Northeastern University Boston, MA 02115
Boston, MA 02115
laird. wwwortheastern...mmm edu
hsu circe@northeasterr.
Asilata Bapat
Robin Walters
Mathematical Sciences Institute
Khoury College of Computer Sciences
Australian National University
Northeastern University Boston, MA 02115
Canberra, Australia asilata. bapat @anu. edu. au
r. wal.teershhorrhhasteem. edu
Abstract
Group theory has been used in machine learning to provide a theoretically grounded approach for incorporating known symmetry transformations in tasks from robotics protein modeling. In these applications, equivariant neural networks use known symmetry groups with predefined representations to learn over geometric input data. We propose MatrixNet, neural network architecture that learns matrix representations of group element inputs instead of using predefined representations. MatrixNet achieves higher sample efficiency and generalization dard baselines in prediction tasks over the several finite groups and the Artin braid group. We also show that MatrixNet respects group relations allowing generaliza- tion to group elements of greater word length than in the training set. Our code is available https: //gi thub. com/ lucas- aucas-laird.. lairddMaaaiiiiee Matrrixet..
several stan-
Introduction
The choice of representation for input features is key design aspect for deep learning. While wide variety of data types are considered in learning tasks, for example, sequences, sets, images, time series, and graphs, neural network architectures admit only tensors input. Various methods exist for mapping different input types to tensors such as one-hot embeddings, discretization of continuous signals, learned token embeddings of image patches words [1, 2]. adjacency matrices [3], positional encodings [1], or spectral embeddings [4].
In this paper consider the question of what feature representations to for learning tasks with inputs coming from a symmetry group. There are many examples of tasks defined over symmetry groups, such as policy learning in robotics [5], reinforcement learning [6], pose estimation in computer vision [7], sampling states quantum systems [8], inference over orderings of set [9], and group- theoretic invariants in pure mathematics. Past work has typically employed fixed representations chosen from among the known representation theory of the group. Representation theory is the branch of mathematics concerned with classifying the set of representations of group G which, in this context, refers to homomorphic realizations of a group in terms of n x n matrices. For groups with well understood representation theory, for example, the symmetric group S, SO(n), this provides a ready set of embeddings for converting group elements into tensors for use in downstream
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

models. Trial and eπrоr or topological analysis has shown that the choice of group representation is critical for learning [10, 11, 12, 13].
Instead of using predefined group element representations, we propose to learn feature representations for each group element using learned group representation. That is, we learn to map group elements to invertible n X matrices which respect the symmetry group structure. There are several advantages this strategy. First, unlike predefined group representations, learned representations allow the model to adapt to the given task and capture relevant information for the learning task. Second, learned representations provide reasonable correlations between the learned features for different group elements since they incorporate algebraic structure into the model. This structure is encouraged using free group generators and group relation regularization. Third, relative to learned vector embeddings, learned matrix representations are very parameter efficient for encoding different group elements, reducing the problem to learning only representations of group generators; using sequence encoding, our method is able to generalize to combinatorially large or even infinite groups. Fourth, the learned representation admits analysis in terms of the irreducible subspaces of the generators, giving insight into the model's understanding of the task.
We integrate our learned group representation into a specialized architecture, MatrixNet, adapted for learning mappings related to open problems in representation theory. We compare against several more general baselines on order prediction over finite groups and estimating sizes of categorical objects under an action of the Artin braid group. Through our experiments we observe that our approach achieves higher sample efficiency and performance than the baselines. We additionally show that MatrixNet's constraints allow for it to generalize to unseen group elements automatically without the need for additional data augmentation.
Our contributions include:
Formulation of the problem of learning over groups as a sequence learning problem in terms of group generators and invariance to group axioms and relations,
The MatrixNet architecture, which utilizes learned group representations for flexible and efficient feature learning over symmetry groups,
The matrix block method for constraining the network to respect the group axioms and an additional loss term for learning group relations,
Empirical validation showing MatrixNet outperforms baseline models a task over the symmetric group and a task over the braid group related to open problems in mathematics.
2 Related Works
Mathematically Constrained Networks Many deep learning methods incorporate mathematical constraints better model underlying data. For instance, the application of graph neural networks [3, 14] to problems with an underlying graph structure has led to state of the art performance many domains. Deep learning models have also been designed for tasks with known mathematical formu- lations by parameterizing components of algorithmic solutions as neural networks and leveraging their structures for more efficient optimization [15, 16]. More broadly the field of geometric deep learning [17] advocates for building neural networks which reflect the geometric structure of data in their latent features. When symmetries are present in data, group equivariant neural networks [18, 19, 20, 21] can enable improved generalization and data efficiency by incorporating known symmetries into the model architecture using the representation theory of groups. Our method also utilizes group representations, but unlike equivariant neural networks, we use learned as opposed to fixed representations. We also focus on modeling functions defined on the group as opposed to between representations of the group
Learning Structured Representations Instead of using predefined representations as inputs, many methods seek to learn mathematically structured representations from data. This idea has been applied in physics [22, 23), robotics [24], world models [25], self-supervised learning [26. 27], and unsupervised disentanglement [28]. Park et al. [29], for example,
combination of
'The term representation is used in both deep learning and mathematics with related but different meanings. We will disambiguate the former a feature representation and the latter group representation.

---

learned symmetry and equivariant constraints to map images to group elements or vectors in group representations. Similar techniques are used in symmetry discovery where the underlying group symmetry is not known a priori [30, 31, 32]. Yang et al. [32] use a generative model to learn latent representations with a linear group action in order to find unknown group symmetries in data. Here, in contrast, we start with a known group and learn matrix representation. Hajij et al. [33] propose algebraically-infore presentation. We also learn a group action but consider linear group actions and the learning signal comes not only from the group presentation but also a downstream task.
neural networks which learn a non-linear group action from a group
Deep Learning for Math Recent work has shown deep learning can be useful for providing examples, insight, and proofs related to open problems in mathematics. One approach is the application of language models to mathematics [34], which has the benefit of flexibility in how the model is prompted yet is difficult to interpret and prone to errors. Meanwhile significant work has been done in the area of symbolic regression and automated theorem proving [35]. Other work applies deep learning to the direct modeling of partial differential equations [36, 37]. These methods can perform exceptionally well on real-world data [38], but suffer when trying to interpret predictions made for the purposes of mathematical research. Another avenue involves training graph neural networks on mathematical data such as group or knot invariants and analyzing the learned representations to see which features significant as way provide intuition to mathematicians [39]. Our method also uses structured inputs and learned features, but uses a sequence model and learned group representation instead of graph neural network with learned node attributes.
3 Background
3.1 Group Theory
Groups encode systems of symmetry and have been used in machine learning to build invariance into neural networks to various transformations [18]. Formally, a group G is set equipped with an associative binary operation o: G G
which satisfies two axioms: (1) there exists an identity E G. such that g = log 9. (2) for each g E G, there exists an inverse g E G such that g o 9 = g o g = Examples of groups include finite groups such as the dihedral group D4
element 1
which gives the symmetries of the square, SO(3), the continuous group of 3D rotations, or (Z, the infinite discrete group of integer shifts.
Since groups may be combinatorially large or infinite, it is essential to encode their elements and compositional structure in a suceinct way. For many discrete groups, generators and relations provide such a description. A set of elements S = {gi..... 9n S G are called generators if every element of G can be written as a composition of 91, 91 ..... In. In In general, each element of G may be written in many different ways in terms of the generators; this non-uniqueness is encoded using relations. A set R (r1...... :.....mm } of words in the generators S are relations for G if each word r; is equal to the identity in G and if generates the entire set of words equal to identity under composition and conjugation. The generators and relations of a group taken together are called a presentation and denoted = (g)...., In. ri,,.., rm). For example D4 = (r, S rd = f2 2== frfr)- Due to relations, group elements do not have a unique word representation. For example fr/ p3 r all represent the same group element. By convention relations are sometimes stated as equalities instead of single group elements, for example frf- = p3. The free group Fs- = (II...., !n,) is defined to have no relations except for those coming from the two group axioms above.
of
An important notion for our discussion is the order of an element. If E G then the order of denoted [g). the smallest l such that g* = e. (For non-finite groups, k may be infinity). The order of the group |G] is simply the number of elements in the group. For any 9, Lagrange's 's theorem implies that [g] is a divisor of |G] [40], which restricts the possible orders an element may take.
3.2 Representation Theory
Abstract group presentations are difficult to work with in many settings. Group representations map group elements to invertible matrices such that composition of group elements corresponds to matrix multiplication. This gives the group a natural action on vector spaces and allow for analysis of the group using linear algebra. Formally, representation of a group G' is group homomorphism
3

---

Φ : G. + (L to the group of invertible x matrices. That is 0(91 92) = (g1) ( (g2). A property of that (1) = Luxn and ((g ) = ((g))= Due to the homomorphism property, is sufficient to define Φ for generators of the group G. For example, 2 x 2 representation of D4 is given by mapping r to a т 7 matrix and f to a reflection over the rraxis.
The representations of many groups are well classified. This provides ready source of tensor representations for group elements to use as inputs for neural networks. For example, for finite groups, by Maschke's Theorem [41], representations may be decomposed into irreducible representations. That is, there exists basis such that the representation matrices are all block diagonal with the same block sizes and these blocks cannot be further subdivided. The irreducible representations may then be further classified by computing character tables.
3.3 Symmetric and Braid Group
The braid group on n strands B, has presentation
(T,, O 0,a ooo, for -- j| 2 0,04+10, 0441000411 for f 2). (I) The braid group intuitively represents all possible ways to braid set of n strands. The generators d; correspond to twisting strand i over i + 1 and o; is the reverse. twisting strand i + 1 over i. It is defined topologically as equivalence classes up to ambient isotopy of n non-intersecting curves R$ connecting two sets of fixed points. The braid group is infinite and though some representations are known, they are not fully classified [42]. The braid group has important connections to knot theory, mathematical physics, representation theory, and category theory.
The symmetric group on n elements, denoted Sn, defined as the of bijections from {1..... n) to itself. It is also quotient of the braid group B,, and has presentation similar to Eqn. 1 but with additional relations of for < S 1. Here o; the transposition (i i+1). The symmetric group has finite order [S,] = n!. Representations of the symmetric group are well understood. The irreducible representations are parameterized by partitions of n. For more details, see [43].
3.4 Categorical Braid Actions
One current active research problem in mathematics concerns actions of braid groups on categories. category is an abstract mathematical structure that has objects and maps or morphisms between objects, satisfying several coherence axioms. For example, the category Vecta has objects which are real vector spaces and morphisms which are linear maps. Functors take objects to objects and morphisms to morphisms between the corresponding objects, satisfying several compatibility conditions. The action of a group G on category C means that each group element E G is associated with an invertible functor Fg C - C, such that any relation æ = y in the group implies that the corresponding functors F- and F. are naturally isomorphic.
maps between categories that
Given a category on which a braid group acts, mathematicians are interested in measuring how objects grow under repeated applications of elements of the braid group. The "size" of object in category may be measured using a tool called HölderrHölder filtrations. For example, Bapat et al. [44] attempt measure growth rates of objects in specific category C, under repeated applications of certain twist functors dr, which define an action the braid group B, on C,. Each object in the category has Jööder-Hölder filtration giving unique vector in 2 of Hööder multiplicities. For more details see [44] and Appendix A.
However, they only able to compute the action in certain cases and a simple formula is elusive. A complete description of the fordan-Höldes multiplicities after applying combinations of ap, to one of the generating objects is only known for = 3; that is, the case of the 3-strand braid group B3. Understanding how these multiplicities evolve under repeated application of braids is a challenging open research problem in mathematics.
4 Methods
We formulate the problem of learning a function on a symmetry group as a sequence learning problem using a presentation of the group in terms of generators and relations. We propose MatrixNer which predicts the label using a learned matrix representation for the group. The homomorphism property of the representation is enforced through combination of model design and an auxiliary loss term.

---

4.1 Problem Formulation
We consider task functions of the form f: G Ee where Gs finite or discrete group and the output space IR' may represent either a regression target or class label. While such tasks appear in computer vision, robotics, and protein modeling, where neural models may lend additional examples and insight towards proving theorems.
are particularly interested in problems mathematics
To efficiently represent group elements in infinite large groups, consider presentation of G = (S R) terms generators = [9 In and relations R = (ri...., .m.. Model inputs g G are represented by sequences of generators (.......: Dic) where g = gi, o gi, is of
arbitrary length l 2 O and S ij S n. For convenience, we can include the identity = e among the generators pad sequences without changing the group element.
Since a single group element may be represented by different sequences, is critical for the model fe be invariant to both the group axioms and relations. That is, we desire
folgt....... Dic e, 9.2411.... 9bs fo ................................ Giz Ioet
giz
(Identity),
(2)
fo fo 9b 91. 9j 9.2411.... 9b. = felg...... 91z 914411
9ic) 9
(Inverses),
(3)
folg
Pir 5, 914411....
9h.) = folg Pi. 964413......
9iz)
(Relations).
4.2
MatrixNet
01020 S.
Co)(i.))
Sequential Multiplication
Matrix Block
Preduct
Flatiomal
d1
W
Manris
Matris
Task
Eap
Model
Wo.
Figure 1: Schematic MetrixNet for predicting order elements of S3. Input generators σi and d2 are mapped to learned representations and sequentially multiplied to provide matrix representation of group element g. The order is then predicted by the task model which an MLP.
We propose MatrixNet (see Figure 1), a neural sequence model which models functions on a group G' by taking input sequences of generators for G. It achieves invariance group axioms and relations through combination of built in constraints and an auxiliary loss term. The key part of the MatrixNet architecture is the matrix block which takes group generator gi as input and outputs an invertible square matrix representation M The matrix representation M, for an arbitrary group element g defined as the product matrix representations of generators needed to generate g. This matrix representation is then flattened and passed to a downstream task model (such as an MLP) which computes the label. what follows, we give more detailed description of the matrix block and some variations on the architecture.
Signed one-hot encoding We define the signed one-hot encoding, modified version of the traditional one-hot encoding, for encoding group generators used as input to MatriXNet. Let (9f) - ,..,,.. be sequence of generators where 0 < ig < and ex E [+1]. The signed one-hot encoding encodes each generator S vector Ug4 EE; = ...... 0, Ex. ....... is in the ith entry. The identity element 9u 1 is mapped to the zero vector it 0 E IR.. The signed one-hot encoding is chosen since it intuitively relates a generator and its inverse as Vg-1
. 0] g which
-g-
4.2.1 Matrix Blocks and Learned Matrix Representations
The matrix block is designed as a parameterized representation of the free group Fs. that is, a homomorphism & Fs + GL(n) which satisfies 3 properties: (1) the matrix ((g) M, is invertible, (2) ((1) = Inxns and (3) if ((g) My then (1g-1) M,-1 In what follows Ugs is

---

the signed one-hot encoding of the generator 9r, and W is a learnable parameter matrix. For group element g 9i ... 9i, the matrix block is defined
A Reshape (Wo (Wugie)
Ma. MatrixExp(444
Mg Mag, Mg M My..
where Reshape reshapes vector in E= into square X matrix. Proposition 1. Matrix Block defines representation of the free group.
Proof. Property (1) is satisfied since the outputs of matrix exponential are invertible. Properties (2)
and (3) follow from 9ig
99 21 O, and properties of the matrix exponential,
M1 MatrixFxppOOxmm) = Taxn
M -1 MatrixExp(-A44 M -1.
9
91
4.2.2 Variations of Matrix Block
We now present some variations of this simple design that satisfy the group homomorphism properties.
Linear Network (MatrixNet-LNN) The first variant replaces the single parameter matrix W with linear network that has two parameter matrices W1, W2. The linear network matrix block changes the computation of the intermediate A matrix to:
A = Reshape W2 (29..))
This is still a linear function of Ugg meaning Proposition still holds by the same argument.
While this variation does not give increased expressivity over the original formulation of the matrix block, the linear network can change the optimization landscape leading to different performance in practice. This variation is called Net..XNet-LNN
The second variation introduces an element wise odd non-linear function f. That f(-z) =f(()) ) as with tanh. The non- lon-linear matrix block modifies
Non-Linearity (MatrixNet-NL)
A = Reshape( (W1
Proposition 2. Non-linear matrix block defines representation of the free group.
Proof. Property (1) satisfied by the same argument in Proposition Since an odd func-
tion, f(W1v)) = (0) and ( Reshape(f (W-1,,,)))) = AA. Therefore Properties (2) and (3) are satisfied by the same argument in Proposition 1.
This variation can be combined with the first Proposition 2 holds for linear transformations of Ak- ( )). Unless otherwise noted we set f tann.
That is, Aş-
Block Diigooaa (MatrixNet-MCC) The third variation inspired by the decomposition of repre- sentations into irreducible representations. For certain classes of groups such finite groups, every representation decomposes such that the matrices M, have consistent block diagonal structure in some basis. Thus to learn arbitrary representation of the group G, it suffices lear block diagonal representation assuming the blocks are large enough.
This variation learns intermediate n, X n, matrices Au; which are combined to form block- diagonal n x n matrix Aş where n = Σ/= nj. The block bock-diaagoall matrix block defined with
Ag = Reshape(W) (W for = 1 to
A = BlockDiag AR, , Anr)

---

Note that MxtrixExp and matrix multiplication preserve the block structure. If the sizes n2 are fixed to all be equal, this formulation can be implemented as a multi-channel matrix block where both A and Mg, are l x nj x ni tensors with e channels. Fach AR, is calculated identically to As in the original matrix block formulation, and BlockDiag is linear, so Proposition still holds. This variation is also compatible with the previous two variations. our experiments, we implement 3-block version called MatrixNet-MC with single linear layer and no non-linearity.
4.3 Enforcing group relation invariances
The matrix block is constrained to learn a representation of the free group Fs. As a consequence MatrixNet will satisfy (2) and (3) desired. However, most groups have relations which cannot be enforced through a simple weight-sharing scheme used in equivariant architectures [19]. We propose to learn the relations through a secondary loss which measures how closely the representation respects the group relations. More concretely, let G R)) be a group with relations r; E R. The loss is:
Lyel E(IM, Inxall)
NER
Since MatrixNet learns a representation that is invariant to group axioms, it is sufficient to sum over only {r and not all compositions of relations. For architectures which do not respect the free group structure, the relations r; alone may not guarantee that all equivalent words have identical feature representations, requiring potentially combinatorial amounts data augmentation. This allows MatrixNet to both efficiently learn group relation invariance and simply verify this invariance without any data augmentation.
5 Experiments
We use two learning tasks evaluate the four variants of MatrixNet and compare our approach against several baseline models. We use several finite groups on well understood task as initial test to validate our approach and then move on to an infinite group, the braid group B3, on a task related to open problems. As baselines, and LSTM and Transformer models on longer sequences. Baseline model parameters were chosen so all of the models have approximately the same number of trainable parameters.
compare to MLP for fixed maximum sequence length
5.1 Order Prediction in Finite Groups
The first task is to predict the order of group elements in finite groups. Elements of finite groups are input as finite sequences of generators as described in Section 3.3. The typical efficient algorithm for computing the order involves disjoint cycle decomposition, making order classification a non-trivial task. See Appendix B.1 for more details the sampling method and data splits.
Models Compared We compare MatrixNet variants and three baselines for order prediction in S (1) MLP with 3 layers with hidden dimension 256 and SiLU activations, (2) Fixed representation input to a 2-layer classifier MLP with 256 hidden dimensions and SiLU activations, (3) LSTM input to a 2-layer LSTM with 256 hidden dimensions with subsequent MLP classifier using SiLU activations, (4) MatrixNet-LAN with 2-layer 256 hidden dimension matrix block and classifier network with SiLU activations. (5) MatrixNet-MC with 2x:2 matrix block size over 5 matrix channels and classifier network with SiL.U activations. (6) MatrixNet-NI. with a 2-layer 256 hidden dimension matrix block with SiLU activations and classifier network with SiLU activations. The precomputed representation is ablated version of MatrixNet using a fixed representation of S10 instead a learned one. For the fixed representation, we use the standard by the permutation matrices corresponding to the group element. In MatrixNetLLN, the activation between layers of the matrix block is set to a linear passthrough while in MatrixNet-NL. the activation is specified to be SiLU. MM enforces 2x22 block diagonal structure the learned representations corresponding to the 2-dimensional irreps of Sin-
x 10 representation given
We also note the use of SiLU activation in our S1o MatrixNet model. Due to the generator self-inverse property we need not consider separate generator inverses, and so the odd function requirement given in Proposition 2 is not applicable.
7

---

Table 1: MatrixNet and baseline performance on S10 order prediction
Model
Parameters
CE Loss (10-2)
Acc
MLP
365584
0.02 0.01
100 +
Rep Ablation
299792
4.8 + 0.5
87 +
LSTM
270505
8.1 1.4
77.3 + 5.2
MatrixNet-LNN
343968
0.9 + 0.3
99.4 + 0.4
Mat
82960
0.02 + 0.01
100 + 0
MatrixNet-Nonlinear
343968
0.0003 +
100 + 0
Table 2: MatrixNet performance on finite group order prediction
Group
IGI
Rep. Size
Classes
CE Loss (10-3)
Acc (%)
$10 S12
106
10
16
0.0003 + 0
100 + 0
10
12
23
0.8 + 0.2
99.2 + 0.4
Cn x C12 x.. C1s
10a
10
35
2.1 2.4
87.3 + 18
S3 85 85 Ss
10
20
12
2.6 + 0.4
98.3 + 0.3
Model Comparison Results Results of the experiments are summarized in Table 1. All variants of MatrixNet achieve classification accuracy of at least 99% across multiple independent trials. Of note is the inferior performance of the precomputed representation baseline compared to the MLP and MatrixNet on both loss and accuracy metrics, suggesting that there is an advantage to a learnable representation. These results on S10 order classification validate that group representation learning can aid learning of tasks defined over groups
Order Prediction over Different Groups In order to demonstrate the flexibility of MatrixNet, show that MatrixNet can be used to predict order across several different sizes and types of groups. In addition to S10. we evaluate MatrixNet on a larger symmetric group S12 an Abelian group Cn x C12 x C13 C14 x Cis and product S₁ x S₁ x S5 x Sg. These product groups provide more complex group structure which MatrixNet must learn for successful generalization, with varying representation structure. The results for these experiments are summarized Table
MatrixNet achieves a high classification accuracy across all additional groups tested. However, accuracy for the Abelian group is lower than the accuracies for other groups tested (87% vs 99%).. One explanation for this decrease in accuracy is due to the large number of valid orders of the group. Additionally, due to the structure of finite Abelian groups, many element orders will be underrepresented random sampling.
5.2 Categorical Braid Action Prediction
In our second experiment, we train models to predict the Jordan Hölder multiplicities from braid words in the braid group B3 (see Section 3,4). The task is formulated regression task with a mean-squared error (MSE) loss function. The Jordan Holder multiplicities are integers, so we evaluate accuracy by rounding the vector entries to the nearest integer. This accuracy is reported as an average accuracy over the three entries the fordan-Höldes multiplicities vector. Elements B3 generated by two generators 0, d2 and their inverses and encoded using signed one-hot encoding. For more details on the dataset generation process and data splits see Appendix B.2.
We additionally performed experiment to evaluate how well MatrixNet generalizes to unseen braid words longer than those seen in training. For this experiment, we compare against the MLP and LSTM since these were the highest performing basclines.
Baseline Comparison Results We trained all of the models for 100 epochs as all of the models except the Transformer converged within 100 epochs. Despite performance converging much faster than 100 epochs for most MatrixNet variants, we found that additional epochs of training improved the model's invariance to group relations with minimal variations in performance. Table shows the performance of the baseline models and MatrixNet variations 50 and 100 epochs of training

---

averaged over 5 runs. The simple MatrixNet model was the worst performing MatrixNet variant slightyy outperforming the baseline models at 100 epochs. All other variants of MatrixNet vastly outperform baselines with both MatrixNet-I.N and MatrixNet-NI. achieving MSE far below the baselines and perfect or near perfect accuracy across all runs. These results confirm the results from the order prediction experiments and demonstrate the advantage of MatrixNet for learning over groups.
Table 3: MSE and accuracy of variations. Results are averaged over 5 runs. See Appendix B.2 for model parameters and training details.
multiplicities for baseline models and MatrixNet
Model Type
Parameters
MSE Epoch 50
MSE Epoch 100
Avg. Acc.
Transformer
63779
3.013 = 0.147
2.895 + 0.024
42.3% = 1.2%
MLP
52099
0.315 = 0.004
0.132 0.009
89.1% = 0.73%
LSTM
51027
0.345 0.149
0.075 + 0.035
93.0% = 3.9%
MetrixNet
42507
0.543 0.458
0.082 + 0.034
95.1% = 0.9%
MatrixNet-LNN
42883
7.14-412.44-4
0.001 + 0.001
99.9% + 0.004%
MecC
41987
0.063 + 0.033
0.014 + 0.006
98.8% + 0.6%
MatrixNet-NL
42883
0.002 = 0.003
6.4e-413.6c-
99.9% = 0.008%
Length Extrapolation Results The results in Figure 2 show how the MSE and average accuracy change as input length increases averaged over 10 runs. A single run omitted from the results for MatrixNet due to training instability. We observe explosive MSE growth for MatrixNet and MatrixNet-Mer but both maintain higher average accuracy than the baselines. The high variance in MSE suggests that both variants are capable of extrapolating despite struggling compared to the other two variants. MatrixNet-LN and MatrixNet-NL. both maintain near-zero average MSE length increases and consequently achieve near perfect average accuracy as length increases.
The discrepancy in extrapolation performance of the MatrixNet variations suggest that MatrixNet-LN and MatrixNet-NL learn better representations than MatrixNet and Mc. To measure this, compare the relational error of the four MatrixNet variations in Table 4. The group BN has only the braid relation 01020 = 020102. We calculate the relational error as the Fröbenius norm of the difference Mo, non-equivalent braids 010102, 027201 for reference under Non-Relational Difference in Table 4.
Mo, Mo,ll- We also compute this distance between two
The relational error results in Table mirrors the extrapolation performance confirming that represen- tation quality important for effective generalization. High relational error compounds over longer word lengths hindering generalization whereas low relational error allows MatrixNet to automati- cally generalize to longer word lengths through invariance to group relations. These results show that MatrixNet, particularly the MatrixNet-LNN and MatrixNet-NL variants, is able to learn group representations invariant to the group relations allowing for effective generalization to longer unseen group words.
Table 4: Relational error of MatrixNet models trained on length extrapolation dataset. The non- relational difference is computed between two non-equivalent braids for comparison. High relational error compounds for longer words resulting in poor extrapolation.
MatrixNet Variation
Relational Error
Non-relational Difference
MatrixNet Mat
13.63 = 6.83
33.64 + 11.28
2.58 + 2.10
9.60 + 2.5
MatriXNer-LN
0.071 + 0.018
4.96 + 0.55
MatrixNet-NI.
0.066 + 0.009
4.99 + 0.45
5.3 Visualizing the Learned Representations
We present some visualizations of the learned representations of the braid group from the highest performing variant, MatrixNet-NL. Figure 2 shows visual plots of the learned representations. In

---

Extrapolation MSE
Extrapolation Multiplicity Accuracy
Average Multiplicity Accuracy
1.0
10
Average MSE
8
0.6
6-
4
0.4
2
0
0.0
10
7
8
9
10
Word Length
Word Length
MatrixNet
MatrixNet-LN
MatrixNet-MC
MatrixNet-NL
LSTM
MLP
Figure 2: Length extrapolation results. Left: The plot shows how MSE grows for increasing word lengths (y-axis truncated for clarity). Right: The plot shows how the average accuracy decays for increasing word lengths. The relatively high accuracy of MatrixNet and MatriXNet-MC. compared to baselines suggests that the high MSE caused by outliers with multiplicity predictions much higher than the ground truth.
the last two plots, the learned representation for two equivalent words approximately equal even though this relation not among the relations y, used the loss Lret. This shows MatrixNet does indeed generalize over relations, allowing it to generate nearly identical representations for equivalent words.
91 rep
92 rep
9191929192 rер
9291929292 rep
Figure 3: Visualization of learned matrix representations. The first two figures show the representa- tions for the generators of B3. The last two figures show the representation for equivalent words that are generated by the relations of B3.
Conclusion
In this paper we have presented MatrixNet, novel neural network architecture designed for learning tasks with group element inputs. We developed variations our approach which structurally enforce group axioms and regularization approach for enforcing invariance to relations. We evaluate MatrixNet on two group learning problems over several finite groups and B₃ and demonstrate our model's 's performance and ability automatically generalize to unseen group elements. In future work we plan to develop interpretability analysis methods based on group representation theory to better understand the structure of MatrixNet's learned representations. Understanding the learned representations may provide valuable insights and explanations of the model outputs assisting with generating conjectures for open mathematical research problems.
The current work relies on the assumption that the studied group is finitely presented which limits us to discrete groups. However, learned group representations may also be useful for learning over Lie groups. In such case, extending our method will require working with infinitesimal Lie algebra generators. Additionally, while the group axioms are strictly enforced by the model structure, the fact the relations are enforced using auxiliary loss terms means the homomorphism property is not exact. Future work may explore methods of reducing this error.
Limitations
10

---

Acknowledgements This work is supported in part by NSF 2134178. Lucas Laird is supported by the MIT Lincoln Laboratory Lincoln Scholars program. Circe Hsu is supported by a Northeastern University Undergraduate Research and Fellowships PEAK Experiences Award. The authors would like to thank Mustafa Hajij and Paul Hand for helpful discussions.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition at scale. arXiv preprint vcXiv:2010.1192) 2020.
16x16 words:
[3] Thomas Kipf and Max Welling. semi-supervised classification with graph convolutional networks. arXiv preprint 1rOXiv::609.0220
2016.
[4] Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 2013.
[5] Dian Wang. Robin Walters, and Robert Platt. SO(2) equivvaiant reinforcement learning. arXiv preprint arXiv:2203.044233 2022.
[6] Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the rubik's cube with deep reinforcement learning and search. Nature Machine Intelligence, 1(8):366-33, 07 2019, doi:
10.
[7] Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit- pdf: Non-parametric representation of probability distributions on the rotation manifold. arXiv preprint 15555,2106.005655 2021.
[8] Denis Boyda, Gurtej Kanwar, Sébastien Racanière, Danilo Jimenez Rezende, Michael S Albergo, Kyle Cranmer, Daniel C Hackett, and Phiala E Shanahan. Sampling using su (n) gauge equivariant flows. Physical Review D. 103(7):075004, 2021.
[9] Jonathan Huang. Carlos Guestrin, and Leonidas Guibas. Fourier theoretic probabilistic inference over permutations. Journal of machine learning research, 10(5), 2009.
[10] Luca Falorsi, Pim De Haan, Tim R Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S Cohen. Explorations in homcomorphic variational auto-encoding. arXiv preprint //Xiv:/807.000.. 1807,04677, 2018.
[11] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation represen- tations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5745-5753, 2019.
[12] Romain Brégier. Deep regression on manifolds: a 3d rotation case study. In 2021 International Conference 3D Vision (3DV), pages 166-174, TEEE, 2021.
[13] A René Geist, Jonas Frey, Mikel Zobro, Anna Levina, and Georg Martius. Learning with 3d rotations, a hitchhiker's guide
(3). arKiv preprint 2xXx:22444.11355, 2024.
[14] Shiang Fang, Mario Geiger, Joseph G. Checkelsky, and Tess Smidt. Phonon predictions with e(3)- equivariant graph neural networks, 2024.
[15] Adeel Pervez, Phillip Lippe, and Efstratios Gavves. Differentiable mathematical programming for object- centric representation learning. In International Conference Learning Representations, 2023. URL https://openreviie
meet/forummiee id-ii ZTr7aypY.
[16] Adeel Pervez, Francesco Locatello, and Stratis Gavves. Mechanistic neural networks for scientitic machine learning. In Ruslan Salakhutdinov, 7ico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 40484-40501. PMLR, 21-27 Jul 2024. URL https: p/troce///rr mlr. .preepees//eee w233//ervee22222 html..
[17] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.
[18] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-290000 2999.. PMLR, 2016.
[19] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International conference on machine learning, pages 2747-2755. PMLR, 2018
11

---

[20] Nathaniel Thomas, Tess Smidt. Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff. and Patrick Riley. Tensor field networks: Rotation-and Lranslation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv: met::820200222 802.08219, 2018.
[21] Arnab Kumar Mondal, Siba Smarak Panigrahi, Oumar Kaba, Sai Rajeswar Mudumba, and Siamak Ravanbakhsh. Equivariant adaptation of large pretrained models. Advances in Neural Information Processing Systems, 36, 2024.
[22] Albert Musaclian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications, 14(1):579, 2023.
[23] Michael Douglas, Subramanian Lakshminarasinhan, and Yidi Qi. Numerical calabi-yau metrics from holomorphic networks. In Mathematical and Scientific Machine Learning, pages 223 252. PMLR, 2022.
[24] Hyunwoo Ryu, Hong in Lee, Grong-Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: Se(3)- equivariant energy-based models for end-to-end visual robotic manipulation learning, 2023.
[25] Thomas Kipf, Elisc van der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2019.
[26] Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. In The Fortieth International Conference on Machine Learning, 2023.
[27] Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačić. Equivariant self-supervised learning: Encouraging equivariance in represen- tations. In International Conference on Learning Representations, 2022.
[28] Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 33: 2020.
[29] Jung Ycon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for cquivariant world models. In International Conference on Machine Learning, 2022.
[30] Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algehra convolutional network. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages
2503 2503-2515, Curran Associates, Inc., 2021. URL https: / http://ppooeeennn.nneeoec www.ups..c.pppprrr... cccpappr_filess paper paper/20211 http///ilee/1441886666e0ee 11111/444882226770069666 (tt.:/62266007100111663333311133
Paper. .apperrppd. pdf.
[31] Alex Gabel, Victoria Klein, Riccardo Valperga, Jeroen S. W. Lamb, Kevin Webster, Rick Quax, and Efstratios Gavves. Learning lic group symmetry transformations with neural networks. In Timothy Doster, Tegan Emerson, Henry Kvinge, Nina Miolane, Mathilde Papillon, Bastian Ricck, and Sophia Sanborn, editors, Proceedings 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML), volume 221 of Proceedings of Machine Learning Research, pages 50-59. PMLR, 28 Jul 2023. URL https: / /tpp//eeeeenng...//////////...ee mlr. press/v221 0//221//abbl/// gagelaaaaaaaaaa htmm.
[32] Jianke Yang. Nima Dehmamy, Robin Walters, and Rose Yu. Latent space symmetry discovery. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller. Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 56047-56070. PMLR, 21-27 Jul 2024. URL https: https://proceeding. mlr. wwl..pres//.. wttppp/rrsss/////ppeepppp http///23/ayaa22243 yangg4g......... html.
[33] Mustafa Hajij. Ghada Zamzmi, Matthew Dawson, and Greg Muller. Algehraically-infommed deep networks (aidn): A deep learning approach to represent algebraic structures, 2021.
[34] Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models, 2023.
[35] Trieu H. Trinh, Yuhuai Wu. Quoe V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, (5577955)447--882 Jan 2024. doi: 10.10-57441556-00220077755...
[36] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i):
Data-driven solutions of nonlinear partial differential equations, 2017.
[37] Joel Janek Dabrowski, YiFan Zhang, and Ashfaqur Rahman. ForecastNet: A Time-Variant Deep Feed- Forward Neural Nerwork Architecture for Multi- Multi-step-Ahee step-Ahead Time-Series Forecasting, page 579-591. doi:
Springer International Publishing. 2020. ISBN 9783030638368.
48.
URL htdp://dx. doi.iixidiooodiooooooooooo...... org/10 10/ 1007 1000/977-333 3-7788--0300 00188-3--00---8------88 -_44
[38] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data driven high-resolution weather model using adaptive fourier neural operators, 2022.
12

---

[39] Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev. Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, and al. Advancing mathematics by guiding human intuition with ai. Nature, 600(7887):70-74, Dec 2021.
doi: 10. 10.10088/4415886-22--00666--6--
[40]
Michael Artin. Algebra. Prentice Hall, 2011.
[41] Heinrich Maschke. Ueber den arithmetischen charakter der coefficienten der substitutionen endlicher linearer substiutiionsguppen.. Mathematische Annalen, 50(4):492-498, 1898.
[42] Camilo Arias Abad. Introduction to representations of braid groups arXiv preprint arXiv:1404.0724,
2014.
[43] William Fulton and Joe Harris. Represeniction theory: first course, volume 129. Springer Science Business Media, 2013.
[44] Asilata Bapat, Anand Deopurkar, and Anthony M. Licata, A Thurston compactification of the space of stability conditions. 2020. http://arxiv arg/abs/2011 1, 079988.
[45] Paul Seidel and Richard Thomas. Braid group actions on derived categories of coherent sheaves. Duke Math. J. 108(1):37-100) 2001. ISSN 0092-7094, doi: 21155/00012-7944400 01044-0010111-----00
[46] Raphaël Rouquier and Alexander Zimmermann. Picard groups for derived module categories. Proc.
London Math. Soc. (3), 87(1):177-225, 2003. ISSN 0024-61 15. doi: 10.1 12/S002441 1503044059.
[47] Asilata Bapat, Louis Beeker, and Anthony M. Licata. q-deformed rational numbers and the 2-calabi-yau category of type. Forum of Mathematics, Sigma, 11, 2023. ISSN 2050-5094, doi: 2..0177/ins.2223332 URL http:/ / /xx doi. org/10 10. 1017/fms 2023. 32.
[48]
Mikhail Khovanov and Paul Scidel. Quivers, Floer cohomology, and braid group actions. J. Amer. Math. Soc., 15(1):203-271 271, 2002. ISSN 0894-0347. doi: 10. 1090//9// 0347-01 1847-01-00333 01-00374-5.
[49] James E. Humphreys. Reflection groups and Coxeter groups. Cambridge University Press, 2000.
[50] Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondłej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMIT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig. Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:3, January 2017. ISSN 2376-5992. doi: (0.77177peerj-cs.100 URL https: //dei. wttpr////...71//ererree- 103.
A More details about the categorical braid group action
The aim of this appendix is to provide few more details about the particular categorical braid group action that we use our experiments.
A.1 Sketch of the construction of the category
The category Cn we consider is the 2-Calabi-Yau triangulated category associated the Dynkin graph of type An. This is an undirected graph with vertices and edges arranged in a line, as shown in Figure 4.
Figure 4: The Dynkin graph of type An
Let P, be the Dynkin graph of type A, Let n. be its doubled quiver, which is a directed graph in which each undirected edge I replaced by pair of oppositely oriented directed edges, as shown in Figure 5.
Figure 5: The doubled quiver pdbl of the Dynkin graph of type A,,
13

---

Recall that the path algebra of directed graph (or quiver) Q over some field lo is generated as free vector space by all possible paths in Q, including the trivial paths each vertex. The product in kQ is given as follows: let q: - be path and p: c -> d be a path. The product py is equal to zero unless
If c, then the product pq is simply the composite path a 4 b 2 d. The path algebra of Q is denoted kQ.
We are interested in quotient of the path algebra of r called the zig-zag algebra, and denoted Zn To obtain Zn we impose the following relations on kldbl In what follows, (ili + 1) represents the unique arrow i + i = 1.
For each i, set
(1+ + 2 +1) = 0 and (i 1 2/i - 1) = 0.
For each i, set
(1 (i ( (1)
Consequently, the zig-zag algebra, any path of length least 3 is automatically zero, and the only surviving paths of length 2 are back-and-forth loops starting from any vertex (and all possible such loops are set be equal). The paths of length 0 and remain as-is.
Let Zn - proj be the category of (graded) projective modules over Zy, The category C, is constructed differential graded version of the bounded homotopy category of complexes of projective modules over 7, in which we identify the internal grading shift with the homological grading shift, and consequently also the triangulated shift. For more details, see, e.g. [44, Section 6].
A.2 Important properties of the category
In this section, we record some important properties of the category Ca- First, the category Cn is triangulated: there is triangulated shift functor [1] []:[C]]] C,, -> C, which is equivalence. An n-fold composition of [1] is denoted [r]].
Denote by HommA, B) the set of morphisms in Cn from an object A an object B. Sometimes
write Hom [A, B) Hom" (A, B), and further write Hom" (A, B) to mean Hom(A, B(nl) for any integer n.
The category C, is generated as a triangulated category by the objects PI...., P.. That is, the smallest triangulated subcategory C, (closed under isomorphisms) that contains the objects PI,... Pn C, itself. These objects correspond to the indecomposable projective modules of the zig-zag algebra Zn
Each object P. is spherical. This means that
{k 0 otherwise.
k or = 2,
Hom" Hom" (P,,
Remark A.1. The reason for the notation is that the ring of endomorphisms of P; of all possible degrees is isomorphic to the cohomology ring of sphere (in this case a 2-sphere).
is a general fact that any spherical object X a triangulated category gives an associated functor xx, called the spherical twist in X (see Seidel-Thomas [45] for more details.) The functor dxx is an auto-equivalence; that is, it has an inverse equivalence o such that compositions in both directions are isomorphic to the identity functor.
In particular, we obtain equivalences or; Cn - Ca.
A.3 The Horden-Hölder filtration
The category C, has bounded f-structure. This means that there is an abelian subcategory An Cn. such that every object X Cn has unique finite filtration whose factors lie Ali] for decreasing values of i. This filtration is called the cohomology filtration. In fact, this abelian subcategory An is also generated by the objects P;: is the extension-closure of the objects P;. Moreover, the objects P. are simple objects of An.
14

---

First consider any object X E An. The category An is finite-length abelian category. It is standard fact that X has a Holder filtration whose factors are simple objects in A,, namely the objects P:-
Now consider general object X Cn. We first consider the cohomology filtration of X, with factors Y E A. For each Y, E A[j], we consider its (shifted) fordan-Hölder filtration, which breaks Y, up further into copies of P. Putting these two together, we obtain finer filtration of the object X, which we also call the Jordan-Hölder filtration of X.
The factors of this Jordan-Hölder filtration shifted copies of P. for all i. Thus number of occurrences of each Pi in the Fordan-Hölded filtration of X, and it is well-know that these counts do not depend the specific choice of the
count the
filtration.
A.4 The action of the braid group
Recall the spherical twist functors o,, : Cn Cn. A remarkable observation of Seidel-Thomas [45] is that these functors (weakly) obey the relations of the n-strand braid group. That is, we have isomorphisms of functors
OP, 20
whenever 1- * 1, and
OP.OPOPP ~ Or
whenever
1-ji 1.
Since these are precisely the relations of the group B,, we obtain an action of B, on the objects of the category Cu.
A.5
Open problems and future directions
Consider the following broad question. Given an object X of C, and a braid g E Bn can we relate the Hölder multiplicities of ((X) to the Jordan-Hölder multiplicities of X? Stated more simply, can we compute the Jordan- Hölder multiplicities of (3P:) for any and any /8?
Answers to this question are known in some cases, and remain open in others. For instance, complete answer was obtained by Manquee-ZZimmrrmmn [46] for the 3-strand braid group B3 acting C3. This answer was rediscovered and refined in terms of more general filtrations (Harder-Narasimhan filtrations) in [44] and [47].
is also known that if 3 = ofp, for some É, then the limit as - oc of the counts of ((X) can be obtained, up to a common scaling factor, by computing the sum of the dimensions of Hom" (Pj, X) for all [47].
However, for the vast majority of values of n and most of the elements of the braid groups B., we do not have good answer this question. While there is vast potential for future work, write down a few specific open problems.
1. Generalise the Rouquier Zimmermann theorem (and its corresponding versions in [44] and [47]) to larger values of n.
2. We can compute finer version of Jordan-Hölder multiplicities: split up the number of occurrences of each P; by degree shift. That is, record the number of occurrences of P. [d] separately for every possible d. This information variable in g+1, in which the coefficient of gd the multiplicity of P.(d]
be encoded in polynomial in one
Generalise the Rouquier- Zimmerrmnnnmmmmmm theorem in this setting to larger values of n.
3. By using more refined version of Jordan-Hölde multiplicities, known Harder- Narasimhan multiplicities, we observe that the possible Harder-Narasimhan factors of any object of the form ((P)) are highly constrained, and satisfy some very nice combinato- rial properties.
This constraint can be explicitly described for any B, via a geometric model (due to Khovanov-Seidel [48]) for objects in the category C,, Nevertheless, the relationship of these constrained sets with the action of B, is mysterious. For example, given an object X, is there an algorithm to write down a braid that will send X to an object with a desired set of -arassmha filtration factors?
15

---

4. Can we use the combinatorial structure mentioned above to algorithmically write down combinatorial actions of braid groups on simpler sets? What properties do these actions satisfy?
5. All of the categorical constructions described in this paper also go through for more general versions of braid groups, known as Artin-Tits braid groups. All of the questions above B.1 Symmetric Group Dataset
remain open for all but the simplest cases of Artin-Tits groups.
B Dataset and Model Parameter Details
We generated a dataset of 500, 000 samples consisting of words of the free group Fio, and labels corresponding to their order as elements of S1o. The first step of dataset generation was to fix maximum word length chosen such that it is possible to sample every element of S10. For generating set corresponding to adjacent transpositions of elements in Sn, this longest word will be of length -(n-") [49], and for S10 choose our maximum length to be 64. We define a uniform distribution on the set of generators do, oi...., On- where do and all other
(i i i 1). Informally, our generating set consists of adjacent transpositions of the form (i i + 1) along with an identity generator. The presence of the identity generator adds variability to word length while enforcing identity invariance. Sample order labels Si0 are computed using the SymPy package [50].
While we do not strictly enforce separation of elements between training, test, and validation sets, statistically unlikely have any significant overlap between the splits. Recall that [S1o] = 10! 3, 628, 800. Our dataset of 500, 000 samples therefore covers at most 13.7% of S10. implying the likelihood of significant overlap between partitions upon reduction is very low. Moreover, because we are sampling unreduced words from Fio of length 64, there are 10 possible words we could sample from, making the probability of direct overlap between partitions effectively zero
B.2 Categorical Braid Action Experiment Details
Dataset Generation An initial dataset of Holderr multiplicities for braid words up length was provided. We implemented a state automaton algorithm from [47] to generate additional examples for longer braid words. This method was compared against the Jordan -öllde multiplicities of the initial dataset to verify correctness.
Baseline Comparison Experiment Details The baseline comparison dataset consists of 47,831 examples with braid words up to length 8. The data was split into 60% training data, 20% validation data, and 20% test which were fixed for all models. All of the models trained using an Adam optimizer with learning rate of le-4 and a batch size of 128. The chosen parameters for the models are:
MatrixNet: Single channel 14 x 14 matrix size
MatrixNet-LN: Single channel 10 x 10 matrix size, 128 dimensions for linear network in the matrix block
MatrixNet-MC: 3-channel x 8 matrix size
MatrixNet-NL: Single channel 10 x 10 matrix size, 128 hidden dimensions and a tanh non-linearity between linear layers of matrix block
MLP: 3-layer MLP with 128 hidden dimensions for each layer and ReLU activation functions followed by a single linear layer output.
LSTM: 6 LSTM layers with 16 dimensional input embeddings and 32 hidden dimensions followed by a 2-layer MLP classifier with 64 hidden dimensions and ReL.U activation.
Transformer: transformer layers with 4 attention heads, 16 dimensional embeddings and 32 hidden dimensions. Used mean pooling and a single linear layer output.
All of the MatrixNet architectures used a 2-1ayer MI.P with 128 hidden dimensions and ReLU activation to compute output after the matrix block.
16

---

Length Extrapolation Experiment For the generalization experiment we generated dataset of all braid words up length which was split into 80% training and 20% validation. We also generated three separate test datasets of 10,000 examples each with braid words of length 8, 9, and 10 evaluate how performance degrades over increasing length. The models were trained for 100 epochs on the training and validation sets and then tested on the three test sets.
Regularization Details All MatrixNet architectures were trained using the regularization loss defined in Section 4.3. We chose the Frobenius norm for the norm and used the braid relation 010202 02 and the inverse 01 02 01 = 22 01 02" 02 The regularization term was added to the loss every 10 training batches.
Hardware Details All of the categorical braid action experiments were run on a machine with single Nvidia RTX 2080 ti GPU.
B.3 Length Interpolation Results
We also performed length interpolation generalization experiment test how well each model generalizes to braid words that are shorter than the maximum length seen during training. The results are not presented in the main body of the paper as all models generalize to shorter braid words.
Table 5: Length 5 interpolation performance of baseline models and MatrixNet variations. Test MSE and accuracy is measured over test which contains braid words of same length training.
Model Type
Test MSE
Test count acc.
Interpolation MSE
Interpolation count acc.
MLP
0.3628
72.2%
0.174
80.4%
LSTM
0.7995
53.2%
0.5442
57.7%
MatrixNet
1.097
59.7%
0.413
67.3%
MatrixNet-LNN
0.0120
99.9%
0.0077
100%
MatrixNet-MC
0.2544
91.1%
0.0691
100%
MatrixNet-NL
6.8e-3
100%
3.8e-3
100%
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide bulleted claims and contributions in the introduction. We provide experimental results and proofs in the methods and experiments sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The Limitations section details the assumptions of the model and the limitations of regularization as method for enforcing group relation invariance.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
The theoretical results are stated and proven in the Methods section as proposi- tions and proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our architecture and regularization method are described in the Methods section. We also provide summary of relevant theoretical background in background section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We plan to open source code upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/ heriiic/guie////eee Weaeee//oooeuumiiie wtt.:/awwwmssinooooooooooooooooe
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // /nips.cc// wwwpic/gguddesssepppppppppaapaee httpe/gguiddee/eeeuuubieeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the model parameters, dataset generation and splits, and optimizer in the appendix. We also give context for the experiments in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We have not performed these exact experiments enough times to produce error
bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The hardware used is reported in the appendix. Our model and data do require significant compute and are small enough to be run on most machines.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ : http:///eeuuuueeeeeeeeeeeee http://uuurrppccppuprreee cc/public/ cccppullic/thiccseeeee wtt/tetttiittteeeeeeeeeeeeeee ceGuidelinees
Answer: [Yes]
Justification: This paper focuses deep learning in mathematical context. There were human participants or personal data involved in this research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses entirely on mathematical problems with no broader socictal impacts.
Guidelines:
21

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not use any pretrained models and generate our own data. The data not sensitive private in nature.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all packages used and acknowledge compute resources used during this research. We do not use any preexisting assets in this research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcae thcode. com/coo/////ooooo conddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We are not releasing new assets as part of this submission. We will provide documentation and code upon publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not crowdsource data experiments in this research. There are human participants involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There are no human subjects involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "ZBCFWcaszT", "reviewer": "Reviewer_jQuD", "summary": "The authors use neural networks to learn group representations. A group element is represented by its generators formatted as a sequence of learned matrix representations. These generators are mapped to a single matrix representation of the group element via the Matrix Block which enforces group axioms. The resulting feature is used for downstream tasks on groups. The authors show that the proposed architecture MatrixNet successfully predicts group element orders for S_10 and Jordan-Holder multiplicities for braid group B_3. Further analysis on word length extrapolation and visualization demonstrates the superiority and usefulness of the approach.", "questions": "In the matrix block, is it possible to introduce commutative operations between different matrix representations of group generators if the input group element comes from an abelian group. It seems possible to introduce further architectural constraints for specific groups.", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In Table 1, the faster convergence of MatrixNet potentially indicates that the proper architectural constraints provide good inductive bias for learning group representations and solving downstream tasks over groups. It’s good to see how one can explicitly build in these constraints in the problem of group representations and that it works better than a less constraining MLP without domain-specific inductive bias.\n2. The paper is well-written.", "weaknesses": "1. It’s unclear why naive MatrixNet and MatrixNet-MC cannot extrapolate to longer word length. \n2. Empirical evaluations are a bit limited. It might be helpful to evaluate groups with different properties (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer jQuD\n>It’s unclear why naive MatrixNet and MatrixNet-MC cannot extrapolate to longer word length. \n\nWhile MatrixNet and MatrixNet-MC underperform compared to our other two variants, it is overstated to say they cannot extrapolate to longer word lengths. Despite their high MSE, both approaches maintain relatively high accuracy compared to our baselines. That said, MatrixNet-LN and MatrixNet-tanh have better extrapolation. The reason for this performance discrepancy is that MatrixNet and MatrixNet-MC have higher relational error indicating they do not learn group representations as accurately. This error compounds for longer words. To help make this difference clearer we computed the relational errors of the different models on the Artin Braid group, shown in Table 1, which we will add to the paper. \n\n| Model | Rel. Error |\n| ------------ | -------- |\n| MatrixNet | 14.78 |\n| MatrixNet-MC | 5.21 |\n| MatrixNet-LN | 0.33 |\n| MatrixNet-tanh | 0.45 |\n\n> Empirical evaluations are a bit limited. It might be helpful to evaluate groups with different properties.\n\nThank you for the feedback. The task we used for the braid group was a primary motivation for our approach. We are limited to $B_3$ since multiplicity counts for larger braid groups are not known. We chose the symmetric group for our initial experiment in section 5.1 as it is a well-studied group that has the unique property that all finite groups are isomorphic to a subgroup of a symmetric group. It also is closely connected to the braid group making it well suited for ablation tests. We believe that these two groups provide a strong foundation for evaluating MatrixNet but we will perform further evaluations on an abelian group as well. \n\n__Questions__:\n\n__Q1__: In the matrix block, is it possible to introduce commutative operations between different matrix representations of group generators if the input group element comes from an abelian group? \n\n__A1__: Yes this is possible. For an abelian group commutativity can be enforced in two ways in MatrixNet. One is with a loss term $L = |M_1 M_2 - M_2 M_1|$ and the other is by choosing the learned matrix representation to be diagonal, i.e. a direct sum of one dimensional representations. Since irreps for an abelian group are one dimensional, this would not harm expressivity of the learned representation and would enforce exact commutativity. We did not use diagonal matrices for the non-abelian groups specifically for this reason. More concretely, MatrixNet-MC with scalar channels is an architecture that is constrained to learn commutative representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sMXQfku0xj", "reviewer": "Reviewer_JC7A", "summary": "This paper studies feature representations of a group element for supervised learning. It considers a regression task where an input is a group element g of a finite group and a target is some label. Firstly, g is decomposed into a sequence of generators (g_1, ..., g_n) such that g = g_1 \\circ ... \\circ g_n. Next, each generator is mapped by a trainable matrix embedding W: Gen(G) --> R^{n \\times n} followed by the matrix exponential so that we can get a group representation M_i = exp(W(g_i)) ∈ GL(n). Then their product M_1 \\circ ... \\circ M_n becomes the feature. Similar variants are also proposed. The performance is evaluated on two synthetic tasks: the order prediction of the symmetric group S_10 and the action prediction of the Braid group B_3.", "questions": "1. Given g, is its decomposition into the generators always uniquely determined?\n1. In the experiment of 5.1, what will happen when we employ the group decomposition while using the fixed representation? I mean, the feature of g is given as concat[\\rho(g_1), ..., \\rho(g_n)] where g = g_1 \\circ ... \\circ g_n and \\rho(g_i) is some representation of g_i (e.g. irreducible rep).\n\nTypo: \n* One of M_{g_ik^-1} would be {M_{g_ik}}^{-1} in the equation below line 203.", "limitations": "Limitations are adressed.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written. Technical details are clear enough. \n1. The idea to construct a feature representation of a group element as a learnable parameter is interesting and it is a possibly promising direction.", "weaknesses": "1. Limited applicability to real tasks. To the best of my knowledge, regression (or classification) of group elements is practically important for continuous groups such as SO(3) (e.g., pose estimation). However, the current approach is only applicable to finite groups. Also, I think the tasks conducted in the experiments are not directly bridged to real problems, and I feel uncertain about how the proposed method is practically valuable. \n1. It is unclear which component significantly contributed to the final performance gain. The proposed method consists of (at least) two parts: the decomposition of g into generators and a trainable representation of a generator. The current experiments are not designed to evaluate them separately. \n1. The experiments are not convincing enough—they are relatively small scale, use synthetic tasks only, and have less variety (two tasks).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer JC7A\n> Limited applicability to real tasks. The current approach is only applicable to finite groups.\n\nOur method is **not** limited to finite groups. One of our experiments focuses on the infinite Artin braid group. You are correct that our approach is not formulated for continuous groups as it is limited to **discrete** groups.\n\n> Also, I think the tasks conducted in the experiments are not directly bridged to real problems, and I feel uncertain about how the proposed method is practically valuable. \n\nThe second task used in our experiments over the Artin-Braid group is a real current research problem in pure math. Mathematicians have only found a way to compute the answer for the simplest braid group $B_3$ – and they do not have a simple or intuitive formula. Our goal is to create a model that can help mathematicians build intuition and formulate conjectures by 1) computing additional data points and 2) providing new insights through inspecting the learned representations. We believe this application is relevant to real mathematical research.\n\n> It is unclear which component significantly contributed to the final performance gain: decomposition of g into generators or a trainable representation of a generator.\n\nWe do test the impact of a tranable representation independent of the decomposition in our first experiment in section 5.1 of the paper by comparing against the precomputed representation. This ablation replaces the learnable representations with the permutation representation of $S_{10}$ and we see significantly worse performance when compared to learnable representation.\n\nWe also show that just decomposing g into generators but not using a learned or fixed group representation does not result in improved performance. We compare against two sequential baselines, a transformer and LSTM, which both take the decomposition of g as input. These models however do not learn a group representation showing that just the decomposition of g into a generator sequence does not explain the improved performance of MatrixNet.\n\n> The experiments are not convincing enough—they are relatively small scale, use synthetic tasks only, and have less variety (two tasks).\n\nWe disagree that only synthetic tasks are used. The task used in our Artin braid group experiment is an unsolved math problem from a recent pure mathematics publication [39]. Mathematicians have only found a way to compute the answer for the simplest braid group B_3 – and they do not have a simple or intuitive formula. Even so, we believe the two groups used, $S_{10}$ and $B_3$, are sufficiently large with $|S_{10}| = 10!$ and $B_3$ being an infinite group. \n\n__Questions__:\n__Q1__: Given g, is its decomposition into the generators always uniquely determined?\n\n__A1__: No, the decomposition of g is not unique. This is a great question and is a huge part of the motivation behind our approach. MatrixNet is designed to be invariant to the choice of decomposition.\n\n__Q2__: In the experiment of 5.1, what will happen when we employ the group decomposition while using the fixed representation? I mean, the feature of g is given as concat[$\\rho(g_1)$, ..., $\\rho(g_n)$] where $g = g_1 \\circ ... \\circ g_n and \\rho(g_i)$ is some representation of $g_i$ (e.g. irreducible rep).\n\n__A2__: You could do this, but concatenating in this way means the feature size grows with the length of the decomposition. Since the decomposition of is not unique, another drawback is that this method would result in different features for two different but equivalent decompositions of g. The length of a decomposition is also unbounded. For example, in the braid group an element can be decomposed into arbitrarily many generators which would result in unbounded feature sizes. Our method by contrast is more efficient since feature sizes are fixed and with low relational error produces unique representations for a group element. \n\n__Typo__: One of $M_{g_ik^-1}$ would be ${M_{g_ik}}^{-1}$ in the equation below line 203.\n\nGood catch. We will fix it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GqMmj627GG", "reviewer": "Reviewer_KZHf", "summary": "The paper describes MatrixNet, a method to learn group representations such that they are optimized for a certain task of interest using a neural network. , The neural network takes in a group element in the form of a sequence of generators that compose to form the group element and forms an intermediate output that contain the matrices that are the group representations for each of the generators. The generators can them be multiplied to form the group representation which is then fed to further neural layers that are task-specific. The network is trained so as to (approximately) satisfy the constraints of group representations using architectural constraints and the loss function. Experiments on predicting the order of a element of the symmetric group and Jordan-Holder multiplicities for the 3-strand braid group.", "questions": "No additional questions.", "limitations": "Yes, authors have listed the limitations of the current submission and suggested directions for future research.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper present new ideas for learning on groups. Design of architectures is clever and done well with good mathematical justification. The architectures provably output group representations.\n\n2. Experiments show that the proposed architecture leads to learning group representations that lead to better results compared to just using \n\n3. The paper also does a good study of the possible variations of MatrixNet and presents experimental results for them.", "weaknesses": "1. One of the weaknesses is that the parts about the braid group are very difficult to follow and probably need a lot more background for readers and attendees for this conference. This includes myself and I was not able to fully follow the details or why that experiment is important.\n\n2. The experiments are also a little weak in my opinion. It was not clear to me how the learned representations were different from precomputed ones and why they were better for a given task. Also, what happens when the size of the group representations is larger. Perhaps for the first experiment, it would be nice to see the results as a function of the size of the group representation. I am not sure if the proposed architecture and learning mechanism can learn good representations when the size is large. This is important to address, in my opinion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer KZHf\n> Parts about the braid group are very difficult to follow and probably need a lot more background for readers and attendees for this conference.\n\nThank you for this feedback. We will make sure to revise this section to add more clarity. Intuitively the braid group defined in the paper represents all of the possible ways to braid a set of $N$ ropes. The braid group is closely connected to many fields of math but in particular we are interested in how it acts on mathematical categories through “categorical braid actions”. Mathematicians are interested in studying “filtrations” of the objects of categories but there isn’t a unique filtration for all objects within a category. The categorical braid actions however act on all such filtrations in the same way and so the multiplicity counts provide a canonical way to describe object filtrations regardless of the specific filtration. We include a more in-depth discussion of the braid group and categorical braid actions in section A of the appendix.\n\n> The experiments are a little weak in my opinion. It was not clear to me how the learned representations were different from precomputed ones and why they were better for a given task. \n\nOur results parallel other results in deep learning showing that learned feature representations often outperform expert-engineered features. The precomputed representations we used are a natural choice for matrix representations for the groups used. For example, the group $S_{10}$ represents all of the ways to permute 10 objects and can intuitively be represented by 10x10 permutation matrices, which is the precomputed representation we use in the experiment. However, depending on the task, this may not be the most useful representation. For example, the sign representation of a permutation is simply $\\rho(\\sigma) = \\pm1$ depending on the parity of $\\sigma$. This would be a useful feature for determining a group element's order since every permutation of odd order has even parity. Our approach is designed to automatically learn a representation that is useful for the given task. \n\n> What happens when the size of the group representations is larger. I am not sure if the proposed architecture and learning mechanism can learn good representations when the size is large.\n\nOur method can scale to large representations using MatrixNet-MC. MatrixNet-MC assumes large representations have a block diagonal structure, which is efficient since it means the number of trainable parameters grows asymptotically linearly instead of quadratically. For well chosen block sizes, this does not harm expressivity since for many groups, all representations will be block diagonal with respect to a good choice of basis with the maximum block size equal to the largest dimensional irreducible representation. Even for very large groups these irreducible representations are low dimensional which can be learned by MatrixNet-MC. \n\nWe omitted results with larger representation sizes as the performance did not change with matrix size. We include additional results for MatrixNet on the braid group task with doubled representation sizes in Table 2. \n\n| Model | MSE | Acc. |\n| ----------- | ---- | ---- |\n| MatrixNet | 0.975 | 85% |\n| MatrixNet-MC | 0.052 | 96% |\n| MatrixNet-LN | 4.5e-4 | 100% |\n| MatrixNet-tanh | 1.1e-3 | 100% |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jRFHdCK1KC", "reviewer": "Reviewer_puxG", "summary": "In this paper, authors study the question of what feature representations to use for learning tasks with inputs coming from a symmetry group. They propose MatrixNet, a neural network architecture that learns matrix representations of group element inputs instead of using predefined representations.\n\nThe main contributions are as follows:\n1. Formulation of the problem of learning over groups as a sequence learning problem in terms of group generators with invariance to group axioms and relations.\n2. The proposed neural network architecture, MatrixNet, achieves higher sample efficiency and generalization over several more general baselines in prediction tasks over the symmetric group and Artin braid group.\n3. The matrix block method for constraining the network to respect the group axioms and an additional loss term for learning group relations.", "questions": "1. The motivation is to solve a mathematical problem or real-world learning task? I hope it can help me understand the paper clearly and inprove the writing.\n\n2. In subsection 4.1, authors formulate the problem. In line 166, can $f$ be seen as a representation of $G$ or not? If it is a representation, can you write a explicite expression as an example? Since if it is a representation, the target domain $R^{c}$ may not be a linear space which makes it unclear for me.\n\n3. Can you show more experiment results? Such as larger symmetric group? Also I didn't understand the meaning of predict the order of the group element? I think we should determine it instead of estimating it? Maybe I did't get your point, but I am open to discuss.", "limitations": "see the weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strength of the paper is building up links among group reprsentation theory, learning tasks and neural networks, more specifically:\n1. Formulation of the problem of learning over groups as a sequence learning problem in terms of group generators with invariance to group axioms and relations.\n2. The matrix block method for constraining the network to respect the group axioms and an additional loss term for learning group relations.", "weaknesses": "The motivation is not so clear to me. The applications of group theory are everywhere in the real-world. Thus, how to choose a good representation for the group which is associated with the learning task is more important. In this paper, authors formulate this problem but do not give a clear answer (see my questions below), and authors focus on solving a more abstract group theory question instead which is already studied a lot by mathematician. \n\nI didn't see any insight behind the paper. For the experiment, I think it only solved a mathematical problem at simple setting which is far from what we expect. For example, how the model works when the group's order become larger?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer puxG\n> The motivation is not so clear to me. The applications of group theory are everywhere in the real-world. Thus, how to choose a good representation for the group which is associated with the learning task is more important. In this paper, authors formulate this problem but do not give a clear answer. \n\nIn the $S_{10}$ experiment in section 5.1 we compare against using precomputed representations and show significantly worse performance compared to the learnable representations of MatrixNet. This ablation shows MatrixNet can automatically learn more useful group representations for a given task. Our results parallel other results in deep learning showing that learned feature representations often outperform expert-engineered features. We discuss our motivation below with your question.\n\n> I didn't see any insight behind the paper. For the experiment, I think it only solved a mathematical problem at simple setting. \n\nThe task used for our experiment on the Artin braid group in section 5.2 is an open mathematical research problem. This task is limited to $B_3$ since the multiplicity counts are not known for any larger braid groups. The experiment over $S_{10}$ in section 5.1 was chosen to provide a well-studied group with known representations to compare precomputed representations against the learned representations of MatrixNet.\n\n__Questions__:\n\n__Q1__: The motivation is not so clear to me. The motivation is to solve a mathematical problem or real-world learning task?\n\n__A1__: Thank you for the feedback. We will make sure to make the motivation and application more clear in the paper. The second task used in our experiments over the Artin braid group is a current open mathematics research problem. Our goal is to create a model that can help mathematicians build intuition and formulate conjectures by 1) computing additional data points and 2) providing new insights through inspecting the learned representations. We believe this application is relevant to real mathematical research.\n\n\n\n__Q2__: In line 166, can $\\mathcal{f}$ be seen as a representation of \\mathcal{G} or not? \n\nIn line 166 the function $\\mathcal{f}$ does not constitute a representation of the group since the output space does not obey the group composition operation. Let $v_i = \\mathcal{f}(g_i)$ be the vector target for a group element. Then if $g_k = g_i \\circ g_j$ it is not the case that $v_k = v_i + v_j$.\n\n\n__Q3.1__: Can you show more experiment results? Such as larger symmetric group?\n\n__A3.1__: Yes, thank you for the suggestion. Using the same data generation scheme and data splits detailed for the $S_{10}$ experiment, we generated 800,000 samples for the group $S_{12}$. $S_{12}$ is roughly two orders of magnitude larger than $S_{10}$ (12! vs 10! elements). MatrixNet achieves an MSE of 3e-4 across all dataset splits with a classification accuracy of over 99%. The only increase to the network size necessary in this case was altering the rep size from 10 to 12. \n\n__Q3.2__: Also I didn't understand the meaning of predict the order of the group element? I think we should determine it instead of estimating it?\n\n__A3.2__: Sorry for confusing terminology.  We will change “predict” to “determine.”  Our model outputs an exact order, not an estimate.  We used the term predict in a colloquial way to refer to the model output.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# NeurIPS Rebuttal

We thank the reviewers for their feedback and insightful comments. We are glad they found our work well-written(__jQuD__, __JC7A__). It is particularly encouraging that many reviewers found our design of architectural constraints to learn group representations novel(__KZHf__), interesting(__JC7A__), and theoretically justified(__jQuD__, __KZHf__). We respond to specific comments below.

__Unclear real-world motivation (JC7A, puxG) 
> (__JC7A__) I think the tasks conducted in the experiments are not directly bridged to real problems.
> (__puxG__) The motivation is to solve a mathematical problem or real-world learning task? 

The primary motivation of our approach is to assist with mathematical research. The second task used in our experiments over the Artin braid group is a current open research problem. Mathematicians have only found a way to compute the answer for the simplest braid group $B_3$ – and they do not have a simple or intuitive formula. Our goal is to create a model that can help mathematicians build intuition and formulate conjectures by 1) computing additional data points and 2) providing new insights through inspecting the learned representations. We believe this application is relevant to real mathematical research.


## Response to Reviewer jQuD
>It’s unclear why MatrixNet and MatrixNet-MC cannot extrapolate to longer word length. 

While MatrixNet and MatrixNet-MC underperform compared to our other variants, it is overstated to say they cannot extrapolate. Despite their high MSE, both approaches maintain relatively high accuracy compared to our baselines. That said, MatrixNet-LN and MatrixNet-tanh have better extrapolation. The reason for this discrepancy is that MatrixNet and MatrixNet-MC have higher relational error indicating they do not learn group representations as accurately. To help make this difference clearer we computed the relational errors of the models on the braid group which we will add to the paper.

| Model | Rel. Error |
| - | - |
| MatrixNet | 14.78 |
| MatrixNet-MC | 5.21 |
| MatrixNet-LN | 0.33 |
| MatrixNet-tanh | 0.45 |

> Should evaluate on groups with different properties.
We chose the symmetric group for our initial experiment in section 5.1 as it is a well-studied group that has the property that all finite groups are isomorphic to a subgroup of a symmetric group. It is also closely connected to the braid group which serves as our motivating problem. We believe that these two groups provide a strong foundation for evaluating MatrixNet but we will perform further evaluations as you suggested.

## Response to Reviewer JC7A
> It is unclear which significantly contributed to the final performance gain: decomposition of g into generators or a trainable representation of a generator.

We test the impact of a trainable representation independent of the decomposition in our experiment in section 5.1 by comparing against the precomputed representation. This ablation replaces the learnable representation with the permutation representation of $S_{10}$ and we see significantly worse performance when compared to learnable representation. We show that just decomposing g into generators but not using a group representation does not result in improved performance by comparing against two sequential baselines, a transformer and LSTM, which take the decomposition as input but do not use a group representation.

> The experiments are not convincing enough—they are relatively small scale, use synthetic tasks only, and have less variety (two tasks).

We disagree that only synthetic tasks are used. The task used in the braid group experiment in section 5.2 is an open math problem from a recent mathematics publication [39]. Mathematicians have only been able to compute the answer for the simplest braid group $B_3$ – and they do not have a simple or intuitive formula. We believe the two groups used, $S_10$ and $B_3$, are sufficiently large with $|S_{10}| = 10!$ and $B_3$ being an infinite group.

## Response to Reviewer KZHf
> It was not clear to me how the learned representations were different from precomputed ones and why they were better for a given task. 

Our results parallel other results in deep learning showing that learned feature representations often outperform expert-engineered features. The precomputed representations we used are a natural choice for matrix representations for the groups used. We use the 10x10 permutation matrices to represent $S_{10}$ but this may not be the most useful representation for every task. Our approach is designed to automatically learn a representation that is useful for the given task.

> What happens when the size of the group representations is larger?

Our method can scale to large representations using MatrixNet-MC. MatrixNet-MC assumes a block diagonal structured representation, which is efficient since it means the number of trainable parameters grows asymptotically linearly instead of quadratically. For well chosen block sizes, this does not harm expressivity since many group representations will be block diagonal with respect to a good choice of basis. We include additional MatrixNet results on the braid group task with representation sizes roughly doubled.

| Model | MSE | Acc. |
| - | - | - |
| MatrixNet | 0.975 | 85% |
| MatrixNet-MC | 0.052 | 96% |
| MatrixNet-LN | 4.5e-4 | 100% |
| MatrixNet-tanh | 1.1e-3 | 100% |

## Response to Reviewer puxG

> For the experiment, I think it only solved a mathematical problem at simple setting. 

The task used for our braid group experiment in section 5.2 is an open mathematical research problem. This task is limited to $B_3$ since the multiplicity counts are not known for any larger braid groups. The experiment over $S_{10}$ in section 5.1 was chosen to provide a well-studied group to compare precomputed representations against the learned representations of MatrixNet.

Meta-review: The paper proposes to learn symmetry groups so that, in turn, to learn optimal representations for downstream tasks at hand. In particular, MatrixNet is proposed, which takes as input generators, then combines them in a Matrix Block that returns an invertible square matrix. Then for a new group element, its matrix representation is the product of the matrix representations of the respective generators that are needed to generate the group element. A parameterized neural network form of the Matrix Block is proposed, as well as variations. An interesting experiment is on a mathematical problem of predicing Jordan-Holder multiplicities, which to be frank is out of my domain, but seems relevant and interesting for the community.

The reviews are mixed, with most reviewers appreciating the novelty and the presentation of the paper. The main complaint is whether the experiments represent a real-world setting, and if a mathematical problem constitutes really a real-world problem. I think it does, or at least it is an interesting new direction to consider for possible ways to validate machine learning models.

Further, while reading the paper, I think some subsections of related work are missing, specifically on 'Symmetry Discovery', eg 

Yang et al., Latent Space Symmetry Discovery, PMLR, 2024
Gabel et al., Learning Lie Group Symmetry Transformations with Neural Networks, TAGML, 2023
Dehmamy et al., Automatic Symmetry Discovery with Lie Algebra Convolutional Networks, NeurIPS, 2021

and in 'Mathematically Constrained Networks' a subsection on mathematical programming for imbuing constraints to neural networks:

Pervez et al., Differentiable Mathematical Programming for Object-Centric Representation Learning, ICLR 2023
Pervez et al., Mechanistic Neural Networks for Scientific Machine Learning, ICML 2024

I suggest the reviewers incorporate the reviewers' suggestions and extend their related work accordingly.
Decision: Accept (poster)