Machine Learning meets Algebraic Combinatorics: A Suite of Datasets Capturing Research-level Conjecturing Ability in Pure Mathematics
Authors: Herman Chau
Helen Jenne
Davis Brown
Jesse He
Mark Raugas
Sara C. Billey
Henry Kvinge
Venue: ICML 2025 oral (2025)
Keywords: Datasets
AI for math
Mathematical reasoning and conjecturing
Algebraic combinatorics
Primary area: applications->everything_else
TL;DR: We introduce a collection of mathematics datasets representing foundational or open problems in algebraic combinatorics aimed at conjecturing capability in machine learning systems

Abstract: With recent dramatic increases in AI system capabilities, there has been growing interest in utilizing machine learning for reasoning-heavy, quantitative tasks, particularly mathematics. While there are many resources capturing mathematics at the high-school, undergraduate, and graduate level, there are far fewer resources available that align with the level of difficulty and open endedness encountered by professional mathematicians working on open problems. To address this, we introduce a new collection of datasets, the Algebraic Combinatorics Dataset Repository (ACD Repo), representing either foundational results or open problems in algebraic combinatorics, a subfield of mathematics that studies discrete structures arising from abstract algebra. Further differentiating our dataset collection is the fact that it aims at the conjecturing process. Each dataset includes an open-ended research level question and a large collection of examples (up to 10M in some cases) from which conjectures should be generated. We describe all nine datasets, the different ways machine learning models can be applied to them (e.g., training with narrow models followed by interpretability analysis or program synthesis with LLMs), and discuss some of the challenges involved in designing datasets like these.

Machine Learning Meets Algebraic Combinatorics: A Suite of Datasets Capturing Research-level Conjecturing Ability in Pure Mathematics
Herman Chau
1
Helen Jenne" 22 Davis Brown 23 Jesse He 24 Mark Raugas? 2
Sara C. Billey
Henry Kvinge 21
Abstract
is now increasing evidence that frontier AI systems are capable of performing tasks requiring high-level reasoning capabilities. These trends have led to excitement around the use of machine learning in mathematics. Much of this research explores the use of LLMs and related models to aid in proof writing and mathematical formalization (Song et al., 2024; Yang et al., 2024). While this is an essential part of the mathematician's 's workflow, there is also need for machine assisted conjecture generation using (what we call) 'raw' mathematical data. Before identifying a claim that they want to try to prove, a mathematician needs to work through many examples to build intuition and better understand their object of study. For example, when trying to better understand the coefficients of partieular family of polynomials (e.g., Kazhdan- Eazhdan-LLsszigg Lusztig polynomials in Section 4), mathematician may search through countless examples, looking for patterns or other features of interest that may be the basis of future theorems.
With recent dramatic increases in AI system capa- bilities, there has been growing interest in utiliz- ing machine learning for reasoning-heavy. quan- titative tasks, particularly mathematics. While there are many resources capturing mathematics at the high-school, undergraduate, and graduate level, there are far fewer resources available that align with the level of difficulty and open ended- ness encountered by professional mathematicians working on open problems. To address this, we in- troduce a new collection of datasets, the Algebraic Combinatorics Datasel Repository (ACD Repo), representing either foundational results or open problems in algebraic combinatorics, subfield of mathematies that studies discrete structures arising from abstract algebra. Further differen- tiating our dataset collection is the fact that it aims at the conjecturing process. Each dataset in- cludes an open-ended research level question and large collection of examples (up to 10M in some cases) from which conjectures should be gener- ated. We describe all nine datasets, the different ways machine learning models can be applied to them (e.g., training with narrow models followed by interpretability analysis or program synthesis with LLMs), and discuss some of the challenges involved in designing datasets like these.
Existing applications of machine learning raw mathe- matics data tend fall into several clusters. The first are toy problems (for which we already know many solutions), which are used by the interpretability and science of deep learning communities as a stand-in for more complicated real-world tasks (Zhong et al., 2024; Nanda et al., 2023; Liu et al., 2023; Lee et al.). Another group uses reinforcement learning methods to search for counterexamples to conjec- tures (Charton et al., 2024; Mehrabian al., 2024; Wagner, 2021). There is also growing body of work coming from the mathematics community where off-the-sheil ML meth- ods are just one of several tools used to make progress on a specific problem (Coates et al., 2024; Wagner, 2021; Bao et al., 2022; Kazalicki & Vlah, 2023; Davies ct al., 2021). Finally in a few instances, foundation models have begun to be deployed to address specific mathematical questions (Romera- Paredess et al., 2024).
1. Introduction
Modern approaches to machine learning (ML) have been shown to be capable of extracting sophisticated patterns from large and complex datasets. At the same time, there
'Equal contribution 'University of Washington PPacific North- west National Laboratory University of Pennsylvania University of California, San Diego Correspondence to: Herman Chau thchau@uu.cdde Henry Kvinge <henry Seatinng@@pnn...g. gov>.
While these works either present interesting methodological progress in ML or valuable results in mathematics, none aim to provide a range of datasets accessible to the broader ML community that represent open or equivalently challenging research-level problems. To fill this gap. we present the
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Machine Learning Meets Algebraic Combinatoric
Algebraic Combinatorics Dataset Repository (ACD Repo)', collection of 9 datasets consisting of many examples along with an associated question(s). Our collection includes both open problems (e.g., the combinatorial interpretation of
orders, graphs, permutations, partitions) (Stanley, 201 la:b). Ideas and tools from combinatorics play an essential role in many other fields of mathematics and continue to have a Schubert polynomial structure constants) and classic prob- lems whose solution is a major result in the field (e.g., a combinatorial method of calculating the characters of irre- ducible symmetric group representations).
strong impact on computer science and physics. Algebraic combinatorics is a subfield of combinatorios that applies combinatorial methods to problems arising from abstract algebra, particularly representation theory and algebraic geometry.
We choose restrict ourselves to algebraic combinatorics (an area of mathematics that studies discrete structures aris- ing from abstract algebra) because (i) it requires less back- ground theory to understand, making it generally more ac- cessible to a broader range of researchers, (ii) there already exist specialized software libraries (e.g.. Sage (Stein et al., 2024)) designed to efficiently compute many quantities of interest in algebraic combinatorics, and (iii) by nature of being discrete, the objects of interest in algebraic combi- natorics tend to be more amendable to representation on a computer.
Partitions: We use the word partition in this work mean an integer partition. An integer partition of n E N is a sequence of positive integers (71, ......., (1k)) such that n ni + 122 +... + n2 and n 2 n2 2 ... 2 f2g. We use the standard notation μr + n to denote that j is a partition of n. A partition (MI...., n*) is often visualized as a Young diagram, with (in English notation) n, left justified square cells in the first row, ng left justified square cells in the second row, etc. See Figure (left) for an example of a Young diagram corresponding to the partition (3, 2).
Each dataset has both an open-ended mathematical ques- tion and a related ML friendly task associated to it. The idea is that model that can effectively solve the ML task has probably learned information that could offer insight into the broader mathematical question. For example, the open question may be finding combinatorial interpretation of Schubert polynomial structure constants (Section 4.6), which are indexed by triples of permutations. In this case the ML task is to predict the structure constant from the three permutations. For each dataset, we provide context and mo- tivation for the problem, the basic statistics of the dataset, as well as the performance of some basic off-the-shelf models.
2
1
2
3
5
3
5
6 7
6
6
Figure I. (Left) A Young diagram for the partition (3, (Cen-
ter) A standard Young tableau for the partition (3, 2,).. (Right) A semistandard Young tableau for the partition (3,2,
Young tableaux: Including extra decorations in the cells in a Young diagram can capture ubiquitous combinatorics found across representation theory and other fields. A Young lableau corresponding
We note that these datasets are not designed to be bench- marks in the traditional sense. High performance in terms of standard metrics such as accuracy may be of little value if one is unable to extract a mathematical insight that leads to a fruitful conjecture. In Section 5 we give two exam- ples illustrating how this might be done: (i) by performing careful interpretability analysis of performant narrow model (Davies et al., 2021; He et al., 2024) or (ii) by using a LLM that can communicate its reasoning via (for exam- ple) code (Austin et al., 2021; Romera-Paareeeee Novikov et al., 2025). We hope that these datasets will en- able the development of even more effective approaches in the future.
Young diagram À H n is label- ing of the cells of X by an alphabet of symbols. In this work we will consider two types of Young tableaux. A standard Young tableau corresponding to partition X + n is a labeling of the cells of A by 1,2, 1,2,..., n. such that the integers strictly increase as one moves down a column or left to right across row (Figure 1 (center)). The definition of a semistandard Young tableau is analogous except that the entries are only assumed to weakly increase one moves from left right along row (see Figure 1 (right)).
et al., 2024;
Permutations: Permutations are familiar in machine leamn- ing from their central role in computer science as well as their relevance to symmetries data and neural networks (Entezari et al.; Ainsworth et al., 2023; Godfrey et al.. 2022: Keriven & Peyré, 2019; Zaheer et al., 2017: Lee et al., 2019). There many ways to represent a permutation. In this paper we use one-line notation, which is best illustrated through an example. Suppose w is the permutation of the set of elements f1,2,3, 4} that swaps and and 3 and 4. Then in one-line notation we write w = 2 1 4 3. 2 is in the first position since 1 is sent to 2, 1 is in the second position
2. The Cast of Characters: Partitions, Permutations, and Partial Orders
The field of combinatorics studies broad range of problems in mathematics centered around discrete objects (e.g, partial
'https://githult httpttthhh...c..oeeeeeeeeeeee wwwpp/nwwwLL.lll//.//o//o https://huggingtftt co/ACDRepo
and
2

---

Machine Learning Meets Algebraic Combinatoric
since 2 is sent to 1, 4 is in the third position since 3 is sent to 4, and 3 is in the fourth position since 4 is sent to 3.
and answer in non-research settings (Saxton al., 2019). In this work we look at large volumes of the raw mathe- matical data associated to research-level problems. Other work in this vein includes the search for counterexamples in graph theory ( (Wagner, 2021), the search for connections between different knot invariants (Davies et al., 2021), the classification of Q-Fano varieties (Coates et al., 2024), and Clifford invariants of ADE Coxeter elements (Chen et al., 2024). Unlike these works which aim to shed light on spe- cific problems, this paper's goal is to introduce datasets so that both the expert and non-expert can explore the use of machine learning for research-level mathematics problems.
Permutations can also be written as sequences of transpo- sitions of adjacent elements. For instance, the permutation a = 3 1 2 can be formed by first swapping 2 and 3 and then the newly adjacent and 3: 1 2 3 + 1 3 2 - 1 2. If we denote transposition of the ith and (i + 1)st element as 8; and read from right to left (as is the convention) then o can be written as $1 $2. A sequence of adjacent transpositions Si, 8ig ...i,,.8,, Si, corresponding to permutation a is called reduced word if there is no other representation that uses fewer than adjacent transpositions to represent a. Two reduced words are considered commutation equivalent if one can be obtained from another by swaps of the form s45; 7 Sj5; where ji > 1. Finally, 3412 pattern is quadruple (aa; aj. agg ae) such that j < k < & but ag ag < 0g < dg. Patterns have deep connections to algebra and geometry (Billey, 1998).
This new field of machine learning looks at applying machine learning methods to al- gorithmic data (Veličković, 2023). Like mathematies, apply- ing machine learning to algorithmic data provides setting where arbitrarily large amounts of data can be generated. Further, as with mathematics, working with algorithmic data allows us to manipulate certain aspects of the problem (e.g.. complexity) in ways not possible in noisy real-world data. The differences between our work and the primary neural algorithmic reasoning benchmark (Veličković et al., 2022), are substantial. Most notably, the focus of these datasets is not scientific discovery like the ACD Repo.
Neural Algorithmic Reasoning:
In the discussion above we implicitly think of permuta- tions of n as bijective functions from (1.2...., n) + n]. Using this perspective, one can define the composition of two permutations. The symmetric group, denoted Sn, is defined the group of permutations n elements using composition as the group operation. The se- quence of transpositions $1 from the previous paragraph gave an example of the composition of two permutations.
4. Dataset Descriptions
Posets: A partially ordered set (poset) is a set P of objects equipped with a binary relation, typically denoted "s", that is reflexive, antisymmetrie, and transitive. This means that for all elements a, b, e E P: (l) S a, (2) if a S 6 and b S then b = and (3) if < b and b S c, then a S Unlike total orders which are more familiar (c.g., Z). in a partial order some pairs of elements may be incomparable. An example of a partially ordered set is the set of all subsets of {1, 3, 4]. ordered by inclusion. This is a partial order and not a total order because 21, is not comparable to (2.3) or to (2, 4). for example. In poset, y covers x if y is greater than with respect to the ordering, and for any 2 such that x < S y. either z = or 2 = y. In this example, (1,2,4) 4} covers {1,2), (2,4), and {1, 4), but not (2), or (4).
Here we describe the datasets that are currently included in the ACD Repo. Further details, including dataset statis- tics, additional problem context, and the methods used for generating the datasets can be found in Appendix B.
Most approaches that use ML to accelerate mathematics research require one to be able to train a model that in some sense understands how to solve a version of the problem. The second (and usually harder) step is to then extract math- ematical insights from this model. We aim to give some sense of the difficulty of step one by providing some initial baselines for off fff-the-shelf models: both narrow (Tables 1, 3, and 4) and L.MSs (Table 2 and Table 6). The details of training and testing can be found in Appendix B. 10. When training narrow models, we use vanilla architectures as most readers will have some intuition and familiarity with these. It is almost certainly the case that one can do much better by refining model architecture, training routine, data repre- sentation, and task structure to better fit the problem. We leave this to users of the datasets.
3. Related Work
There is a growing body of work that uses machine learning based methods to assist in mathe- matics research. With the growing popularity of theorem proving languages such as Lean (Moura & Ullrich, 2021) or Coq (Coq development team, 2004), many of these papers focus more on the proof-creation part of the mathematician's workflow (Song et al., 2024; Yang et al., 2024; Azerbayev et al.). Others aim for more structured, text-based question
AI for Mathematics:
4.1. Computing Characters of Irreducible Representations of the Symmetric Group (Foundational Result)
One way to understand the algebraic structure of permuta- tions (symmetric groups, S,)) is through their representation
3

---

Machine Learning Meets Algebraic Combinatoric
Dataset
Logistic regression
MLP
Transformer
Guessing largest class
Lattice paths
n = 10
66.2%
90.6% = 0.8%
65.3% 0.0%
66.2%
95.8% +: 0.3%
69.4%
66.3%
12
66.5%
98.6 % 0.1%
86.2% 14.2%
66.3%
Weaving patterns
6
70.4%
86.1 % #: 0.2%
85.9% :: 2.3%
63.3%
7 = 7
85.8%
99.3 % = 0.2%
99.9 % # 0.4%
85.0%
Claster algebra quivers
40.3 %
86.5 % # 1.9%
92.9% 0.5%
17.7%
Grassmanian cluster algebras
%
99.5 %.d.
50.0%
Schubert polynomials
88.8%
93.1 1% 2.6%
94.6% + 1.0%
52.3%
5
90.6%
97.5 % 0.2%
96.2% + 1.1%
49.9%
6
89.7%
99.8 16 1. 0.0%
91.3% 1. 8.0%
50.1%
mHeight
91.4%
99.4 % 0.3%
99.7% 0.4%
91.4%
% 0.6%
99.9%
93.2%
10
%
99.9%
0.6%
94.2%
Table 1. Off--hhe-stee she-th model accuracy on classification datasets. Results are averaged over three random weight initializations with 95% confidence intervals after hyperparameter search outlined in Table B.10
theory (Sagan, 2013), which converts abstract algebraic questions into linear algebra questions that are often eas- ier to solve. A representation of group on vector space V. is a map G + GL(V) that converts elements of G' to invertible linear maps from V to V and respects the compositional structure of the group. A basic result in rep- resentation theory says that all representations of a finite group can be decomposed into atomic subspace building blocks called irreducible representations. Amazingly, irre- ducible representations are themselves uniquely determined by the value of the trace on matrices, ((g), where g ranges over subsets of G called conjugacy classes. These values are called characters.
ML task: Train a model that can take two partitions of n, A and μr, and predict the corresponding irreducible symmetric group character X Does this model learn a known algo- rithm? If it does not, what method does the model learn? Any novel approaches computing irredueible symmetric group characters would be of high interest to the mathemat- ics community. This is framed as a regression task.
Input/output: Two integer partitions + an integer.
How hard is it?: As can be seen in Table 3, narrow models struggle this task. This may relate to the long-tail distri- bution of characters (see Figures 3-5) the complexity of the problem (Ikenmeyer et al., 2024).
The representation theory of symmetric groups has rich combinatorial interpretations. Both the irreducible repre- sentations and the conjugacy classes of S, are indexed by partitions of n and thus the characters of irreducible rep- resentations of S,, are indexed by two partitions of n. For X, + write Xd. This combinatorial connection is not superficial; there are algorithms (e.g., the Murnaghan- Nakayama rule (Stanley, 2011b)), which allow calcula- tion of irreducible characters via simple manipulation of the Young diagrams for and μ without any reference to more abstract algebraic structure. We provide datasets for 18, 20, 22.
4.2. The mHeight Function of a Permutation (Key Construction)
Truly challenging open problems in mathematics often re- quire the development of new mathematical constructions (or even entire new areas of mathematics). This dataset represents a modest example of this. The mHeight function is a statistic associated with a permutation that relates to all 3412 1-1--ptttemss in the permutation. It was developed and plays crucial role in the proof by Gaetz and Gao (Gaetz & Gao, 2024) which resolved a long-standing conjecture of Billey and Postnikov (Billey & Postnikov, 2005) about the coefficients on Kazhdan-Lusztit polynomials (Section 4.4).
The task of predicting the mHeight function represents an interesting opportunity to understand whether non-trivial intermediate step an important proof can be learned by machine learning.
"The chameeer 204413 isO"
4

---

Machine Learning Meets Algebraic Combinatoric
Let a di... d1...0n Sn be a permutation containing at least one occurrence of 3412 pattern. Let (a;, ajj ak. at) be 3412 pattern so that i < < k < E but dg < dg < a; < 0j (see Section 2 for more details). The height of (0g, 4j, ak, de) is 0; 0g. The mHeight of a is then the minimum height over all 3412 patterns in o. We provide datasets for 71 = 8,9, 9, 10.
Dataset
al -ini
GPT-4o Mini
GPT-4o
In context learning
72 10
95.4%
89.5%
95.5%
2
97.0%
97.0%
97.0%
Program synthesis
10
94.2%
94.2%
94.2%
22 11
95.5%
95.5%
95.5%
== (67542123
"The mHeight function is 2"
Table 2. Success of GPT-4o-mini and GPT-40 solving the mHeight function task via either in- in-context learning or program synthesis. Hyperparameters for these experiments can be found in Section B.11.
ML task: Predict the mHeight of a permutation. Since mHeight can take limited number of values for small n, this is framed as a classification problem.
ML task: Predict whether Young tableau indexes cluster variable. This is a binary classification task.
Input/output: One permutation - one of several possible integers.
Input/output: A SSYT of shape 3 x 4 with entries drawn from f1,2..... 12] + Boolean indicating whether the tableau indexes a cluster variable or not.
How hard is it?: Both LLMs (Table 2) and narrow models (Table 1) achieve high accuracy (though note that the dataset is heavily imbalanced for larger n). Understanding whether the models actually learn some approximation of mHeight or exploit other correlations remains to be studied.
How hard is it?: Even naive methods can result in high accuracy (Table 1), suggesting that there is something inter- esting here to learn.
4.3. Grassmannian Cluster Algebras and Semistandard Young Tableaux (Open Problem)
4.4. The Coefficients of Kazhdan--uszztz Polynomials (Open Problem)
The Grassmann manifold Gr(k; ni) is the set of full-rank x n matrices up to equivalence of elementary row operations (equivalently the space whose points are k-dimensional sub- spaces in R.) Grassmannians are of fundamental geometric importance and are a central tool in a model of quantum field theory known as supersymmetric Yang-Mills theory (Golden et al., 2014).
Kazhdan-Lusztie (KL) polynomials are integer polynomials in a variable q that (for our purposes) are indexed by a pair of permutations (Kazhdan & Lusztig, 1979). We will write the KL. polynomial associated with permutations x and w as (z.w(g). For example, the KL. polynomial associated with permutations x w 6 8 9 10 I 11 2 2 is
4 3 2 7 6 5 10 9 8 11 and
(a.w(() 1 16g + 103"" + +3377 5609" + 5200 2754" + 6og" 3y
Among the many aggebraic-commbinttriial properties of Grassmannians an algebraic structure on its coordinate ring making it something called cluster algebra (Scott, 2006; Williams, 2014). A recent result of Chang, Duan, Fraser, and Li (Chang et al., 2020) parameterize cluster vari- ables of the Grassmannian coordinate ring in terms of equiv- alence classes of semistandard Young tableaux (SSYTss. Not every SSYT indexes a cluster variable and a natural question to ask is which are valid cluster variable indices. necessary condition is that the tableau is of rectangular shape. We follow the set-up of (Cheung et al., 2023) who first applied machine learning to this problem, though we choose a different method of sampling tableaux that do not index cluster variables. The math question is thus to find a concise combinatorial characterization of those SSYT that index a cluster variable. We provide a dataset for Gr(3, 12), restricting to rank 4. This corresponds to rectangular SSYT of shape 3 x with entries drawn from {1, ...,, 12).
(this example was computed by (Warrington)). KL poly- nomials have deep connections throughout several areas of mathematics. For example, KL polynomials are related to the dimensions of intersection homology in Schubert cal- culus and the representation theory of the Hecke algebra. They can be computed via recursive formula (Kazhdan & Lusztig, 1979), nevertheless, in many ways they remain mysterious. For instance, there is no known closed formula for the degree of (g).
Mathematicians would like a better understanding of the coefficients on powers of q in (q). For example, one question concerns the coefficient on term g flx)-((w)-1/2 (where {(x) is statistic called the length of the permuta- tion), which is known as the p-coefficient. Both the constant term and coefficient on g are well-understood. We provide datasets for n 5, 6,
x = (1 3245768)
"In Pz.(() the coefficient on 9 is 1"
"This semistandard Young rableau indexes cluster variable"
w = (4127856)
6
9
11 12
ML. Tasks: Predict one or more coefficients of Pa.
5

---

Machine Learning Meets Algebraic Combinatoric
given x and w. Empirically, for fixed n and power of g, coefficients tend only take a few values as classification task.
polynomials indexed by permutations of Sn Developed study the cohomology ring of the flag variety, they have deep connections to algebraic geometry, Lie theory, and rep- resentation theory. Despite their geometric origins, Schubert polynomials can be described completely combinatorially (Billey et al., 1993; Bergeron & Billey, 1993), making them a well-studied object algebraic combinatories. An impor- tant open problem in the study of Schubert polynomials is understanding their structure constants. When two Schubert polynomials are multiplied, their product is a linear com- bination of Schubert polynomials, G,6 = En Ca Byy The lae are conjectured to have combinatorial description or formula (most likely related to permutations 3, and ?). To give an example of what we mean by combinatorial description, the structure constants of Schur polynomials (a special type of Schubert polynomial) count the number of semistandard tableaux satisfying certain properties.
frame this
Input/output: Two permutations + one of several integers.
How hard is it?: We provide both accuracy (Table 4) and Fl-scores (Table 5) for these imbalanced datasets. Narrow models perform well.
4.5. The Kobinson-Schensteed Knuth Correspondence (Foundational Result)
The Sebinson-Schensted Knuth (RSK) algorithm (Robin- son, 1938; Schensted, 1961) gives bijection between pairs of semistandard Young tableaux of the same shape and matrices with non-negative integer entries. The special case we consider (which is sometimes called the Robinson- Schensted algorithm) restricts to a bijection between pairs of standard Young tableaux and permutations in Sn. This correspondence is significant in algebraic combinatorics be- cause it connects two of the most fundamental objects in the field (see (Stanley, 1984) for additional history and context).
Each instance in this dataset is a triple of permutations (a, 7), labeled by its coefficient c/ in the expansion of the product 6,, G.. Not all possible triples of permuta- tions are included; the dataset consists of an approximately equal number of zero and nonzero coefficients. We provide datasets for n = 4, 5, 6.
Given its fundamental importance, it would be interesting to see whether a model learns the RSK algorithm given enough examples of the correspondence. To this end the dataset consists of triples: two standard Young tableaux of size and their corresponding permutation (obtained via the RSK algorithm). We provide datasets for = 8, 9.
(1 (1243). == 43 #= (153244
"The structure constant CA on S, in S, S, is
ML task: Train a model that given three permutations a, 8,8% can predict the associated structure constant ca, Since these structure constants only take a few values for small permutations, we frame this as classification task.
"The permuration
(1 6 67528447
Input/output: Three permutations -> one of several inte- gers.
ML task: Given a pair of standard Young tableaux, pre- dict the corresponding permutation. The task framed as regression and for the permutation target we use vector representing its descent set (we found this to be an easier target for regression than representing the permutation in 1 l-line notation).
How hard is it:: Both small MLPs and transformers can achieve high accuracy (Table 1) as well as LLMs via pro- gram synthesis (Table 6). Some of the latter is an artifact of how we originally sampled zero-valued structure constants.
Input/output: Two standard Young tableaux of the same shape + a permutation.
4.7. Partial Orders on Lattice Paths (Open Problem)
Consider northeast lattice paths that travel along the edges of a grid from (0. to (a, b), only taking steps north and east and never passing through the diagonal y = x, where a and b are relatively prime. (Schiffler, 2023) defines two order relations on such paths called the matching ordering ($ (r) and the Lagrange ordering (SL), motivated by ques- tions in number theory. The matching ordering assigns a number to each lattice path based on the number of perfect matchings of an associated snake graph, while the Lagrange ordering assigns a number to each lattice path equal to the Lagrange number of a certain continued fraction. These numbers each define the respective partial order. Mathe- maticians would be interested to better understand the rela-
How hard is it?: None of the small, narrow models learned this task in our experiments, achieving results similar or worse than simply guessing the mean of all permutations (Table 3). Like symmetric group character calculation, the RSK algorithm has intricate combinatorial rules that may require larger and more capable models or more elaborate training strategies.
4.6. Schubert Polynomial Structure Constants (Open Problem)
Schubert polynomials (Bernstein et al., 1973; Demazure, 1974; Lascoux & Schützenberger, 1982) are a family of
6

---

Machine Learning Meets Algebraic Combinatoric
tionship between these orders (Apruzzese & Cong, 2023). We provide datasets for lattice paths from (10, 9), (11, 10), (12, 11), and (13, 12).
Task: Find simple rules determine which of different mutation equivalence classes quiver belongs to. This is framed as a classification task.
Input/output: An adjacency matrix + one of 7 different possible labels.
lattice path pair cover in the Lagrange partial order'
How hard is it?: Transformers and MLPs achieve reason-
able accuracy on this task Table 1). (He al., 2024) was able to train far more performant model (accuracy 99%) and re-discover several known characterization theorems using specialized graph neural network architecure.
Given a pair of lattice paths (w, w'), train a model that can predict whether w' covers w (see Section 2) in either the matching or Lagrange order.
ML task:
4.9. Weaving Patterns (Open Problem)
Input/output: Two lattice paths - whether the pair is cover relation in matching or Lagrange order.
Weaving patterns are n 1-matrices with {1, ......... ,n. entries introduced by Felsner (1997) to study the num- ber of reduced decompositions of the permutation a = n. n - ... up to commutation equivalence. The number of such objects also counts the number parallel sorting networks, the number of rhombic tilings of regular poly- gons, and is connected to the study of the higher Bruhat orders (Chau, 2024). An O(n²) algorithm for determining if given 21, n -matrii is a valid weaving pattern exists but gives no additional insight into the structure of weaving patterns and correspondingly the asymptotics of reduced decompositions.
How hard is it?: MLPs achieve good performance (Ta- ble 1). On the other hand, we found it challenging to train performant transformers.
4.8. Mutation Equivalence of Quivers (Open Problem)
Quivers and quiver mutations are central to the combina- torial study of cluster algebras, algebraic structures with connections to Poisson Geometry, string theory, and Teich- muller theory. Quivers are directed (multi)graphs, and quiver mutation is local transformation centered at cho- sen node of the graph that involves adding, deleting and reversing the orientation of specific edges based on set of combinatorial rules. A fundamental open problem in this area is finding an algorithm that determines whether two quivers are mutation equivalent (one can traverse from one quiver to another by applying mutations). Currently, such algorithms only exist for special cases, including types A (Buan & Vatne, 2008), D (Vatne, 2010), and B, C, and δ (Henrich, 2011 1). The and Ć' types correspond to the classes BD and BB in our dataset, respectively. Consistent with Sage we use the naive notation, which specifies a quiver by indicating the two ends of the diagram, which are joined by path (Musiker & Stump, 2011). 1). To our knowledge, the remaining classes in this dataset (E, DE, BE) lack characterizations. Recent work has explored whether deep learning models can learn to correctly predict if two quivers are mutation equivalent (Bao et al., 2020). (He et al., 2024) utilized an alternative version of this dataset to re-discover known characterization theorems. The dataset consists of adjacency matrices for quivers drawn from 7 different mu- tation equivalence classes (A, D, E, DE, BE, BD, and B BB).
The enumeration of reduced decompositions up to com- mutation equivalence has been studied by many including Knuth and Stanley. An exact formula is likely out of reach, so asymptotic upper and lower bounds are of great interest. ML models that can detect necessary or sufficient conditions for a matrix to be valid weaving pattern have the potential to lead to substantial improvements in the upper bound.
Each dataset is mixture of enriched weaving patterns and non-weaving pattern matrices with (1,2, .,.2,....7) eentries.. We provide datasets for = 6, 7.
1
2
3
4
5
1
2
3
5
6
"This matrix weaving
pattern"
1
3
4
5
6
2
3
4
5
6
ML task: Train model to classify whether 21,2..... n matrix a weaving pattern or not. This task framed as binary classification.
Input/output: Annxn- 1, 21, ........)mmariii +> binary label.
corresponds to muration equivalence class D,o"
How hard is it?: Especially in the n = 7 regime, even small ML.ss and transformers achieve high accuracy (Table 1).
7

---

Machine Learning Meets Algebraic Combinatoric
5. Case Studies of Applying ML to ACD Repo Datasets
code that solves mathematical task. This is often known as program synthesis.
We applied this procedure to the Schubert polynomial struc- ture constants dataset. There are various additional steps that one can use in program synthesis to improve code gen- eration (for instance, in PRomera-Paredeer et al., 2024) the authors interleave code generation with the application of an evolutionary algorithm). For our experiments we only performed single round of 100 generated Python programs with no additional steps. We were surprised to find that (if we provided the proper mathematical context in our prompt) ol -iiim, Claude, and sometimes 4o were able to produce Python programs that achieved 100% accuracy on the test set. In originally generating this dataset, we subsampled from the set of zero valued structure constants because these are far more common than non-zero valued structure con- stants. Interestingly, examination showed that in some cases the LLMs had reversed engineered our sampling algorithm, which essentially involved applying a single transposition to one of the three permutations indexing a non non-zero structure constant to obtain a zero structure constant.
We begin this section by providing two examples for how conjectures can be generated using combination of tools from machine learning and an expert-in-the-loop. Both utilize a dataset (or a derivative of dataset) from the ACD Repo. Our purpose is not to propose novel ML methods nor to describe novel mathematics, but rather to provide the reader with prototype examples of how the ACD Repo can be used in practice. For canonical instance where machine learning assisted conjectured generation was used to prove new mathematics see (Davies et al., 2021) (for the conjecture) and (Blundell et al., 2022) (for the theorem).
Graph Neural Networks, XAI, and Quiver Mutation Equivalence: (Davies et al., 2021) was one of the first works to show that mathematical conjectures can be made through careful analysis of a machine learning model trained to solve a task around an open problem in mathematics. The basic outline is (I) train a model on the ML friendly task related to the open problem of interest, (2) analyze the resulting performant model to understand how it making its predictions, and (3) generate conjectures based on this analysis.
It turns out that in the cases we applied program synthesis to (n = 4, 5) this process introduces a spurious correlation in the training and evaluation data?: structure constant 22,9 A O if and only if E(a) ((3) + ((>) o 0 mod 2 where e is a statistic on permutations called the length. We note that the length statistic is not included in the dataset itself or in the prompt we gave the model to initiate program synthesis.
(He et al., 2024) applied this approach to a modified ver- sion of the quiver mutation equivalence dataset from the ACD Repo (Section 4.8). They showed that they could re- discover characterization theorems for type D and type D quivers. To do this, they trained novel variant of a mes- sage passing graph neural network specifically designed to be able to capture the presence of subgraphs (which have been used to determine membership to several other notable mutation equivalence classes). Having trained a model that achieves high accuracy, the authors used latent space clus- tering within classes to break down the problem into distinct cases. Then they used a graph neural network explainability technique called PGExplainer (Luo et al.. 2020) to identify the subgraph motifs that characterize each of these clusters. In the end they re- re-discovered theorems from (Vatne, 2010) and (Henrich, 2011) 1).
Though this exercise did not provide any useful mathemati- cal insights, it does show that the LLM-based approach to conjecture generation carries the benefit that in some cases the LLM can exploit background tools and concepts (such as the length of permutation) not available to narrow models. More speculatively, it may be that program synthesis tends to elicit such background information a greater extent than other methods. For comparison, asking these models to solve this task directly using in-context learning, resulted in only 50-60% accuracy.
6. Discussion
Program Synthesis for Schubert Polynomials: The pre- vious example highlighted the use of narrow models for conjecture generation. In our second example we describe the use of foundation models for a less labor- but more compute-intensive version of conjecture generation. Con- jecture generation is crucial component in current evo- lutionary approaches to mathematical discovery, notably Cooeera-Paredes et al., 2024; Novikov et al., 2025) (though note that there are other non-LLM-bran based approaches to con- jecture generation (Gauthier& Urban, 2023; Raayoni et al., 2021; Fajtlowicz, 1988)). The basic idea is that because computer code is more interpretable than raw ML predic- tions, one an LLM to generate (human-interppretall interpretable)
Narrow Model Performance: We trained small MI.Ps, decoder only transformers, and logistic/linear regression models on each dataset in the ACD Repo and measured their test performance using either accuracy (classification tasks) or mean squared error (regression tasks). The goal of this exercise was to approximately measure the relative difficulty of training a performant model (though note that a performant model alone does not translate to mathematical discovery). We found that MLPs tended to perform most consistently across tasks and therefore might be a good
"This issue has since been fixed.
8

---

Machine Learning Meets Algebraic Combinatoric
place to start when exploring the use of smaller models. Transformers also performed well but struggled in few cases (such as the lattice paths dataset).
terns (but care must be taken so that the model does not simply discover one's sampling procedure).
Unlike other domains where data is more rare, the process of mathematical discovery using machine learning may in- volve more directly optimization over both a model and dataset. That is, modifying model and dataset so that the model learns more informative, robust, and effective repre- sentations.
Though we cannot prove why some dataset/architecttre pairs failed to perform well (especially for datasets cor- responding to open problems), several factors should be considered. Firstly, the loss landscape for tasks involving discrete structures like permutations seems to be less fa- vorable for architectures and training approaches that we currently use in deep learning. For example, many proper- ties in combinatorics are highly sensitive to small changes to the input (e.g., the parity of a permutation). Such tasks may be intrinsically challenging within current frameworks (Hahn & Rofin, 2024).
Finally, there exist near limitless number of open problems in mathematics and finding the optimal ones for inclusion was challenging. We aimed to find problems such that (i) some working mathematicians care about the solution, (ii)
a significant amount of data (at least > 10k instances, but preferably many more) could easily be generated, and (ii)
Further, the representation of input data can have a large impact on model performance. For instance, we found it challenging to train narrow model (MIP or transformer) to predict the parity of a permutation represented in one- line notation. But this was feasible when permutations were represented via their inversion vector. While effective representations can sometimes be found when a solution is already known, this strategy is not available for open problems. When designing these datasets we chose not to optimize input or output representation, leaving this to users.
there is an ML friendly simplification of the problem (this can be challenging with extremely abstract problems for instance).
A Challenge to the Interpretability Community: There are now many cases where researchers in mechanistic in- terpretability have worked out detailed descriptions of the ways in which models of various sizes solve tasks. For example, reverse engineering the ways in which small trans- formers perform mathematical tasks such as modular arith- metic (Nanda et al., 2023). While such examples may fall short of larger goals such as being able to audit foundation models on fuzzy real-world tasks, they align perfectly with the needs of Al for math where an algorithmic description of the method by which a model has learned to solve an open problem could easily translate into a theorem (or least novel algorithm). As such, we propose these datasets as an achievable but high impact opportunity for the field of mechanistic interpretability.
Unsurprisingly, we found that larger datasets were generally associated with better model performance. This is true even in the case where generating a larger dataset required increasing n. and thus making the problem potentially more complex (e.g., working with partitions of n + 1 rather than partitions of n). There were some tasks however that seem hard even when the dataset size is increased. As can be seen in Table 3, performance regressing symmetric group characters is very poor. This may relate to the complexity of the task (calculating symmetric group characters is known to belong to # #P (Hepler, 1996)) or to the distribution of symmetric group characters which has very long tail.
7. Conclusion
In this paper we introduce the Algebraic Combinatories Dataset Repository, a collection of datasets representing open problems or research-level foundational results that have been structured for machine learning. With all the advancements in model capability and model explainability, we believe that machine learning holds the potential to dra- matically accelerate progress in mathematics. Signs of this are already starting to appear, e.g. (Novikov et al., 2025). On the other hand, mathematics is a vast field and broad, ML-fueled acceleration across sub-disciplines will require tools that can be used by researchers across broad range of institutions. We hope that these datasets will fuel the development of these tools.
Challenges in the generation of mathematics datasets: Imbalance is an issue in several different respects. On the one hand, traditional class imbalance comes up frequently and can be quite extreme for large values of n where asymp- totic properties begin to take hold. More broadly, the mathe- matically interesting objects are frequently interesting pre- cisely because they are rare. For given task, it may be the case that randomly sampled instances will be uninteresting with high probability because they can be predicted or clas- sified for straightforward reasons. One way to mitigate this situation is to subsample for harder examples. This is what we did for several of the datasets in the ACD Repo including Weaving Patterns where we imposed some additional con- straints on the non won-wnaving pattern 2 n matrices to make them harder to distinguish from true weaving pat-
9

---

Machine Learning Meets Algebraic Combinatoric
Acknowledgements
Billey, S. and Postnikov, A. Smoothness of Schubert va-
rieties via patterns in root subsystems. Adv. in Appl. Math., 34(3)447-446,, 2005. ISSN $196-8858,1096 2074. doi: 10. 10.1016/j.aam..... //doi.orgi wwwp://..aa.224.88...0 2004.08..03 00..
This research was supported by the Mathematics for Arti- ficial Reasoning in Science (MARS) initiative at Pacific Northwest National Laboratory. It was conducted un- der the Laboratory Directed Research and Development (LDRD) Program at Pacific Northwest National Labora- tory (PNNL), a multiprogram National Laboratory operated by Battelle Memorial Institute for the U.S. Department of Energy under Contract DE- DE-AC05-76RL01830
URL https: 01 // aam 2004. 08. .03..
Billey, S. C. Pattern avoidance and rational smoothness of schubert varieties. Advances in Mathematics, 139(1): 141-156, 1998.
Billey, S. C., Jockusch, W., and Stanley, R. P. Some combi- natorial properties of schubert polynomials. Journal of Algebraic Combinatorics, 2(4): 1993.
Impact Statement
This paper presents work whose goal is to simultancously advance the field of Machine Learning and Mathematics. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Blundell, C., Buesing, L., Davies, A., Veličković, P., and Williamson, G. Towards combinatorial invariance for kazhdan-lusztig polynomials. Representation Theory of the American Mathematical Society, 26(37):1 2022.
References
Buan, A. B. and Vatne, D. F. Derived equivalence classifi- cation for cluster-tilted algebras of type d,, Journal of Algebra, 319(7):272--2733 2008.
AI Safety Institute, U. Inspect AI: Framework for Large Lan- guage Model Evaluations. URL https:://// //gitt///tt /iithb.. /ttp////ooeeemeettBt/////eeeeeee httptttttssssssssess.....
Chang, W., Duan, B., Fraser, C., and Li, J.-R. Quantum affine algebras and Grassmannians. Math. Z., 296(3-4): 1539-1583, 2020. ISSN 022-5874414332-122 doi: 10. 10.1007/600209-00000 000-00000-0000009-00009 URL https: //t////oiii//./ o 0ttg:/0001007//0000---0-0-22-000
Ainsworth, S., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Apruzzese, P. and Cong, K. On two orderings of lattice paths. arXiv preprint arXiv:2310. 16963, 2023.
Charton, F,, Ellenberg, J., Wagner, A. Z., and Williamson, G. Patternboost: Constructions in mathematics with little help from AI. arXiv preprint arXiv:2411.00560 2024.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le. Q. et al. Program synthesis with large language models. arXiv preprint 10V::2008.07732 2021.
Chau, H. On enumerating higher bruhat orders through dele- tion and contraction. arXiv preprint arXiv:2412.10532, 2024.
Azerbayev, Z., Schoelkopf, H., Paster, K., Dos Santos, M., McAleer, S. M., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations.
Chen, S., Dechant, P.-.,, He, Y.-H., Heyes, E., Hirst, E., and Riabchenko, D. Machine learning clifford invariants ade coxeter elements. Advances in Applied Clifford Algebras, 34(3):20, 2024.
Cheung, M., Dechant, P., He, Y., Heyes, E., Hirst, E., and Li, Clustering cluster algebras with clusters. Advances in Theoretical and Mathematical Physics, 2023.
Bao, J., Franco. S., He, Y.-H., Hirst, E., Musiker, G., and Xiao, Y. Quiver mutations, seiberg duality, and machine learning. Physical Review D, 102(8):086013, 2020.
Coates, T., Kasprzyk, A., and Veneziale, S. Machine learn- ing detects terminal singularities. Advances in Neural Information Processing Systems, 36, 2024.
Bao, J., He, Y.H.,, Heyes, E., and Hirst, E. Machine learning algebraic geometry for physics. arXiv preprint arXiv:2204. 10334, 2022.
Coq development team, T. The Coq proof assistant reference manual. LogiCal Project, 2004. URL http: coo. inria. fr. Version 8.0.
Bergeron, N. and Billey, S. Re-graphs and schubert polyno- mials. Experimental Mathematics, 2(4):257-269, 1993.
Bernstein, I. N., Gel'fand, I. M., and dan', S. I. Schu- bert cells and cohomology of the spaces g/p. Russian Mathematical Surveys, 11, 1973.
Davies, A., Veličković, P., Buesing, L., Blackwell, S., Zheng, D., Tomašev, N., Tanburn, R., Battaglia, P., Blun- dell, C., Juhász, A., et al. Advancing mathematics by
10

---

Machine Learning Meets Algebraic Combinatoric
guiding human intuition with ai. Nature, 600(7887):70- 74, 2021.
Ikenmeyer, C., Pak, I., and Panova, G. Positivity of the sym- metric group characters is as hard as the polynomial time hierarchy. International Mathematics Research Notices, 2024(10):8442-844) 2024.
Demazure, M. Désingularisation des variétés de schubert généralisées. In Annales scientifiques de l'École Normale Supérieure, volume 7, pp. 53 23-88,175 1974.
Kazalicki, M. and Vlah, D. Ranks of elliptic curves and deep neural networks. Research in Number Theory, 9(3): 53, 2023.
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, The role of permutation invariance in linear mode connectiv- ity of neural networks. In International Conference Learning Representations.
Kazhdan, D. and Lusztig, G. Representations of coxeter groups and hecke algebras. Inventiones mathematicae, 53(2):165-184, 1979.
Fajtlowicz, S. On conjectures of graffiti. Discrete Mathe- maticy, 72(1-3):113-1188 1988.
Keriven, N. and Peyré, G. Universal invariant and equivari- ant graph neural networks. Advances in Neural Informa- tion Processing Systems, 32, 2019.
Felsner, S.
On the number of arrangements of pseudo- lines. volume 18, pp. 257-267. 1997. doi: 10.1007/ PL00009318. ACM Symposium on Computational Ge- ometry (Philadelphia, PA, 1996).
Lascoux, and Schüitzenberger, M.-P. Structure de hopf de l'anneau de cohomologie de I l'anneau de grothendieck d'une variété de drapeaux. CR Acad. Sci. Paris Sér. I Math, 295(1 (61(1))669-6333 1982.
Gaetz, C. and Gao, Y. On the minimal power of q in Kazhdan-Lusztig polynomial. Advances in Mathematics, 457:109941, 2024.
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. Set transformer: A framework for attention-based permutationninvarinn neural networks. In International conference on machine learning, pp. 3744-3753. PMLR, 2019.
Gauthier, T. and Urban, J. Learning program synthesis for integer sequences from scratch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7670-7677, 2023.
Lee, N., Sreenivasan, Kee J. D., Lee, K., and Papail- iopoulos, D. Teaching arithmetic to small transformers. In The Twelfth International Conference on Learning Representations.
Godfrey, C., Brown, D., Emerson, T., and Kvinge, H. On the symmetries of deep learning models and their internal rep- resentations, Advances in Neural Information Processing Systems, 15:11893311999 2022.
Liu, Z., Gan, E., and Tegmark, M. Seeing believing: Brain-inspired modular training for mechanistic inter- pretability. Entropy, 26(1):41, 2023.
Golden, J., Goncharov, B.. Spradlin, M., Vergu, C., and Volovich, A. Motivic amplitudes and cluster coordinates. Journal of High Energy Physics, 2014(1):1-56 2014.
Luo, D., Cheng, W., Xu, D., Yu, W., Zong, B., Chen, H., and Zhang, X. Parameterized explainer for graph neural network. Advances in neural information processing systems, 33: 13:1962--16631 2020.
Hahn, M. and Rofin, M. Why are sensitive functions hard for transformers? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14973-15008, 2024.
Mehrabian, A., Anand, A., Kim, H., Sonnerat, N., Balog, M., Comanici, G., Berariu, T., Lee, A., Ruoss, A., Bu- lanova, A., Toyama. D., Blackwell, S., Paredes, B. R., Veličković, P., Orseau, L.,, Lee, J., Naredla, A. M., Precup, D., and Wagner, A. Finding increasingly large extremal graphs with alphazero and tabu search. In Proceedings of the Third International Joint Conference on Artifi- cial Intelligence, IJCAI '24, 2024. ISBN 978-1-956792. 04-1. doi: 10.249633/jcci. / ///oi. org/ 10..001000 243.244433111 ijcai.iia.... 2024/772.
He, J., Jenne, H., Chau, H., Brown, D.. Raugas, M., Bil- ley, S., and Kvinge, H. Machines and mathematical mutations: Using GNNs to characterize quiver mutation classes. arXiv preprint arXiv:2411.07444 2024.
Henrich, T. Mutation classes of diagrams via infinite graphs. Mathematische Nachrichten, 284(17-18):1184-2200
2011. doi: URL
URL https:
com/ 1002 h0t.:/mmmmaaaam0000222222 200910224..
Moura, L. d. and Ullrich, S. The lean 4 theorem prover and programming language. In Automated Deduction CADE 28: 28th International Conference on Automated
Hepler, C. T. On the complexity of computing characters of finite groups. 1996.

---

Machine Learning Meets Algebraic Combinatoric
Deduction, Virtual Event, July 12-15, 2021, Proceed- ings, pp. 225-635, Berlin, Heidelberg, 2021. Springer- doi: 10.1007/
Schensted, C. Longest increasing and decreasing subse- quences. Canadian Journal of mathematics, 13:179-191, 1961.
Verlag. ISBN 978-3-030-79875-8. 3-030-77777----77 URL https:/ httpps://dooiooooee /oo. 10. 1 007 97000/9978830 00030-097070----20
Schiffler, R. Perfect matching problems in cluster algebras and number theory. arXiv preprint 2xXiv:3302.0218. 2023.
Musiker, G. and Stump, C. A compendium on the cluster algebra and quiver package in sage, 2011. URL https: //arxiv. orgg abs /11//.1111.. 4844 844
Scott, J. S. Grassmannians and cluster alge- bras. Proceedings of the London Mathemati- cal Society, 92(2)345-380, 2006. doi: https:
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Stein- hardt, J. Progress measures for grokking via mechanistic interpretability. The Eleventh International Conference Learning Representations, 2023.
//lcc.ogg/..11222002611151513111 URL https:
/ wow.onathssc...nllneee/eeeeeeeee http////aaaas/..111///..22/
Song, P., Yang, K., and Anandkumar, A. Towards large language models as copilots for theorem proving in lean. arXiv preprint arXiv:2404. 12534, 2024.
Novikov, A., Vu, N., Eisenberger, M., Dupont, E., Huang, P.S., Zsolt Wagner, A., Shirobokov, S., Kozlovskii, B., Ruiz, F. J. R., Mehrabian, A., Kumar, M. P., See, A., Chaudhuri, S., Holland, G., Davies, A., Nowozin, S., Kohli, P., and Balog, M. Alphaevolve: A coding agent for scientific and algorithmic discovery. Technical report, Google DeepMind, 2025.
Stanley, R. P. On the number of reduced decompositions of elements of Coxeter groups. European Combin., 5(4):359 372, 1984. ISSN 0195-6698. doi: 10.1016/ $0195-6698884)8003399
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z.,, Lin, Z., Desmaison, A., Antiga, L., and Lerer,
Stanley, R. Enumerative combinatorics volume 1 second edition. Cambridge studies advanced mathematics, 201 la.
A. Automatic differentiation PyTorch. 2017.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V.,
Stanley, R. P. Enumerative combinatorics volume 2 second edition. Cambridge studies in advanced mathematics, 2011b.
Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas. J., Passos, A., Cour- napeau, D., Brucher, M., Perrot, M. and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Stein, W. et al. Sage Mathematics Software (Ver- sion 10.1). The Sage Development Team, 2024.
http:/ / http://www//eaaaee www.wwwwggm.th.ccmmm org.
Raayoni, G., Gottlieb, S., Manor, Y., Pisha, G., Harris, Y., Mendlovic, U., Haviv, D., Hadad, Y., and Kaminer, Generating conjectures on fundamental constants with the ramanujan machine. Nature, 590(7844):67-73, 2021.
Vatne, D. F. The mutation class of D,, quivers. Communi- cations Algebra, 337 2010.
Veličković, P., Badia, A. P., Budden, D.,, Pascanu, R., Ban- ino, A., Dashevskiy, M., Hadsell, R., and Blundell, C. The clrs algorithmic reasoning benchmark. In Inter- national Conference on Machine Learning, pp. 22084- 22102. PMLR, 2022.
Robinson, G. d. B. On the representations of the symmetric group. American Journal of Mathematics, pp. 745-760, 1938.
Romera-Paredee B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J., Ellenberg, J. S., Wang, P., Fawzi, O., et al. Mathematical discoveries from program search with large language models. Nature, 625 (7995):468-475. 2024.
Veličković, P. Neural algorithmic reasoning. The Gra- dient, 2023. URL https::/ttmi https::/ttherradientttt neural-aie
Wagner, A. Z. Constructions combinatorics via neural networks. arXiv preprint arXiv:2104.14516, 2021.
Sagan, B. E. The symmetric group: representations, com- functions, volume
binatorial algorithms, and symmetric funtittiooonnn 203. Springer Science Business Media, 2013.
Warrington, G.
Klmu & KLC versions 1.0. w3. uvm. edu/research/
tic/klc.htmd Accessed: 2024-06-05.
Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv: 904.0/557, 2019.
Williams, L. Cluster algebras: an introduction. Bulletin of the American Mathematical Society, 51(1):1-26, 2014.
12

---

Machine Learning Meets Algebraic Combinatoric
Yang, K., Swope, A., Gu, A., Chalamala, R., Song, P. Yu, S., Godil, S., Prenger, R. J., and Anandkumar, A. Leandojo: Theorem proving with retrieval-augmented language models. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, J. Deep sets. Ad- vances in neural information processing systems, 30, 2017.
Zhong, Z., Liu, Z.., Tegmark, M., and Andreas, The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Machine Learning Meets Algebraic Combinatoric
Dataset
Linear regression
MLP
Transformer
Guess training label mean
S, characters
18
1.5920
10400
2.7447 10 8.80015 x 10
2.4913 x 10th 1.4350 x 10?
1.5920 x 10
20
4.2007
10t2
4.2254 x 10 4 5.123 10""
5.3897 10 3.6464 x 11
4.2007 10
22
8.0395 10
1.1192 10 + 4.9321
10
1.3797 10 6.2799 1
8.0395 10
RSK
0.21
0.43 + 0.05
1,51 + 0.02
0.21
= 9
0.21
0.96 + 0.07
3.85 = 0.09
0.21
Table Off-the-shelf model MSE regression datasets. Results averaged three random weight initializations with 95% confidence intervals after a hyperparameter search outlined in Appendix B.10.
Dataset/cnefficient
MLP
Transformer
Guessing largest class
1
99.8% 1. 0.2%
99.9% 1 0.1%
73.7%
99. 99.5% + 0.4%
99.2% + 1.0%
97.0%
9²
99.9% + 0.1%
100.0% = 0.0%
99.9%
1
99.9% + 0.0%
100.0% 0.0%
80.9%
9
99.9% 1: 0.0%
99.9 1: 0.0
95.8%
99.9% + 0.0%
99.9 + 0.0
99.5%
9-
99.9% 1: 0.0%
99.9 1 0.0
99.9%
Table 4. Baseline model classification accuracy for predicting KL polynomial coefficients for 5, 6. Results are averaged three random weight initializations with 95% confidence intervals. The MLPss had layer dimension 256 and depth and were trained with a learning rate of 0.0005. The transformers had dimension 256, depth 6, 8 heads, and were trained with learning rate of 0.0005.
A. Dependence on n
Many problems in algebraic combinatorics have natural dependence parameter n (e.g., permutations are parameterized by the number of elements that they permute). We have chosen to structure datasets in the ACD Repo to reflect this, with the majority of datasets belonging to a series {Dn We provide a few values of n and, in many cases, the code to generate others. Generally, there are two properties that change as n - co. First, the size of Dn grows as n grows. The rate of growth depends on the specific problem, with many D,, growing exponentially (such as those datasets that depend on the number of permutations of n)). On the other hand, the problems can also become harder in various ways as n grows.
Experimentally we have found that larger values of tend to lead to better model performance regardless of this potential increase in difficulty. For example, we ran 5 2-layer MLP models for 500 epochs on the lattice path datasets corresponding
Dataset/coefficient
MLP
Transformer
Guessing largest class
7
1
99.7% 1: 0.1%
99.9% 1 0.4%
73.7%
0
93.9% + 3.7%
92.7% - 7.6%
97. 97.0%
9
50.0% + 0.0%
100.0%
0.0%
99.9%
1
99.9% 1: 0.0%
100.0% 0.0%
80.9%
99.0% + 1.5%
98.0 - 3.7
95.8%
4²
97.4% 4: 5.2%
86.2 + 47.7
99.3%
9²
87.9% d: 4.5%
88.3 17.1
99.9%
Table 5. Baseline model classification macro F1-scores (to account for class imbalance) for predicting KI. polynomial coefficients for
n 5, 6. Results are averaged over three random weight initializations with 95% confidence intervals. The ML.s had layer dimension 256 and depth 4 and were trained with learning rate of 0.0005. The transformers had dimension 256, depth 6, 8 heads, and were trained with a learning rate of 0.0005.
14

---

Machine Learning Meets Algebraic Combinatoric
Dataset
Claude 3.5 Sonnet
GPT-4o Mini
GPT-4o
In contexl learning
76.4%
64.7%
58.8%
59.5%
53.5%
57.0%
58.5%
51.5%
37.0%
Program synthesis
3
94.1%
82.4%
100.0%
n 1
65.0%
100.0%
100.0%
n 5
99.8%
64.2%
64.7%
Table 6. Success of Claude 3.5 Sonnet, GPT-4o Mini, and GPT-4o solving the Schubert polynomial structure constant task via either in-context learning or program synthesis. Hyperparameters for these experiments can found in Section 1. As described Section models that achieved 100% had effectively learned the strategy by which subsampled zero-valued structure constants. This has now been fixed.
Lattice Path Dataset
quiver classification
E, D, quiver classification
1.00
accuracy
0.95
0.55
0.90
0.90
0.0
0.85
0.25
0:00
0.80
0.21
eCourncy -
0.75
0.6
13 scone 10
0.70
Sccuracy: 1-11
0.70
score. -1
0:65
eccuracy: 12
0.85
score h-12 12
0.60
0.60
Lattice path endpoint
number vertices
Figure 2. (Left) Performance on the Lattice Path Dataset as a function the lattice path endpoint (larger endpoint values means longer and 1, the training set size increases but the problem may also grow harder. (Center) Performance the
paths). As grows in type E' versus type D quiver classification task as a function of the depth, which must be specified for type E' quivers on n = 10, 11, 12 vertices, and (Right) the number of vertices
to grids of size 6 x 5,... . 13 x 12. We see in Figure 2 (left) that with an interesting exception of moving from 7 x 6 to 8 x 7, performance across a range of dimensions improves as n grows. There are exceptions, however. We looked at sampling from greater depth when exploring the quiver mutation equivalence dataset (Section 4.8). This effectively means allowing a greater number of mutations to be applied the initial quiver when generating the dataset. As shown in Figure (center), we find that performance somewhat degrades even though the size of the datasets increase. We suspect that exploration of the complexity of these problems (where it is known) might be avenue for shedding light on this phenomenon.
B. Dataset details
All datasets are stored as . txx files with one datum instance per line. In this section we will describe each file and explain how to interpret it. Functions capable of loading and parsing each file are available on the GitHub page.
B.I. Computing Characters of Irreducible Representations of the Symmetric Group
Since the conjugacy classes of the symmetric group S, are indexed by integer partitions of n, characters are constant on conjugacy classes, and the irreducible representations of S,, are also indexed by integer partitions of n, the task is to use pair of integer partitions of n to predict the character of the corresponding irreducible representation of the symmetric group.
Within each file. two integer partitions are provided followed by integer corresponding the character. For instance, the line
[3, 1, 11, 22,2,
3,1.1 says that the character
-2. X2,2,1
15

---

Machine Learning Meets Algebraic Combinatoric
In all cases the characters are heavily concentrated around with very long tails. This likely contributes the difficulty of the task and could be overcome with some simple pre- and post-processing. We have not chosen do this in baselines.
Characters of 18
There are 118, 580 training examples and 29, 645 test examples. The maximum character value 16, 336, 320. The minimum character value is 223, 040.
00000
20000
Insouctie cherrstter Stp
Figure 3. Histogram of Si8 characters within the interval [-500, 500]
Characters of S20
There are 298, 661 training examples and 74, 819 test examples. The maximum character value is 249, 420, 600. The minimum character value is -17, 59,,592,90 960.
30013
100000
50233
Figure 4. Histogram of S20 characters within the interval [-500, 500]
Characters of S22
There are 763, 109 training examples and 190, 726 test examples. The maximum character value is 5, 462, 865, 408. The minimum character value is 2279, 734, 796.
500000
400000
100083
-00
Ineducilee Charecteer VETUEE Str
Figure 5. Histogram of S22 characters within the interval [-500, 500]
16

---

Machine Learning Meets Algebraic Combinatoric
0
2
3
4
Training set
6, 716
508
78
9
Testing set
1, 672
136
3
Table 7. Statistics for the mHeight dataset for
8.
0
3
5
Training set
49, 092
3, 161
524
77
9
Testing set
12, 317
759
118
19
3
0
Table Statistics for the mlleight dataset for
Sage (Stein et al., 2024) was used to calculate character values. Data generation scripts can be found on the GitHub page.
B.2. The mHeight Funetion of a Permutation
This dataset contains permutations labeled by their mHeight. An example calculation of mHeight is shown in Figure Permutations are written in I-line notation followed by the symbol ";" and then the value the mHeight function on the permutation.
(6, 8, 7, 5, 4, 9, 3, 0, 1, 2);
can be read as saying that the permutation 6 7 4 3 0 has mHeight 1.
We provide datasets for permutations of size -8,9, 10. The files called:
mHeighhtB_ttriin txt
mieight____est.tte
mHeighht___traim txt
mieight_9_test..c
#teiight_0_ttratie
mHeight_ tiieetght_1_ttsttttttt txt
The dataset was generated using Python script which can be found on the GitHub page. Dataset statistics can found in Table 7-Table 9.
Permutation
= 345
patterns
3 1 2
45 2 3512
mHeight of a
max(3 2,4 2,3 1) =2
Figure 6. An example of the calculation of mlHeight permutation.
5
6
Training set
352, 494
17,
Testing set
503
Table Statistics for the mHeight datasct for n 10.
17

---

Machine Learning Meets Algebraic Combinatoric
Cluster variable
Not cluster variable
Total
Training set
74, 329
74,329
148, 658
Testing
18,
18, 582
37, 164
Table 10. Statistics of the Grassmannian cluster algebra dataset.
B.3. Grassmannian Cluster Algebras and Semistandard Young Tableaux
This dataset relates to a cluster algebra associated with the Grassmann manifold Gr(k, n). Each cluster variable is indexed by a rectangular SSYT with & rows with entries drawn from (1.....n) The rank these rectangular SSYT (and the rank of the associated cluster variable) given by their number of columns. Following (Cheung et al., 2023), this dataset we focus on Gr 12) and hence look rectangular SSYT with rows filled with entries drawn from {l...., 12). We further restrict rank SSYT. This leaves with collection of 4 arrays whose entries increase weakly across rows and strictly down columns.
To give two examples, the SSYT in Figure 7 are valid (left) and invalid (right). Note that both are genuine SSYT of shape 3 x 4 with entries from {l...., ,....,,12. 12].
1
2
4
7
1
5
7
5
6
6
11
4
5
7
9
9
9
12
12
10
11
Figure 7, An example a valid (left) and invalid (right) tableau from the Grassmannian cluster algebra datasct.
The dataset consists of collection of rectangular SSYT each with a label indicating whether it indexes a cluster variable or not. Those that do not index cluster variable are labeled with a '0' and those that do are labeled with a '1'. The valid examples are drawn from (Cheung et al., 2023) and can be obtained from https: //ggttthii com/ erhirrt/ Gras smanni aMsM We generated our own negative examples because we found that the model learned some spurious correlations result of the sampling strategy used in (Cheung al., 2023). To sample random rectangular SSYT, took advantage of the random.e lement method in the 'Tableaux' class in Sage.
The datasets, which can be found on the GitHub page, are contained in files named:
3344_1__invalld__raii..ct
3_44_R__invalid_teett
3 1_4_12__vaid__tess
3_4_12_valid__rriin
In the files we use braces and demarcate rows of the diagram, that
[[1, 2, 71, [5, 6, 6, 11], [9, 9, 12, 12]]
corresponds to the tableau in Figure left. Dataset statistics be found in Table 10.
B.4. The Coefficients of Kachdan-Lusztig Polynomials
Lazzhan-Lusztig polynomials are integer polynomials in a variable q which are parameterized by two permutations. The tasks associated with this dataset are to predict the different coefficients of the polynomial (organized by monomial degree). Thus the input is two permutations v, and the output is sequence of integers which are the coefficients of the polynomial. For instance, if
2 1113544 6 9 and w 23 0 5 9 6 8 4 then
P Po, = 4q 12 + 134² + 6g +
18

---

Machine Learning Meets Algebraic Combinatoric
Coefficient
Train
Test
496
2, 123
024
707
Table 11. Statistics for the constant term in the
5 KL polynomial dataset.
Coefficient
Train
Test
0
11, 219
2, 793
267
77
2
34
10
Table 12, Statistics for the coefficient in the 5 Kl. polynomial dataset.
and this written the line
0213546978 2305967814 1, 13, 12,
Datasets were generated using C code from Greg Warrington's website (Warrington), computing (e. for all pairs of permutations of size 5, 6,
The files we provide are:
kl polyyoomaals___trrinn.
kl pol.yromiall___.eettttr
kl -opllyyommalll___rraiinn
kl http://ommiias__ttsttt_ss
Lo-pllyyoomlals____tttttt
k1 ai-pollyoomlala__tttrii...t
kl ppppyoooiall____eetttte..
Statistics can be found Table 11-Table 22.
B.5. The Robinson Sobinson-Schenstee--kniie Correspondence
The Robinson-Schenstted Knuth correspondence bijection between permutations and pairs of standard Young tableaux. For our initial baselines we use Young tableau pairs input and the corresponding permutation output, but the dataset could be used in the opposite direction.
The permutations are stored in files starting with i nnpu permutati on and the pair of tableaux are stored in files labeled by outputt.ableaa tableu.. Permutations are stored using their inversion vectors (a binary sequence). Tableau rows are separated by 'I' and '." For instance,
[[[1, 3, 4], [2, 71, (5), [6]], [[1, 2, 6], [3, 41, [5], [7]]]
corresponds to the pair of Young tableau in Figure 8. our baselines, this task is formulated as regression with the aim of regressing the sequence of (01 entries in the inversion vector. We found that the inversion vector was easiest to work with relative to other representations of permutations in this setting.
Coefficient
Train
Test
11,
2,876
6
Table 13. Statistics for the coefficient on q² the n 5 KL polynomial datasct.
19

---

Machine Learning Meets Algebraic Combinatoric
Coefficient
Train
Test
336, 071
83, 922
78, 649
19, 758
Table 14. Statistics for the constant term in the
KL. polynomial dataset.
Coefficient
Train
Test
397, 386
99, 354
13, 253
3, 311
2
3, 483
883
3
535
117
63
15
Table 15, Statistics for the coefficient on in the 6 KL polynomial dataset.
Coefficient
Train
Test
0
412, 707
103, 177
1, 705
441
2
242
46
Table 16. Statistics for the coefficient on o² in the 6 KL polynomial dataset.
Coefficient
Train
Test
414, 688
103, 670
32
10
Table 17. Statistics for the coefficient on q² the 6 KL polynomial dataset.
Coefficient
Train
Test
17, 479, 910
4, 370, 771
881,37777
709,
Table 18. Statistics for the constant term in the 7 KI. polynomial dataset.
Coefficient
Train
Test
19, 291, 150
4, 822,, 214
660, 600
165, 768
266,
66,
3
80, 173
19,963
4
18, 831
4, 762
5
3, 221
819
6
711
201
Table 19, Statistics for the coefficient g in the KL polynomial dataset.
20

---

Machine Learning Meets Algebraic Combinatoric
Coefficient
Train
Test
20, 072, 738
5, 017, 962
170, 412
42, 748
46,
11, 568
16, 227
4,
4
7, 621
1, 9.9
5
4, 023
1, 065
6
287
349
7
1, 153
287
8
785
183
9
350
139
37
12
121
47
13
42
22
14
4
1
Table 20. Statistics the coefficient on q² the n 7 KI. polynomial dataset.
Coefficient
Train
Test
20, 291, 535
507, 2831
22, 094
5, 498
779
1,213
660
442
390
146
195
61
206
50
115
37
34
14
24
11
18
14
15
4
1
Table 21. Statistics for the coefficient on q² in the = 7 KI, polynomial dataset.
Coefficient
Train
Test
0
17, 910 910
4, 370, 771
2, 811,370
549
Table 22. Statistics the coefficient on q² the 7 KI. polynomial dataset.
21

---

Machine Learning Meets Algebraic Combinatoric
4
1
2
3
4
5
5
7
Figure 8. The Young tableau pair that are obtained by applying the Sihinsoo-Sccenstee---e Knuth algorithm to the permutation 6725341 1.
We store files for n 8,9, 10. These are named:
Amput_permuttiion____aaaaaa
Ipput_perrutations___ssess.cc
ipput_perruterionn___..taa....
Impot_perrruaatloos___..s.ee..
output_cableeuu__trraii..co
Sttput_tablleau___testt..
ottput_cableesu__.ttrii.....
ottput_fabbleaa___ttstt..
output__aabless_11_trai...co
outputt_tableau__1_tter.cco
Sage (Stein et al., 2024) was used to generate tableau pairs corresponding to each permutation. The script available on the GitHub page.
B.6. Schubert Polynomial Structure Constants
This task involves predicting the structure constants of Schubert polynomials. These polynomials are each indexed by permutations so structure constants are indexed by a triple of permutations. Hence, the input to the model triple of permutations v, w, u and the output is integer. For instance, since we have the relationship
S1235481235 S123615 + S12153
one data instance
[1, [1,2,3,5, 41, 2,,,,,,,,,,,,,,,,, 6,
We partition the datasets so that the dataset associated with value n has structure constants for pairs of S, with v E S,, Note that there some repetition since S,-1 is a subset of S,. We store files for n = 4,5, These are named:
schubert_st
schubert_st cct.epeeestacccrr..effffiiieeete
22

---

Machine Learning Meets Algebraic Combinatoric
Training set
851
833
Testing
Table 23. Statistics for the Schubert polynomial structure constants dataset for 4.
Training 831
42, 619
170
Testing set
10, 681
10, 680
41
Table 24, Statistics for the Schubert polynomial structure constants dataset for
Sage (Stein et al., 2024) was used to generate and multiply Schubert polynomials for each pair of permutations a and / in Sn. The basis expansion Sa * Sg was obtained from this and each nonzero term in this expansion was used as an instance. For any n, most structure constants will be zero. generate balanced dataset, we computed S., * Sg for all elements in we applied random number of transpositions (where the number of transpositions was sampled from a geometric distribution) to y to get 7', checked that e = and added this to the dataset. Therefore, dataset contains all non-zero structure constants but only a fraction of zero structure constants.
S, X Sn and for each c², *
Dataset statistics can be found in Table 23- 23-Table 25.
B.7. Partial Orders on Lattice Paths
This dataset contains pairs of lattice paths starting at (0, 0) and ending (n, 1) that are only allowed take unit steps either north east, and must stay below the line y n- v. They are thus encoded by sequence of 1's (for steps east) and 0's (for steps north) of length (n 1) + 2n + 1. Each pair of lattice paths is covering pair in exactly one of the two partial orders, the Lagrange order or the matching order (pairs that are covers in both are few and were removed). The task is to predict which partial order a covering pair belongs to.
Each line a file lattice paths are separated by ':
concatenation of two 0-1 sequences (one for each path) for a length 4n row of 0's and l's. The
For an x 2 grid, the sequence:
1, 1,1,1,0,011,1,10
corresponds to the two lattice paths in Figure 9, The first is in red and second is in blue, with segments traversed by both paths colored red.
We store files for n = 10, 11, 12, 13. These are named:
lagrange__over__teese
2
3
Training set
4, 20200,00 040
4. 0939,, 033
109, 217
2, 262
9
Tosting set
052, 062
1. 02022, 898
27, 110
568
Table 25. Statistics for the Schubert polynomial structure constants dataset for 6.
23

---

Machine Learning Meets Algebraic Combinatoric
Figure 9, An example of two lattice paths from (0,0) (3, 2). These correspond relation.
Lagrange order
Matching order
Total
Training
7, 5589
11, 433
Testing
895
2,
Table 26. Statistics for the lattice paths dataset for paths from (0, 0) to (10,9).
hagraagg_ccoerrs_tss___11110._0
bggrangg_ccoerr__ttt___1111111
lagrange_covrrr_tts_______._____
lagrange_covrr3_ttaaa11........
lagrange_coees__ttaaa____1,11_
hggrangg__coerra_ttraan________
legrangg_ccoerrs_rrra__________,
aatchllg_covers_teet___9.99999
macching_oovera.tett__11110....0
matcchnng_coerrs_tet__11111..1..
matchiingg.coerrs_tsss__11____..
matechingg.coerrs_trraa__.......
natchinng_cooers_ttaia____11
matchilgg_covrrs_taaaa__________
matchiingg_covers_trra____11111
The first word ('Lagrange' or 'matching') gives the label, the third word says whether is train or test, and the final two numbers give n and n -
Sage (Stein et al., 2024) was used to compute the covering pairs in the Lagrange and matching ordering; the script used generate the data is available on the GitHub page. We removed instances that were covering pairs in both the Lagrange partial order and the matching partial order (this was 21, 40, and 79 instances for 10, 11, and 12 respectively). The dataset statistics can be found in Table 26-Table 29.
Lagrange order
Matching order
Total
Training
427
424
39, 851
Testing set
6, 601
3, 355
9, 956
Table Statistics lattice paths dataset for paths from 0) to (11, 10).
24

---

Machine Learning Meets Algebraic Combinatoric
Lagrange order
Matching order
Total
Training sel
93, 218
46, 976
140, 194
Testing set
23, 324
14,
35, 073
Table 28. Statistics for the lattice paths dataset for paths from (0, to (12, 11).
Lagrange order
Matching order
Total
Training set
331, 005
166, 304
197, 369
Testing set
82, 789
41, 580
124, 369
Table 29, Statistics for lattice paths dataset for paths from 0) to (13, 12).
B.8. Mutation Equivalence of Quivers
The task associated with this dataset is matching a quiver to one of several possible mutation equivalence classes. Thus, the input quiver with nodes encoded by classes: An1, BB BD E1, D1, D E1, En The files are organized by train and test for each of these classes. All mutation classes were generated using Sage (Stein al., 2024), the code is available the GitHub page. For the quiver mutation classes that are not mutation finite, the datasets contain quivers generated up certain depth, which is the distance from the original quiver, measured by number of mutations. The depth is specified in the filename and was chosen to achieve as close to a balanced dataset as possible.
adjacency matrix and the label is one of 7 different equivalence
The file names are:
A_1_Jmmatricesttest.coo
A A_11_maatrices_trahn.cos
BB [B_1__Gechho0_aaatticesstaaaaa.e
BD B0__1_.ettts_mmarriccss.ee..o.
BE_ BEJ_L_eecch__mmaattccee_ee....
BE B__i_deettt@_mmatiicce.taaaaa
D_ D_11_batrries_test.ccr
_.11_baatrices_taain.co
DE D___1_.eethh_mmatrttceee.taaaaa
E htt_depths__bnarrcce_ttteee
Within file, each row is flattened adjacency matrix encoded in row major order. The statistics of the dataset can be found Table 30.
25

---

Machine Learning Meets Algebraic Combinatoric
A11
B11
BD
BE1
D11
D
F11
Total
Training sel
11, 940
27, 410
23, 651
22, 615
25, 653
23, 528
28, 998
163, 795
Testing set
984
6,
5,912
5, 653
413
5, 6,881
7, 249
409,
Table 30. Statistics of the quiver mutation equivalence dataset.
Weaving pattern
Non-wcaving pattern
Training
634
116
Testing set
275
467
Table 31. Statistics of the weaving pattern dataset for 6.
B.9. Weaving Patterns
Weaving patterns size n x n are a special type of matrix containing entries in (1,2....,m). They correspond to representations of the longest word permutation of n elements (the permutation that sends - n, n 1, etc.). This task involves trying identify weaving patterns among matrices that look like weaving patterns but are not.
Each matrix is stored on single line in now-major format. For instance,
(0, 1, 3, 3, 2, 3, 4, 2, 3, 2, 1, 4, 3, 2))
We provide files for 6, The files
reavying_patterrs___Ltraa.....
weavinggraateerrs___taaaal
weavingg_atterrs___LLeet.
weaving_patternss 7_test.txt
Positive examples were generated by program written in Java script. Negative examples were generated by loading positive examples and Python and perturbing them. Code for both of these can be found on the GitHub page.
Dataset statistics can be found Table 31 and Table 32.
B.10. Small Model Hyperparameters
For our baselines, we train encoder-only transformer models, standard feedforward multi-layer perceptron (MLP) models with ReLU non-lincarities, and logistic regression on the classification tasks and the same architectures of transformers and MLPs along with linear regression for the regression tasks.
To optimize MLP models we performed a simple grid search across
learning rates: 0.001, 0.0005, 0.0001,
depth: 1, 2, 3, and 4,
uniform hidden dimension: 32, 64, 128, and 256.
To optimize the transformer models we performed a similar grid search but with hyperparameters
learning rates: 0.001, 0.0005, and 0.0001,
Weaving pattern
Non-weaving pattern
Training
17, 388
06,
Testing set
7,310
21,
Table Statisties of weaving pattern dataset for n 7.
26

---

Machine Learning Meets Algebraic Combinatoric
Dataset
Learning rate
Depth
Hidden dimension
Lattice paths
256
256
12
0.001
256
Weaving patterns
0.001
2
256
=
0.001
2
256
Cluster algebra quivers
0.001
4
256
Grassmanian cluster algebras
256
Schabert polynomials
0.0005
4
64
0.001
4
256
0.001
3
256
mHeight
0.0005
2
128
128
256
S. chameters
18
0.001
4
128
20
0.0001
4
256
2 m 22
0.001
256
RSK
256
256
Table 33. Optimized hyperparameters for the MLP baselines found Table and Appendix B.
model dimensionality: 20, 40, and 80,
depths: and 4,
number of heads: 4, 6, and 8.
MLPs and transformers all trained for 60 epochs and the best test recorded. The best MLP hyperparameters for each dataset can be found in Table 33. The best transformer hyperparameters for each dataset can be found in Table 34,
All optimization was performed in Pytorch (Paszke et al., 2017) with the Adam optimizer (with Pytorch default settings beyond the learning rate) on one Nvidia A100. After finding an optimal hyperparameter setting, we trained 3 models, averaged their performance, and generated 95% confidence intervals to measure the variability in performance with these hyperparameters.
Linear and logistic regression was performed with sklearn (Pedregosa al., 2011) using default hyperparameters.
B.11. LLM Evaluation Procedure
We evaluated Claude 3.5 Sonnet. GPT-4o-mini, GPT-4o, and ol-mini on the mHeight and Schubert polynomials datasets using simple in-context learning setup and simple program synthesis setup. When using in-context learning, we tried using 0, 10, 50, and 100 examples and also tested using chain-of-thougle prompting. Finally, we explored providing the model with some background on the problem. The best setting depended on the model and task. An example template for the in-context learning prompt when chain-of-though is used is:
"You are tasked with solving classification problem. Here is high-level information about the dataset: An (iddaaaset_iifaaaaaa \nAn" f" sar___ho__atrr] + "Before answering with your Python code, reason in a step-by-step manner as to get the right answer, An
27

---

Machine Learning Meets Algebraic Combinatoric
Dataset
Learning rate
Depth
Hidden dimension
Heads
Lattice paths
n
0.001
12
0.0005
Weaving patterns
v 6
0.0001
80
T 7
0.0001
80
Cluster algebra quivers
0.0005
80
Grassmanian cluster algebras
Schuhert polynomials
0.0001
80
v. 5
0.0005
40
n 6 lligigh
0.0005
80
n 8
0.0001
n 9
0.001
8,, charucters
7L 18
0.001
6
80
n. 20 7 22 RSK
0.001
80
4
40
8
n Table 34. Optimized hyperparameters for the transformer baselines found in Table 1 and Appendix B.
0.001
28

---

Machine Learning Meets Algebraic Combinatoric
where 'dataset_info info'' is description of the dataset and task and 'few few_shot_str" are the few-shot examples. When chain-of- thought reasoning used the last component is changed to
"Do not provide any additiona toaal reasoning in or explanation.
Just provide your answer at the end on its own line in the form 'ANSWER:
SANSWER' (without quotes) where SANSWER is the answer to the question."
In the code synthesis version of the experiments, models were asked to write Python program that solves the task using only Sage (Stein et al., 2024), Numpy, and SymPy. No model ended up using either Sage SymPy. The same prompt was used to generate 100 examples. The best program was chosen by evaluating each on the test set.
Your job is write a Python function that solves the classification problem. You will be given some examples of a classification problem from the (dataset)' dataset.
Write a function 'predict' that takes an input in a Python list and returns an integer as the classification result.
Here is information about the dataset:
(dataset_info)
Avoid using machine learning or model calls; rather, embed the logic in Python code.
Rather than use shallow pattern matching or using simple patterns, try to analyze the underlying combinatorial logic of the examples. Note that the datagenerating process for this dataset is a combinatorial algorithm. You may want use numpy and sympy for math operations sage for cominatorics, however this is opticnal. If you do use them, *make sure to import them within your function*.
Below are a few examples from the training set: (training_examples)
(instructions)
Your final answer should be valid Python code enclosed in triple backticks. This program will be evaluated the test set.
All our experiments with LLMs used AI Inspect (Al Safety Institute).
C. Datasheets
We provide datasheet information that holds for (almost) all datasets in the collection here and provide answers that vary across each dataset in their relevant section.
Who funded the creation of the datasets? This research was supported by the Mathematics for Artificial Reasoning in Science (MARS) initiative at Pacific Northwest National Laboratory. It was conducted under the Laboratory Directed Research and Development (LDRD) Program at Pacific Northwest National Laboratory (PNNL), multiprogram National Laboratory operated by Battelle Memorial Institute for the U.S. Department of Energy under Contract DE-AC05-76RL01830
Are there recommended data splits (e.g., training, developpeentvvaiiaationn testing)? All datasets are provided in preset splits.
Are there any errors, sources of noise, or redundancies in the dataset? Not that the creators are aware of.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? It self-contained.
29

---

Machine Learning Meets Algebraic Combinatoric
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege by doctor- patient confidentiality, data that includes the content of individuals' non-public communications)? No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? No.
Over what timeframe was the data collected? All datasets were generated between June 2024 and December 2024.
Were any ethical review processes conducted (e.g., by an institutional review board)? N/A
Was the "raw" data saved in addition to the preproossed/dleneed/bbeeeld data (e.g,, to support unanticipated future uses)? No, but in most cases we supply the code to re-generate the raw data (see ht ht com/ panl/ML4AllCommb)
Is the software that was used to preprocess/ceenn/lbel the data available? Yes, https://github.com
paal/ML4A100 4AlgComb.
Is there repository that links to any or all papers systems that use the dataset? Yes, htttt://gtttthhb. com/pnn1 /MM./MA44Allmmaa
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? All datasets will be compressed to a single zip file and stored on Google Drive. It does not currently have a DOI.
C.1. Symmetric group characters datasheet
For what purpose the dataset created? To study whether machine learning models can learn to predict the characters of irreducible representations of the symmetric group.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? The dataset was created by Henry Kvinge at Pacific Northwest National Laboratory.
What do the instances that comprise the dataset represent (e-g., documents, photos, people, countries)? Characters of irreducible representations of the symmetric group of size n.
How many instances are there in total (of each type, if appropriate)? See Appendix B.1.
Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from larger set? contains all possible instances for a fixed n.
What data does each instance consist of? Each instance consists of two partitions of and an integer.
Is there label target associated with each instance? Yes, the label the final integer.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? N/A
How was the data associated with each instance acquired? The creators used the open-source mathematics software system SageMath to calculate character values.
What mechanisms procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? SageMath was used for the calculations and Python was used to sort, format, split the data. This was all done on a consumer laptop.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? N/A
30

---

Machine Learning Meets Algebraic Combinatoric
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how they compensated (e.g., how much were crowdworkers paid)? Henry Kvinge wrote the code in SageMath generate this dataset.
Was any propocesiin/deaaingllllnngg of the data done (e.g., discretization or bucketing, tokenization, part-of- speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Partitions are listed as sequences of numbers enclosed by brackets (see Appendix B.I.
Has the dataset been used for any tasks already? No.
What (other) tasks could the dataset be used for? This dataset could used in instances where the aim is to understand the ability of machine learning to perform challenging mathematical tasks.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/clanned/lbbeeel that might impact future uses? No.
Are there tasks for which the dataset should not be used? so, please provide a description. No.
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) behalf of which the dataset was created? If so, please provide description. No.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? All datasets will be compressed a single zip file and stored on Google Drive. It does not currently have DOI.
Will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO, https: ctteetreaatt////// iee.attoiiivooooomso.mo.m org/pub. pub ii--dmaii/cccc0
Have any third parties imposed IP-based or other restrictions on data associated with the instances? No.
C.2. mHeight datasheet
For what purpose the dataset created? This dataset created to study machine learning model's ability to learn the meeggh of a permutation.
Who created the dataset (e.g., which team, research group) and on behalf which entity (e.g., company, institution, organization)? The dataset was created by Herman Chau at the University of Washington.
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Instances are permutations (represented in one-line notation) followed by the integer corresponding to their mHeight.
How many instances are there in total (of each type, if appropriate)? See Appendix B.2.
Does the dataset contain all possible instances is it sample (not necessarily random) of instances from larger set? Yes, all instances are included for the values of provided.
What data does each instance consist of? An instance consists of a permutation and its corresponding mHeight.
Is there label target associated with each instance? The label is the mHeight.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? Individual instances are unrelated.
How was the data associated with each instance acquired? The creators generated the data using Python scripts which are provided at https: http:/gtttth.ccoo/ wwtpom/pnn//M44/// /MM.4A1gComm
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? Python was used to generate the data on a consumer laptop.

---

Machine Learning Meets Algebraic Combinatoric
If the dataset is a sample from larger set, what was the sampling strategy (e.g,, deterministic, probabilistic with specific sampling probabilities)? N/A
Who was involved in the data collection process (e.g,, students, crowdworkers, contractors) and how were they compensated (e.g,, how much were crowdworkers paid)? Herman Chau wrote Python code generate the data,
Was any preprocssingg/deaninn/aaaa speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? No.
of the data done (e.g., discretization or bucketing, tokenization, part-of-
Has the dataset been used for any tasks already? No.
What (other) tasks could the dataset be used for? This dataset could be used to train a model to predict the mHeight of permutations and an intermediate task for predicting the smallest non-trivial zero coefficient of Kazhdan-Lusztie polynomials.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/ckenned/laeeeed that might impact future uses? No.
Are there tasks for which the dataset should not be used? If so, please provide a description. No.
Will the dataset be distributed to third parties outside the entity (e-g., company, institution, organization) on behalf of which the dataset was created? so, please provide a description. No.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? All datasets will be compressed a single zip file and stored on Google Drive. It does not currently have a DOI.
Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO, https: www.eeettttttcceeeee org htppppulliic public-domain/ceee domain/coo/
Have any third parties imposed IP-hased other restrictions the data associated with the instances? No.
C.2.1. SEMISTANDARD YOUNG TABLEAU FOR GRASSMANNIAN DATASHEET
For what purpose was the dataset created? This dataset was created to study machine learning model's ability to identify whether an semistandard Young tableau indexes a valid cluster variable the Grassmannian cluster algebra.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? The code for the positive examples in the dataset was created by an external team consisting of Man-Wai Cheung, Pierre-Phillipp Dechant, Yang-Hui He, Elli Heyes, Edward Hirst, and Jian-Rong Li (Cheung et al., 2023). The negative dataset was created by Herman Chau at the University of Washington.
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Instances represent semistandard young tableau.
How many instances are there in total (of each type, if appropriate)? See Appendix B.3.
Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from a larger set? The positive instances are all possible instances with exceedingly high probability. An equal number of negative instances were obtained by randomly sampling without duplicates. See Appendix B.3 for details on sampling.
What data does each instance consist of? An instance consists of a 3 x SSYT.
Is there label target associated with each instance? The label is given in the filename.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? Individual instances unrelated.
How was the data associated with each instance acquired? The creators generated the data using Python scripts.
32

---

Machine Learning Meets Algebraic Combinatoric
What mechanisms procedures used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? Python used to generate the data.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? Positive examples are generated probabilistically until no new instances are generated after large number of iterations. Negative examples sampled uniformly random. See Appendix B.3 for details on sampling.
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g,, how much were crowdworkers paid)? Herman Chau generated this data. He developed his own code to generate the negative examples and used code from Man-Wai Cheung, Pierre-Phillips Dechant, Yang-Hul Hui He, Elli Heyes, Edward Hirst, and Vian-Rong Li (Cheung et al., 2023) to generate positive examples.
Was any preproesiin//eeennnnabliiiggg of the data done (e.g., discretization bucketing, tokenization, part- of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Semistandard Young tableau were flattened to fit on single line in the data files.
Has the dataset been used for any tasks already? A similar dataset used in (Cheung al., 2023), with different negative examples.
What (other) tasks could the dataset be used for? This dataset could be used for any tasks around the study of the Grassmannian cluster algebra.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/dlaned/labeeed that might impact future uses? No.
Are there tasks for which the dataset should not be used? If so, please provide a description. No.
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. No.
Will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO, https: /ttetttrraaa/eee///e/ vecommons. org/ publ i cc--ommin/cct
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
C.2.2. KAZHDAN-LUSZTIC POLYNOMIAL DATASHEET
For what purpose was the dataset created? This dataset was created to study machine learning model's ability to learn and predict the coefficients of Kazhdan KauzdinnLuszzig polynomials.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? Henry Kvinge created these datasets using code from (Warrington).
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? An instance consists of pair of permutations followed by the Kazhdan-Luszai polynomial corresponding to the pair of permutations. The polynomial is listed via its coefficients (starting with the constant term) up to the largest non-zero coefficient.
How many instances are there in total (of each type, if appropriate)? See section Appendix B.4.
Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from larger set? For given n, all KL polynomials are included.
What data does each instance consist of? An instance consists of pair of permutations followed by the Kazhdan- Lusztig polynomial corresponding to the pair of permutations.
Is there label target associated with each instance? The labels be taken be the coefficients, the degree, etc.
Is any information missing from individual instances? No.
33

---

Machine Learning Meets Algebraic Combinatoric
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? Individual instances unrelated.
How was the data associated with each instance acquired? The creators generated the data using C code from (Warrington) and Python scripts.
What mechanisms or procedures were used collect the data (e.g,, hardware apparatuses or sensors, manual human curation, software programs, software APIs)? C code from (Warrington) and Python was used generate the data.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g,, deterministic, probabilistic with specific sampling probabilities)? N/A.
Who was involved the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g,, how much were crowdworkers paid)? Henry Kvinge generated the data.
Was any pregooesiing/eeainn//bliigg of the data done (e.g., discretization bucketing, tokenization, part-of- speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Polynomials are stored as a sequence of coefficients ending with the final non-zero coefficient.
Has the dataset been used for any tasks already? No.
What (other) tasks could the dataset be used for? This dataset could be used to study various properties of the coefficients of Mazhdan-Luszztye polynomials.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/clenne//aheeeed that might impact future uses? No.
Are there tasks for which the dataset should not be used? so, please provide a description. No.
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. No.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? datasets will be compressed a single zii file and stored on Google Drive. It does not currently have a DOL
Will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO, https: / CCO.https://crat.. wtt.o/eeeaivicmmoeeeeee org/ httpppplldddaa//ceooooooo
Have any third parties imposed IP-hased or other restrictions on the data associated with the instances? No.
C.2.3. SOBINSON-SCHENSTED SCHENSTED DATASHEET
For what purpose was the dataset created? This dataset was created to study machine learning model's ability learn the Robinson-Schenstedd correspondence.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? The dataset was created by Henry Kvinge and Helen Jenne at Pacific Northwest National Laboratory.
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Each instance in the dataset is pair of standard Young Tableaux of the same shape, along with their associated permutation under the
Knuth correspondence.
How many instances are there in total (of each type, if appropriate)? See Appendix B.5.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? The dataset contains all possible instances for n and 10.
34

---

Machine Learning Meets Algebraic Combinatoric
What data does each instance consist of? An instance in the dataset consists of pair of standard young tableaux along with its associated permutation.
Is there a label or target associated with each instance? The target permutations are given separate files.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? N/A
How the data associated with each instance acquired? The creators used the open-source mathematics software system SageMath to generate the tableau pairs corresponding each permutation. (See https: /doc. sagemath. Org/htmll / en wtt.://ffeeeeeeeeeeeeee combrec/ceeeeeeeeeeeeooooooooooe comb nat sage/ combi nat haattattrsssee httl for the relevant documentation).
What mechanisms procedures human curation, software programs, software APls)? SageMath was used to generate the data and Python was used to sort, format, and split the data.
used to collect the data (e.g,, hardware apparatuses sensors, manual
If the dataset is a sample from larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? N/A
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g.. how much were crowdworkers paid)? Henry Kvinge and Helen Jenne used code from Sage to generate this dataset. The author of the original implementation of the -Scnnnsteh--------ee Schensted-Knuth correspondence in SageMath QuiverMutation Type class is Travis Scrimshaw.
Was any prepoecssingdcearigggllllnng of the data done (e-g., discretization bucketing, tokenization, part-of- speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Standard Young tableau were flattened to fit single line. Permutations are written as list of numbers.
Has the dataset been used for any tasks already? The Robinson-Schensted correspondence an important algorithm in the algebraic combinatorics community, but this specific dataset has not been used before.
What (other) tasks could the dataset be used for? This dataset could be used for any tasks involving the RSK algorithm.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/clenned/lbbeee that might impact future uses? No.
Are there tasks for which the dataset should not be used? If so, please provide a description. No.
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. No.
How will the dataset will be distributed (e.g., tarball website, API, GitHub)? Does the dataset have digital object identifier (DOI)? All datasets will be compressed single zip file and stored on Google Drive. It does not currently have DOL
Will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO. https: / http://raaaatteeeeeeeeeeeeeeeeat www.oooro/.....ooooo..ooo.ooe.oe publiccdomain//
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
C.2.4. SCHUBERT POLYNOMIAL. STRUCTURE COFFFICIENTS DATASHEET
For what purpose was the dataset created? This dataset was created to study machine learning model's ability to predict the structure constants of Schubert polynomials.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? The dataset was created by Henry Kvinge and Helen Jenne at Pacific Northwest National Laboratory.
35

---

Machine Learning Meets Algebraic Combinatoric
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Instances represent the structure constants that come from multiplying Schubert polynomials together. For fixed n, instance
[a, B, 7.c
where 3 3 E Sn, Y is another permutation (possibly larger or smaller symmetric group), and E Z2o.
How many instances are there in total (of each type, if appropriate)? See Appendix B.6.
Does the dataset contain all possible instances is it sample (not necessarily random) of instances from larger set? For any n, most structure constants will be zero. To generate balanced dataset, computed S. Sg for all elements in S, x S,, and for each ea,3 * O, we applied random number of transpositions (where the number of
transpositions sampled from geometric distribution) to y to get y checked that c 0 and added this to the dataset. Therefore the dataset contains all non-zero structure constants but only fraction of zero structure constants.
What data does each instance consist of? An instance in the dataset corresponding to consists of two permutations from S,, a permutation from another (possibly larger smaller symmetric group), and integer.
Is there a label or target associated with each instance? The final integer the instance is the label.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g,, users' movie ratings, social network links)? Instances that have the same first two permutations o, B are drawn from the same basis expansion of S. * S3-
How was the data associated with each instance acquired? The creators used the open-source mathematics software system SageMath to generate and multiply Schubert polynomials for each pair of permutations a and 3 in Sn for n = 4, 5, 6. The basis expansion Sa * S3 was obtained from this and each term in this expansion was used as an instance.
What mechanisms procedures were used to collect the data (e.g,, hardware apparatuses or sensors, manual human curation, software programs, software APIs)? SageMath was used for the calculations and Python was used to sort, format, split the data.
If the dataset is sample from larger set, what the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? For any n, most structure constants will be zero. To generate a balanced dataset, we computed S. S3 for all elements in S, x Sn, and for each c², # 0, we applied random number of transpositions (where the number of transpositions was sampled from a geometric distribution) to 7 to get y checked that e = 0 and added this to the dataset. Therefore the dataset contains all non-zero structure constants but only fraction of zero structure constants.
Who involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g,, how much were crowdworkers paid)? Henry Kvinge and Helen Jenne wrote the code in SageMath to generate this dataset.
Was any prepoocesingddeanigglabbiinn of the data done (e.g., discretization or bucketing, tokenization, part-of- speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Permutations represented as lists of integers enclosed by brackets.
Has the dataset been used for any tasks already? Schubert structure constants are an area of intense interest to the algebraic combinatories community, but this specific dataset has never been used before.
What (other) tasks could the dataset be used for? This dataset could be used for any tasks around the study of Schubert polynomial structure constants.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/clenned//bbeeed that might impact future uses? No.
Are there tasks for which the dataset should not be used? so, please provide a description, No.
36

---

Machine Learning Meets Algebraic Combinatoric
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) behalf of which the dataset was created? If so, please provide description. No.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOD) All datasets will be compressed to a single zip file and stored on Google Drive. It does not currently have a DOI.
will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO, https: pttttp:://creeatieecomoee .rgg
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
C.2.5. PARTIAL ORDERS ON LATTICE PATHS DATASHEET
For what purpose was the dataset created? This dataset was created to study a machine learning model's ability to differentiate the Lagrange and matching partial orders on lattice paths from (0, 0) to (n, 1) which do not pass above the diagonal.
Who created the dataset (e.g,, which team, research group) and on behalf of which entity (e.g., company, institution, organization)? The dataset was created by Helen Jenne at Pacific Northwest National Laboratory.
What do the instances that comprise the dataset represent (e.g,, documents, photos, people, countries)? Instances represent two lattice paths (p, 1), where q is cover of in either the Lagrange ordering the matching ordering.
How many instances are there in total (of each type, if appropriate)? See Appendix B.7.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? The dataset contains the vast majority of possible instances, but pairs (p. q) that were covering pairs in both the Lagrange partial order and the matching partial order were thrown (this was 21, 40, 79, and 183 instances for 10, 1 11 12, and 13, respectively)
What data does each instance consist of? An instance in the dataset consists of two lattice paths represented as binary sequences.
Is there label or target associated with each instance? The label is given in the filename; the Lagrange and matching covers are saved in separate files.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?
Each lattice path does not necessarily have a unique cover, so there are instances in the dataset that have the same first lattice path. This is the reason for the dataset imbalance: lattice paths have unique covers less often in the Lagrange partial ordering.
How was the data associated with each instance acquired? The creators used the open-source mathematics software system SageMath to generate all lattice paths from (0, 0) (n, n 1) that stay below the diagonal y _" J. and compute the Lagrange number T(()) and matching number M (p) associated to each lattice path p. The matching order (resp. Lagrange order) dataset consists of lattice paths (p. q) such that (g) > M(p) (resp. I(q) > L(p)) and there is not path r such that M (()) > (r) (p) (resp. L(g) > L(r) L(p)).
What mechanisms or procedures were used to collect the data (e-g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? SageMath was used for the calculations and Python was used to sort, format, and split the data.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? The dataset is not a sample from larger dataset, but we did throw out instances that are covering pairs in both the Lagrange partial order and the matching partial order (this was 21, 40, and 79 instances for 10, 11, and 12 respectively)
37

---

Machine Learning Meets Algebraic Combinatoric
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how they compensated (e.g,, how much crowdworkers paid)? Helen Jenne wrote the code in SageMath to generate this dataset.
Was any propooesinngclening/abbiiii of the data done (e.g,, discretization or bucketing, tokenization, part-of- speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? We converted lattice paths to binary codes of 0's and I's for storage.
Has the dataset been used for any tasks already? The Lagrange and matching orderings area of recent interest in the algebraic combinatorics community, but this specific dataset has never been used before.
What (other) tasks could the dataset be used for? This dataset could be used for any tasks around the study of the Lagrange and matching orderings.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/clenned//aheled that might impact future uses? No.
Are there tasks for which the dataset should not be used? If so, please provide a description. No.
Will the dataset be distrihuted to third parties outside of the entity (e-g., company, institution, organization) on behalf of which the dataset was created? so, please provide a description. No.
How will the dataset will be distributed (e.g., tarball website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? All datasets will be compressed single zip file and stored on Google Drive. It does not currently have DOL
Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO, https: http//reaatttecoooooooooe org/ ptpppuublccooe wwwiiccdmmimccett
Have any third parties imposed IP-based other restrictions on the data associated with the instances? No.
C.2.6. MUTATION EQUIVALENT QUIVERS DATASHEET
For what purpose was the dataset created? To study the problem of determining whether two quivers are mutation equivalent.
Who created the dataset (e.g., which team, research group) and on behalf which entity (e.g., company, institution, organization)? The dataset was created by Helen Jenne at Pacific Northwest National Laboratory.
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? The instances are 11 11 adjacency matrices corresponding to 11 vertex quivers.
How many instances are there in total (of each type, if appropriate)? See Appendix B.8.
Does the dataset contain all possible instances is it sample (not necessarily random) of instances from larger set? For the mutation classes A and D, the dataset contains all possible instances. The other mutation classes are not finite, is not possible to generate all instances. For these mutation classes, specify depth d and then generate the dataset so that it contains all quivers at most d mutations away from the original quiver.
What data does each instance consist of? Each instance consists of an 11 x 11 adjacency matrix which represents an vertex quiver.
Is there label or target associated with each instance? Yes, the label is specified in the file name in which the instance is stored.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g., users" movie ratings, social network links)? Yes
38

---

Machine Learning Meets Algebraic Combinatoric
How was the data associated with each instance acquired? The creators used the open-source mathe- maties software system SageMath to generate all quivers from each mutation class up to the specified depth. (See https: //doc. algebra_quivers wwwpi//eer/iiivviieeeeeeeeeeeeee quiver wwtivve_mutaatin..om type. html for the relevant documentation).
combinat heatt//sae///moooe wttpg///oomiiat//ccce /lluterrr
What mechanisms procedures
used to collect the data (e.g., hardware apparatuses sensors, manual human curation, software programs, software APIs)? SageMath was used for the calculations and Python was used to sort, format, and split the data. Computations were done on a consumer laptop.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? Sampling was done by applying 9 uniform, randomly sampled mutations of the quiver in non-finn finite type cases. For finite type all examples are included.
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e-g,, how much were crowdworkers paid)? Helen Jenne used code from SageMath to generate this dataset. The authors of the SageMath QuiverMutationTypp class are Gregg Musiker and Christian Stump, and Hugh Thomas.
Was any preprceeiinnddaaninglaallenn of the data done (e.g,, discretization or bucketing, tokenization, part- of--speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Graphs were converted to adjacency matrices and then flattened for storage.
Has the dataset been used for any tasks already? A similar dataset was used in (Bao et al., 2020) to study the ability of Naive Bayes and convolutional neural networks to classify quivers according to mutation class. Their dataset also included type A, D, and F. mutation classes, but for smaller values of n (the number of vertices) and consequently, fewer instances of each class. (Bao et al., 2020) inspired this dataset, as it seemed interesting to investigate whether better classification performance could be achieved with larger values of n. (He et al., 2024) used subset of this dataset to rediscover some known characterization theorems from quiver mutation equivalence classes.
What (other) tasks could the dataset be used for? This dataset could be used for any tasks focused on matching quivers to their mutation equivalence class.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/ckeaned//lbeeee that might impact future uses? No.
Are there tasks for which the dataset should not be used? If so, please provide a description. No.
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. No.
How will the dataset will be distributed (e.g., tarball website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? All datasets will be compressed single zip file and stored on Google Drive. It does not currently have a DOL
Will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO. https: / iereativecommons wtrgg/publice mubblicddomii//cee
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
C.2.7. WEAVING PATTERNS DATASHEET
For what purpose was the dataset created? This dataset was created to study machine learning model's ability to illuminate properties of weaving patterns.
Who created the dataset (e.g., which team, research group) and on behalf which entity (c.g., company, institution, organization)? The dataset was created by Herman Chau of the University of Washington (true weaving patterns) and Davis Brown of Pacific Northwest National Laboratory (false weaving patterns).
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Instances consist of a (1, 2, .......7 m : -vauued matrix followed by a 0 if the matrix a weaving pattern and a if not.
39

---

Machine Learning Meets Algebraic Combinatoric
How many instances are there in total (of each type, if appropriate)? See Appendix B.9.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? Each dataset contains all weaving patterns. The non-weaving patterns are a sample from all {1.2..... 72 m matrices obtained by permuting two the entries in the row of weaving pattern and checking that the resulting matrix is not weaving pattern.
What data does each instance consist of? Instances consist of a (1,2,...,71) m]-valued matrix followed by a 0 if the matrix is a weaving pattern and a if not.
Is there a label target associated with each instance? The final or in the row is the label.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g,, users' movie ratings, social network links)? No.
How was the data associated with each instance acquired? All examples were generated using Python script.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? All examples were generated using a Python script.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g,, deterministic, probabilistic with specific sampling probabilities)? Each dataset contains all weaving patterns. The non-weaving patterns are a sample from all (1,2...., n.....,.-vaauee -valued matrices obtained by permuting two of the entries in the row of a weaving pattern and checking that the resulting matrix not a weaving pattern.
Who was involved in the data collection process (e.g,, students, crowdworkers, contractors) and how were they compensated (e.g,, how much were crowdworkers paid)? Code was written by Herman Chau of the University of Washington (true weaving patterns) and Davis Brown of Pacific Northwest National Laboratory (false weaving patterns).
Was any prepraeesinghhemmigghheiingg of the data done (e.g, discretization or bucketing, tokenization, part- of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Matrices were flattened to row-major format.
Has the dataset been used for any tasks already? No.
What (other) tasks could the dataset be used for? The dataset could be used for other tasks related to the study of weaving patterns.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/cteancc/labeeee that might impact future uses? No.
Are there tasks for which the dataset should not be used? If so, please provide a description. No.
Will the dataset be distributed to third parties outside of the entity (e-g., company, institution, organization) on behalf of which the dataset was created? so, please provide a description. No.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have digital object identifier (DOI)? All datasets will be compressed to single zip file and stored on Google Drive. It does not currently have a DOI.
Will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)? CCO, https: CCO.httss://crattte
org/ borg/pubie wwtppppwwbaaia///ooppppppppp
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
40


Reviews:
[{"review_id": "f5ts29G5dq", "reviewer": "Reviewer_zx12", "summary": "The paper introduces nine datasets arising from problems in algebraic combinatorics. These datasets are meant to test capabilities of machine learning models on symbolic math tasks. The authors clearly motivate and describe each of the nine problems and the resulting benchmarks. Several baseline methods are tested on the benchmarks. Useful discussion is provided showing how development of ML models for these benchmarks can lead to progress in mathematics.\n\n**Update after rebuttal**\n\nThe authors sufficiently addressed my objections, so I raised my score and I feel I can recommend the paper for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for all their thoughtful comments on the paper and for pointing out issues with the GitHub page. We provide responses to the points in the review below. \n- *“…it seems that for the majority of the datasets the baseline methods … could achieve quite high performance.”*\n     - This is a good point and one that we tried to address in lines 086-091. Our goal is a little different than a standard benchmark where model performance alone is what we care about (this is one reason we ultimately decided to remove the word 'benchmark' from the paper). When looking for mathematical insight, being able to train a model to perform the task associated with an open problem is a necessary, but not sufficient condition. We must also be able to extract mathematical insight from the model. The model that has just learned a ‘bag of heuristics’ might be highly performant but would probably be hard to extract clean theorems from. The inclusion of the benchmarks was meant to give the reader a better sense of how challenging the first step is (getting a performant model). We have now clarified this in the text.\n- *Installation instructions*\n     - Thank you for pointing this out. These have now been added to the [GitHub repository](https://github.com/icml2025-43403439/43403439/blob/main/README.md#environment-installation).\n- *Java program*\n     - This was code written by one of our mathematician collaborators for their research prior to the inception of the ACD repo. The Java version has now been replaced with a Python version.\n- *Errors when loading data*\n     - Thank you for pointing these out. We now believe we have fixed all the issues listed.\n- *One link to download all datasets*\n     - This is a good idea! We now have a single script (`load_datasets.py`) which can be run to download, all datasets, unzip them and move them to the correct folder (other than KL polynomials, RSK, and Schubert, which are too large to download programmatically via the API). \n- *Additional references*\n     - Good catch! We have added both.\n- *Training and architectural hyperparameters*\n     - The range of hyperparameters (including number of layers, depth, etc.) that we explored can be found in Section C.1 of the Supp. Material. We will add a table with the specific hyperparameters we used for each dataset (an abbreviated copy of the transformers table can be found below). We have also added training epochs (60), batch size (varied between datasets), loss function (cross-entropy for classification and MSE for regression), and Adam hyperparameters (the Pytorch default other than learning rate) into Section C.1.\n| Dataset | LR | Depth | Dimension | Heads |\n| -| - | - | - | - |\n| Lattice, $n=10$ | 0.001 | 4 | 80 | 8 |\n| Lattice, $n=11$ | 0.0005 | 4 | 80 | 4 |\n| Lattice, $n=12$ | 0.0005 | 6 | 40 | 8 |\n| Weaving, $n = 6$ | 0.0001 | 4 | 80 | 8 |\n| Weaving, $n = 7$ | 0.0001 | 4 | 80 | 8 |\n| Quivers | 0.0005 | 6 | 80 | 4 |\n| Grassmannian | 0.0005 | 6 | 80 | 4 |\n| Schubert $n = 4$ | 0.0001 | 4 | 80 | 6 |\n| Schubert $n = 5$ | 0.0005 | 4 | 40 | 6 |\n| Schubert $n = 6$ | 0.0005 | 4 | 80 | 4 |\n| mHeight, $n = 8$ | 0.0001 | 4 | 80 | 4 |\n| mHeight, $n = 9$ | 0.001 |  6 | 20 | 8 |\n| mHeight, $n = 10$ | 0.001 | 4 | 80 | 8 |\n| $S_n$, $n = 18$ | 0.001 | 6 | 80 | 6 |\n| $S_n$, $n = 20$ | 0.001 | 4 | 80 | 8 |\n| $S_n$, $n = 22$ | 0.001 | 4 | 40 | 8 |\n| RSK, $n = 8$ | 0.001 | 4 | 20 | 6 |\n| RSK, $n = 9$ | 0.001 | 4 | 80 | 4 |\n- *Logistic regression optimization?*\n     - A short description is provided in Section C.1. We used the default settings for logistic regression provided by sklearn (we have now replaced the word 'standard' by 'default'). \n- *MLPs vs. transformer performance*\n     - This is a great question to which we mostly don’t have answers (especially for open problems where the solution is unknown). We see the choice of data representation as one critical unknown for many of these datasets. For example, what is the best way to represent a permutation when using a transformer vs MLP? We have also seen evidence that the transformer's sensitivity bias limit it in certain math settings (e.g. parity problems).\n- Typos\n     - These have now all been addressed, outside of the supplementary material being split from the main body. Unfortunately, ICML does not let us submit these as a single document.\n- *The meaning of Table 3*\n     - Correct. We have added a note to clarify this.\n- *RSK regression task*\n     - Permutations are represented as their inversion vector, a $\\{0,1\\}$ vector whose first entry is 1 if the permutation inverts the ordering of 1 and 2, whose second entry is 1 if the permutation inverts the ordering of 1 and 3, etc. This vector is the target of regression. Doubtless better frameworks could be devised by one interested in this task, but that was not our goal in the baselines. Clarification has been added to Section B.6.", "claims_and_evidence": "The main contribution of the paper is the introduction of a novel benchmark, so the focus is on presenting the benchmark and not on stating and verifying claims. However, the authors want to show that the datasets are challenging for standard ML approaches, and to this end baseline experiments are conducted. I'm not completely sure how to interpret the results -- in fact, it seems that for the majority of the datasets the baseline methods (especially MLP) could achieve quite high performance.", "methods_and_evaluation_criteria": "The authors in general choose appropriate methods and evaluation criteria for providing the baselines.", "theoretical_claims": "The authors introduce a bit of background theory in algebraic combinatorics to motivate the problems resulting in the datasets. I'm not familiar with this domain of mathematics, so I myself wasn't able to judge how appropriate the problems are -- but I believe the problems indeed are well-motivated and appropriate, as the authors seem to be well-versed in algebraic combinatorics.", "experimental_designs_or_analyses": "I didn't see any issues with the experimental design.", "relation_to_broader_scientific_literature": "This work is related to the literature on applying ML to math problems, and more specifically, to math benchmarks. There are several categories of existing math benchmarks: formal mathematics benchmarks, natural language math benchmarks of varying difficulty, and synthetic math problems. This work falls in the last category, and because there are not many such benchmarks, I find the contribution of the authors very useful.", "essential_references_not_discussed": "I think the authors in general include relevant references, however, it would be perhaps good to discuss [1], where also synthetic math benchmarks for ML are created, and [2], where the studied ML task is to generate formulas describing integer sequences from OEIS.\n\n[1] Saxton et al.: Analysing Mathematical Reasoning Abilities of Neural Models. ICLR 2019\\\n[2] Gauthier, Urban: Learning Program Synthesis for Integer Sequences from Scratch. AAAI 2023", "other_strengths_and_weaknesses": "### Strengths\n\nThe paper introduces a novel and interesting benchmark which connects ML research with more advanced mathematics. In my opinion, such benchmarks are highly desirable, and may stimulate developments on both ML and mathematics side.\n\nThe math problems and resulting data are well-described and motivated, making them accessible for non-experts.\n\n### Weaknesses\n\nSome technical details related to training the baselines are missing, for instance:\n- What training hyperparameters were used (n. of epochs)? \n- What architectural hyperparameters were used (n. of layers)?\n- How logistic regression was optimized?\n\nMost of the problems can be solved with the baseline methods with quite high accuracy, which makes one wonder how difficult the benchmarks actually are.\n\n### Conclusion \nIn general, I find the work very useful, and once the authors resolve the issues raised (the problems with the code, missing technical details), I'm willing to raise the score.", "other_comments_or_suggestions": "It would be better to provide appendix in the same pdf as in the main text -- now some links (e.g. Table 3) do not work.\n\nIn Table 2, in the caption you mention Claude 3.5 Sonnet but in the table itself it is not present.\n\nThe captions of the tables should be placed above, per formatting instructions.\n\n\"this example comes from (war)\" -- this reference needs fixing\n\nTable Table --> Table\nthese these --> these\nMachine Learning meets --> Machine Learning Meets", "questions_for_authors": "1. I see that MLP in general performs better than the transformer, but sometimes it's the other way around (e.g., on \"cluster algebra quivers\"). Do you maybe have a hypothesis why?\n2. What are the numbers presented in Table 3? is it mean squared error?\n3. I didn't understand how the RSK problem is cast as a regression task -- could you elaborate on that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4nZ5i9f2HP", "reviewer": "Reviewer_vbej", "summary": "The authors introduce a collection of datasets called the Algebraic Combinatorics Dataset Repository, which contains 9 datasets including an open-ended research question and many examples that should be used to derive conjectures. The authors describe the mathematical background of each dataset as well as results of training some neural models on them. The authors demonstrate how conjectures might be derived based on training machine learning models on this data, and leave the dataset as a good testbed for looking for conjectures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for providing feedback on the paper, especially for pointing out that we may need to provide more motivation for our choice of baselines and for suggestions on the paper’s structure. We provide responses to the points in the review below.\n- *Tension between baselining on MLPs and transformers while only providing example approaches that use more sophisticated methods.*\n     - Thank you for bringing this up. We agree that we may not have effectively communicated the purpose of our baselines. Most approaches to extracting mathematical insight from an ML model first require one to have a model that is performant on some task related to the problem one cares about. Of course, this is just a necessary but not sufficient condition (the second step, extracting insights seems to be harder in our experience). Many potential users of these datasets (especially those coming more from the mathematics community) will have some experience and intuition for MLPs and transformers. Far fewer will have experience with GNNs or program synthesis paradigms. We hope that the baselines we provide will give the broadest possible audience a sense of how \"hard\" it is just to get a performant model (acknowledging that problem \"hardness\" ultimately depends on the method used). We also did not want to bias users toward only applying program synthesis approaches which, though powerful, also have limitations. We have added in a short section at the beginning of Section 4 clarifying the purpose of the baselines.\n     - This all being said, we also recognize that program synthesis will be a topic of special interest at ICML this year so we will plan to move some additional program synthesis details back into the main body of the paper in our next revision.\n     - Finally, we agree with the reviewer that adding a third example in Section 5 that uses either MLPs or transformers would be appropriate given their central place in the baselines. We have included a short description of some results we obtained on a subclass of weaving patterns. We obtained these by using clustering to identify prototypical patterns in the Shapley values associated with a performant model. Analysis of these prototypes guided us to a formal characterization of the matrices corresponding to this subclass.\n- *Program synthesis results first in Section 5*\n     - This is a good point! We have now made this change.\n- *Mention of FunSearch*\n     - Omitting this was an oversight. We have added it in, thanks!\n- *Essential references*\n     - Thank you for pointing these out! We have added these.\n- *By what process did you select these problems for your dataset?*\n     - In choosing questions we aimed to have decent coverage of the major areas of research in algebraic combinatorics. One way of doing this is to ensure that the major combinatorial gadgets (e.g., permutations, Young tableaux, etc.) are featured. Algebraic combinatorics draws from different subfields of algebra, for instance representation theory or algebraic geometry. We also tried to make sure we had some coverage from this angle as well. It is easy to make up problems which are open but which no one cares about. To avoid this, all of our open problems were suggested by two experts in the field. The machine learning researchers on the team then checked that these problems seemed reasonable for application of machine learning and then reformulated them in an ML-friendly format. We have now added several sentences at the beginning of Section 4 addressing this.\n- *Header on Section 4.2*\n     - Thank you, we have changed this.\n- *Location of challenges section*\n     - We agree with this comment and have made this change.\n- *Use of 'can't'*\n     - We removed the contraction, thanks!", "claims_and_evidence": "The authors claim that the use of problems from algebraic combinatorics is a natural choice for their dataset, and I agree with them. The dataset is nicely curated to be useful to researchers with lesser mathematical background, as the problems do not require much background to understand. Although I am not familiar with the particular problems included int he dataset, the authors do a good job motivating their importance/relevance for such a dataset.", "methods_and_evaluation_criteria": "The authors target the field of using machinel earning models on raw mathematical data to make conjectures. I believe that the methods they experiment with make sense, and are generally well motivated. \n\nHowever, the justification for machine learning methods assisting with making conjectures involves a GNN/XAI based approach, and a program synthesis approach using LLMs. However, the authors train MLPs and transformers on their data and report results stemming from that. This seems to be at contrast with the more explainable methods they cite and use, because they don't give any explanation of how one might use MLPs/transformers to extract conjectures. In this way the experiments they run on the datasets seem to be confused with the broader goal of the dataset and the submission..", "theoretical_claims": "There are no theoretical claims in this submission.", "experimental_designs_or_analyses": "The concern I have about experimental design is the same as mentioned in the above section. While the authors train on transformers and MLPs, they don't actually mention whether or h ow these would be useful models/training regimes for extracting conjectures. The GNN based approach, and the program synthesis approach they actually experiment with seem to be much better candidates for demonstrations for researchers on how one may go about producing conjectures.\n\nI think the paper should contain much more information in the main body about the program synthesis experiment.", "relation_to_broader_scientific_literature": "I think the submission has a nice spot in the broader scientific literature for AI-for-math. At present there have not been (to my knowledge) datasets of this kind that support mathematical conjecturing from raw data. This is probably in part due to the AI-for-math research having a larger community presence of AI researchers, who are less familiar with mathematical results than the researchers from the maths world. I think this dataset provides a good starting point for researchers interested in this area.", "essential_references_not_discussed": "I think the related works is quite slim in the present form. I think the authors can include at least the two following citations: the Ramanujan Machine (which generated conjectures relating to well known constants like \\pi and e) and Graffiti, which generated conjectures in graph theory. I think the authors can also include further references in conjecturing, there are many in the literature. (Notably, what I suggested does not include machine learning, but I still think these are useful to include).", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "At the header of section 4.2 I think it may be better to leave the (Key Tool in the solution of a recently solved conjecture) in the body of the section, not in the header. My feeling is that this is too verbose.\n\nIn section 5, the part about the challenges of making such datasets does not feel to be in the appropriate spot. This may be better included in the limitations of the work.\n\nI also believe that the program synthesis for schubert polynomials, as an example, should appear first in the experiment section. This is because this is an experiment carried out by the authors. The GNNs example including (He et al. 2024) is not a contribution of this paper, and should be secondary to experiments carried out by the authors.\n\nOn line 434 in the conclusion, I recommend changing `can't` to aovid using contractions. \n\nAlso, I think you may mention FunSearch (Deepmind, '23) in your discussion of your program synthesis based approach - that methodology may be successfully applied to find conjectures fitting the datasets.", "questions_for_authors": "1. Can the authors clarify the purpose of evaluating MLPs and transformers on each of the problems? Would it not be better to use some other more interpretable methods? (I do understand you don't want to do the work for people who would use your dataset! But how might one actually use MLPs and transformers to create conjectures).\n2. By what process did you select these problems for your dataset? This information should also be included in the paper for enabling others to find suitable problems for further exploration.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0juf3uSY3U", "reviewer": "Reviewer_pA6U", "summary": "The paper introduces the Algebraic Combinatorics Dataset Repository (ACD Repo), a collection of nine datasets to use AI in advancing research-level algebraic combinatorics. \nA key contribution from author is the dataset focus on open problem with a large collection of examples. A clear focus from author is not just creating a new benchmark (models on some problems can easily do >90%) but more to see if AI can extract insights which finally lead to conjecturing.\nThe paper provides initial baselines on different datasets using different AI models with or without the language component.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the interesting questions. We provide answers below.\n- *Is there any particular reason why some problems are not interesting outside a certain boundary? For example, looking at the Github it says that for a particular problem N can be only 6/7/8.*\n     - This is a good question. In most cases we have provided the code for a user to generate any $n$ they are interested in. Of course, for sufficiently large $n$ it becomes too compute or memory intensive to generate or store the full dataset (storing all permutations of 20 elements is expensive) and at some point, even computing individual instances becomes expensive. In most cases, we decided to provide several datasets that sit at what we would consider the “sweet spot”, large and complex enough to be used to train model ML architectures but small enough that researchers with limited compute budgets could still work with them (this is generally in the 10K to 10M range). We will plan to add this reasoning into a section at the beginning of Section B.\n- *There are math areas where even small Ns lead to intractable problems. Can you clarify if this is the case for the area as well?*\n     - One can easily find problems where data is very hard to generate beyond the trivial size or where specific families of examples are hard to generate. One example of the latter comes from Kazhdan-Lusztig polynomials with $\\mu$-coefficient (Section 4.4) which is neither 0 nor 1. This is an area of interest to researchers. Unfortunately, the first such instance appears for $n=10$ at which point there are $(10!)^2$ polynomials (not necessarily distinct). We had initially aimed to include a problem around $\\mu$ coefficients but abandoned it because the computational burden was too great.", "claims_and_evidence": "No particular claim has been made.", "methods_and_evaluation_criteria": "Authors are not proposing a new method, but a new dataset.", "theoretical_claims": "There is no theoretical claim in the paper.", "experimental_designs_or_analyses": "The work includes experimental results of standard architectures to the datasets.", "relation_to_broader_scientific_literature": "This is one of the first attempt to propose challenging and open problems for the AI community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n1. Their focus is really at the conjecturing process, which is a sizable chunk of the research activity. \n2. The paper includes examples of how interpretability analysis and LLMs can be used to extract mathematical insight and generate conjectures.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Is there any particular reason why some problems are not interesting outside a certain boundary? For example, looking at the Github it says that for a particular problem N can be only 6/7/8. \n2. There are math areas where even small Ns lead to intractable problems. Can you clarify if this is the case for the area as well?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6wu9FjpNCH", "reviewer": "Reviewer_eme9", "summary": "(i) The paper introduces the Algebraic Combinatorics Dataset Repository (ACD Repo), a collection of nine datasets designed to challenge machine learning models in conjecturing and problem-solving in modern algebraic combinatorial mathematics. (ii) This dataset contains foundational results and open problems in algebraic combinatorics, including topics such as computing characters of irreducible symmetric group representations, mHeight function of a permutation, and Schubert polynomial structure constants. (iii) The paper evaluates various machine learning models, including logistic regression, MLPs, and transformers, highlighting challenges in interpretability and performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful feedback and questions. We especially appreciated the questions about differences in model performance, which we also have. We provide responses to the points in the review below.\n\n- *Addition of examples of conjectures powered by machine learning that were later validated by proofs*\n     - We agree that providing some precedent for this work would be helpful. In the paper we now cite the following. (i) A specialized graph neural network architecture was developed in [1] to explore a problem related to Kazhdan-Lusztig polynomials and insights from this work were used to prove some new theorems in [2]. (ii) In [3] the authors use machine learning to study the geometry of affine Deligne-Lusztig varieties and prove a new lower bound on the dimension of these geometric objects. (iii) We have added a description of some of our own ongoing work where an interpretability analysis of a model trained on a subclass of weaving patterns led to a new theorem.\n- *Differences in architecture performance, especially with respect to lattice paths*\n     - Thanks for noting this. We have added some additional text to Section C which discusses this issue and speculates on some reasons that a given architecture may have underperformed on a dataset. \n          1. **Data representation:** Many of the datatypes (e.g., permutations, partitions, and lattice paths) are rarely used as input data within machine learning research. Consequently, we have a limited understanding of what representations are optimal for a given architecture (there are at least a dozen common representations of permutations in algebraic combinatorics and many look very different). Further, some small-scale experiments we have run suggest (unsurprisingly) that the best representation is partially determined by the solution to the task.\n          2. **Implicit bias or architectural priors:** We speculate that in certain cases architectural priors or implicit bias coming from the training routine may be misaligned with the problem solution. We note on line 1326 that the claimed sensitivity bias of transformers may be at odds with certain problems in combinatorics where the solution requires learning a map that is very sensitive to small changes to input (e.g., permutation parity).\n     - These issues would probably be best explored in smaller, ‘toy’ settings where possible solutions are well-known, rather than the open problems of this collection. \n     - We were also surprised by the inability to obtain good performance with transformers on the lattice paths dataset and had several team members train on these independently to confirm this issue. Currently, our best guess is that because we restrict to covering relations (a small subset of all possible order relations), our dataset ends up being sparsely sampled in terms of the problem space. We have seen transformers underperform in similar settings. \n- *Dataset diversity claim*\n     - The reviewer is correct that this is mostly speculation. We will soften our language. \n- *The computational cost of training models*\n     - We have added a table to the appendix to capture these statistics.\n- *Concatenating the appendix pdf with the main paper*\n     - Unfortunately, ICML does not let us submit these as one document.\n- *Mention of Lean, Coq, and Graffiti*\n     - Thank you for pointing this out, we now reference these.\n- *Some datasets are relatively small, raising concerns about generalizability to more complex mathematical settings.*\n     - We would argue that most of these datasets (other than possibly mHeight) represent complex mathematical settings. The datasets corresponding to open problems must be complex in some sense since mathematicians have made careers out of trying to solve them and yet they remain open. Even the datasets associated with problems that are not open, such as characters of the irreducible representations of the symmetric group (where a combinatorial algorithm has been known for over 70 years) still has many questions associated with it (for instance, when characters are zero or not). In all cases we provide or point to code that allows the user to generate larger datasets for larger $n$.\n     - Finally, while we hope that our datasets will be useful to evaluate the efficacy of ML for math techniques, we stress than a non-generalizing method that helps solve one of these open problems would be a huge accomplishment.\n\n[1] Davies, Alex, et al. \"Advancing mathematics by guiding human intuition with AI.\" Nature 600.7887 (2021): 70-74.\n\n[2] Blundell, Charles, et al. \"Towards combinatorial invariance for Kazhdan-Lusztig polynomials.\" Representation Theory of the American Mathematical Society 26.37 (2022): 1145-1191.\n\n[3] Dong, Bin, et al. \"Machine Learning assisted exploration for affine Deligne–Lusztig varieties.\" Peking Mathematical Journal (2024): 1-50.", "claims_and_evidence": "The claims about novelty and difficulty of dataset are justified, but I think the following claims are problematic.\n(i) The paper claims that ML models can generate useful conjectures in algebraic combinatorics. However, there is not explicit example of a novel conjecture validated by human experts is provided.\n(ii) The effectiveness of different ML models is demonstrated, but unexpected performance gaps (e.g., transformers performing worse than MLPs in the lattice path dataset) are not sufficiently analyzed.\n(iii) The claim that dataset diversity aids in generalization is plausible, but its empirical justification is limited to performance metrics rather than deeper mathematical insights.", "methods_and_evaluation_criteria": "Overall, the methods and evaluation make sense. The datasets are well-structured, covering a range of combinatorial problems suitable for ML-based conjecturing. Evaluation metrics focus on accuracy, but additional measures such as model interpretability and robustness could strengthen the analysis. Some datasets are relatively small, raising concerns about generalizability to more complex mathematical settings.", "theoretical_claims": "The paper does not introduce new theoretical results but leverages known results in algebraic combinatorics to structure the datasets.", "experimental_designs_or_analyses": "As shown in the Methods and Evaluation Criteria section, while results are clearly presented, deeper analysis is missing in cases where models underperform, such as the transformer results in Table 1. Furthermore, more discussion on the effect of dataset difficulty and feature representations on model success rates would enhance the findings.", "relation_to_broader_scientific_literature": "The work builds on existing ML approaches for mathematical reasoning and shifts the focus from theorem proving to conjecturing. This work is also closely related (and potentially contributing to) formal proof verification (e.g., Lean, Coq), and the  Fajtlowicz’s Graffiti conjecture-generating systems.", "essential_references_not_discussed": "It seems that the paper includes all essential references in related works.", "other_strengths_and_weaknesses": "Strengths: The paper contributes novel and cutting-edge datasets with real mathematical significance. The datasets are diverse and encourage broader engagement with ML in mathematics.\nWeaknesses: The lack of discussion on surprising experimental results and the absence of clear examples of ML-generated conjectures that have led to mathematical insights.", "other_comments_or_suggestions": "Below are some suggestions for improvement:\n(i) Discuss the computational cost of training models on the datasets.\n(ii) Provide qualitative examples of conjectures generated by ML models that are non-trivial and potentially useful to mathematicians.\n(iii) Moves the appendix pdf from supplementary material to the paper.", "questions_for_authors": "(i) Why do transformers perform so poorly on the lattice path dataset, underperforming even MLPs and random guessing?\n(ii) Can the authors provide concrete examples of new conjectures discovered by ML models that were later validated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduced a suite of benchmark problems that captures research-level mathematics. Current benchmarks in AI4Math focus almost exclusively on pre-college math such as IMO. The proposed benchmark fill in this gap and is a valuable contribution to the community. The paper received unanimous accepts from reviewers. The authors are encouraged to incorporate the reviewers' suggestions in the camera-ready version.
Decision: Accept (oral)