HATSolver: Learning Gröbner Bases with Hierarchical Attention Transformers
Authors: Mohamed Malhou
Ludovic Perret
Kristin E. Lauter
Venue: ICLR 2026 Oral (2026)
Keywords: Hierarchical Attention Transformer
Groebner Basis
Symbolic Computation
Multivariate Polynomial Equations
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Efficient hierarchical attention transformers for learning to solve non-linear equations through by computing groebner bases.

Abstract: At NeurIPS 2024, Kera (2311.12904) introduced the use of transformers for computing Groebner bases, a central object in computer algebra with numerous practical applications. In this paper, we improve this approach by applying Hierarchical Attention Transformers (HATs) to solve systems of multivariate polynomial equations via Groebner bases computation. The HAT architecture incorporates a tree-structured inductive bias that enables the modeling of hierarchical relationships present in the data and thus achieves significant computational savings compared to conventional flat attention models. We generalize to arbitrary depths and include a detailed computational cost analysis. Combined with curriculum learning, our method solves instances that are much larger than those in Kera (2311.12904).

HATSOLVER: LEARNING GRÖBNER BASES WITH HIERARCHICAL ATTENTION TRANSFORMERS
Mohamed Malhou
Ludovic Perret
FAIR, Meta Superintelligence Labs Sorbonne Université, CNRS, LIP6 F-75005 Paris, France mma imalhouueel mawoouumeta.cmm com
EPITA, EPITA Research Lab (LRE)
Le Kremlin- France
ludovic perreteepita. fr
Kristin Lauter
FAIR, Meta Superintelligence Labs k dlautez er @met a. com
ABSTRACT
At NeurIPS, Kera et al. (2024) introduced the use of transformers for comput- ing Gröbner bases, a central object in computer algebra with numerous practical applications. In this paper, we improve this approach by applying Hierarchical Attention Transformers (HATs) solve systems of multivariate polynomial equa- tions via Gröbner bases computation. The HAT architecture incorporates tree- structured inductive bias that enables the modeling of hierarchical relationships present in the data and thus achieves significant computational savings compared to conventional flat attention models. We generalize to arbitrary depths and in- clude detailed computational cost analysis. Combined with curriculum learning, our method solves instances that are much larger than those in Kera et al. 2004))
INTRODUCTION
Systems of multivariate non-linear equations are ubiquitous in mathematics and its applications, emerging naturally in fields as diverse cryptography, coding theory, optimization, computer vi- sion, biology, etc c.g... Perret (2016); Colotti et al. (2024); Fontán et al. (2022); [Buchberger 2006):[Wane & Xia (000));Boulic,, et al. (2011). In contrast systems of linear equations, solving a system of multivariate non- non-linear equations, also known as the PoSSo problem, is a well-known, NP-hard Garey & Johnson (1979) computationally hard problem.
Among existing techniques, Gröbner bases Cox et al. (2007); Buchherger (1965) 2006) is the most widely used approach for solving PoSSo. Indeed, the set of common solutions of a polynomial system - also known as its variety - is studied via the ideal generated by those polynomials. Gröbner basis provides canonical generating set polynomial ideal, allowing in particular efficiently solve PoSSo. More generally, Gröbner basis is a powerful tool enabling to address a wide range of problems related polynomial ideals Cox et al. (2007), such as membership testing (determining if polynomial belongs to an ideal), elimination (reducing systems to fewer variables), finding algebraic relations (syzygies) among generators of an ideal, etc...
The versatility of Gröbner bases renders their computation inherently difficult but also appealing. From a theoretical point of view, this is illustrated by the folklore result about the double-exponential worst-case complexity for computing a Gröbner basis Mayr & Meyer (192)). This theoretical state- ment holds for very peculiar example and does not fully capture the actual hardness solving PoSSo in practice. In particular, number solutions. In such a setting, the worst-case complexity drops to single exponential.
usually assume that the variety is radical and has finite
From an algorithmic point of view, the historical method for computing Gröbner bases was intro- duced by Buchberger in his PhD thesis |Buchbergrr (1965) 2006). Over the past 25 years, sig- nificant improvements have been made, leading
efficient algorithms such as F4 and Fş

---

Faugere (1999.
2002) which implemented in major computer algebra systems, such as Maple, Magma or Singular, and open-source projects such as Msolve Berthomieu et al. (202I).
Although tremendous progress has been made, Gröbner bases remain computationally difficult, and their numerous applications make the design of efficient algorithms both challenging and rewarding. Notably, the security of cryptographic standards such as AES Cid et (0.6) (2024) or new lattice-based post-quantum standards hinges directly on the hardness of solving algebraic equations
Cid et al.
2 Albrecht et al. (2014); Steiner (2024).
2025). started to explore advanced ma- chine learning techniques for polynomial system solving. Peifer et al. (2020) employs reinforcement
Recent works, e.g. Peifer al. (2020); Kera al. (2024
learning perform S-pair selection, critical step in Buchherger's algorithm. [Kera al. (2025) uses deep learning to identify and eliminate computationally expensive reduction steps during the computation of Border bases; another fundamental tool for solving systems of equations Kehrein & Kreuzer (2055) Most notably, Kera et al. (2024) demonstrates that Transformer models are capa- ble of learning Gröbner bases computation Kera et al. (2024). The authors propose reframing the problem as a supervised learning task, where a model is trained on pairs of polynomial systems and their corresponding Gröbner bases. To enable this, they address two previously unexplored algebraic challenges of efficiently generating random Gröbner bases, and constructing diverse non-Gröbner sets that generate the same idcal as given Gröbner basis (the "backward Gröbner problem"). Their solution focuses O-dimensional radical idcals, which are common in applications.
A notable limitation of the approach presented in Kera et al. (2024) is its restricted scalability to larger polynomial systems. In their experiments, the authors were only able to handle systems with up to five variables (n & 5). Moreover, they had to significantly reduce the density of the polynomial systems for 3, 4, and 5 meaning that the input polynomials were made much sparser. This reduction in density (p << 1) was necessary to avoid overwhelming the model and hardware with excessively long input and output sequences, direct consequence of the quadratic memory and computational cost of the attention mechanism in standard Transformers. As a result, the method has not been demonstrated on denser or higher-dimensional systems, highlighting a key scalability bottleneck that must be addressed.
1.1
MAIN RESULTS
To overcome this challenge, we propose replacing the multi-head attention layer in the Transformer the most computationally intensive component of the model-with a hierarchical atten- tion mechanism that leverages the inherent tree-like structure of multivariate polynomial systems. The hierarchical attention layer operates in two distinct stages: bottom-up and top-down. In the bottom-up phase, attention computed locally at each hierarchical level, beginning at the term level (E = 0), progressing to the polynomial level, and culminating the system level. Subsequently, in the top-down phase, information is propagated back to the leaf nodes using various strategies, including cross level attention and simple additive aggregation. This hierarchical approach signifi- cantly reduces the sequence lengths processed by the attention layers, resulting in substantial com- putational cutbacks the problem dimensions grow. Notably, successfully computed Gröbner bases for systems of up to 13 variables and degree 11, which compares favorably with efficient tools such Msolve ve and STD-FGLM (see Table
To accelerate the model's learning process, we employ curriculum learning strategy, training the model on datasets with progressively increasing levels of difficulty and problem sizes. We validate the merit of this method by applying it to the base model of Kera et al. (2024), enabling it to solve variables full density. This surpasses the previous results of
systems with where the largest reported success was limited to n 5 variables and density of only p = 0.2.
(2024).
2 PRELIMINARIES
2.1 SELF ATTENTION
Self-attention is fundamental mechanism in transformer architectures Vaswani al. (2017), en- abling models to dynamically contextualize each element of an input sequence by attending to all other elements. This mechanism is crucial for capturing long-range dependencies and modeling
2

---

complex relationships within sequential data, which is essential for tasks in natural language pro- cessing, vision, and beyond.
Given set of queries Q E xx keys K E gxxd and values V Rixd the self-attention mechanism computes a similarity score between each query and all keys:
s s(Q, K) = softmax QK
R
The softmax function normalizes the scores across all keys for each query, converting them into a probability distribution that sums to 1.
The output the attention layer is a weighted sum of the values V, where the weights are the attention scores:
An(Q, K, V) - sQ2, K)V E Raad
In the context of hierarchical or structured data, such as trees or graphs, self-attention can be applied to sets of embedding vectors E Rnxd corresponding to the leaves child nodes of a parent node. The embedding vector for the parent node can then be computed by pooling function pl E). which aggregates the information from its children. Common pooling strategies include mean pooling and selection of a specific child embedding:
P(E) = C egg. P(E)
E Or p(E) = Ep.: km
We tensorize the attention function Att K, V) and pooling functions p by extending them to operate on tensors with an arbitrary number of leading dimensions, which typically represent batch size other contextual groupings.
2.2 GRÖBNER BASES
Let & be a field, k(x] = k[ [1 [I Fnl be the polynomial ring in variables over k. Let I be the ideal / //,,...,...
" hi hi..... hm E k(ar....., [n]]] C k[x] generated by . 3t C 7 is called Gröbner basis Cox et al. (2007):
fi. Sm E x k. A finite set G .g.....,
BBuchberger 1965 2006) for with respect to an admissible monomial order < (see appendix A.I]
if:
(It(g))..... It(g:)) = (It(T)
where It(1) = (I(f) E / {0}} is the set of leading terms of all nonzero polynomials in / and the notation (5) refers to the ideal generated by the set S.
As defined below, Gröbner basis not unique, which motivates introducing the concept reduced Gröbner basis. G' is reduced if each 9i is monic and and monomial of ga lies in (It(G {9.))).
For any ideal I and monomial order <, there exists a unique reduced Gröbner basis.
2.3 SHAPE POSITION SYSTEMS
Gröbner bases fundamental computational tool in computer algebra that provide algorithmic solutions to fundamental problems such as computing the variety associated to (fi..... fm) S k(x). The definition of Gröbner basis, and their properties, depend on the monomial ordering.
In particular, it can be proved that lexicographic Gröbner basis of vero-dimensional (i.e. finite number of solutions) radical ideal has triangular shape, which generically is as follows:
G (1( 91 (x,), x2 92 (3nn)...,nnn1 #n- (xw))
where h, G1. ... - E k are univariate polynomials in the last variable o., with deg gi deg h for all 1....,
1.
C k[x] is said to be in shape position if its reduced Gröbner basis has the triangular form as in equation]] [[] Following Kera et al. (2024), we will restrict our attention to ideals such that their Gröbner basis is in shape position.

---

2.4 DATASET GENERATION
We adopt the backward generation introduced by Kera et al. (2024) to construct training datasets consisting of pairs (F, G) where F is random looking system of polynomial equations and is its corresponding reduced Gröebner basis which is in shape position. Kambe et al. (2025) proved that the samples generated by this algorithm are sufficiently general, ensuring a rich and diverse training set. The backward technique proceeds as follows: 1) draw h, 91....... In-1 uniformly at random from k[r, <a subject to the degree condition above and 2) generate non-Gröbner training inputs F U, PUzG by multiplying G' with random unimodular upper-triangular matrices Un and U2 and permutation matrix P.
The resulting dataset is balanced: each (F, G) pair satisfies (F) (G), F is not Gröbner basis, yet G is. This algorithm as described above generates systems with |F |G| with n the number of variables. However, the actual algorithm of Kera et al. (2024) generates
equations by using U2 E kl 2n. |** rectangular unimodular trinnula matrix. We refer the reader to Kera
[n]*XR
al. 2024) Section 4.3 and Kambe et al. (2025) for more details.
METHOD
3.1
LIMITATIONS OF FLAT ATTENTION
Writing a system of multivariate polynomial equations as sequence of tokens grows rapidly with the number of variables and the total degrees of the equations. For instance, the number of different monomials with n 5 variables and total degree d < i0 (*t") = 3003. system of + equa- tions could therefore contain over 7-3003-7 = 147147 tokens, assuming each term is encoded using n+2 tokens (see section 3.4). This explosion in token count makes training sequence-to-sequee model, such as transformer, equations are highly structured and can naturally be represented as trees. This raises the question of whether attention-basee attend directly to every other token, a token could attend primarily to its siblings (i.e., those sharing the same parent) and to other tokens indirectly through their parent nodes, and
such data particularly challenging. However, these systems of
models can be adapted to tree-like structures: rather than having each token
System
Equation 1
Equation 2
Term 1
Term 2
Term 1
Term
<bos>
C1
E2
E2
C5
E2
31
<SEP>
C3
E3
E0
C2
E1
EU
Figure I: Hierarchical representation of the tokenized system of equations p1 = 2z +52
P2 =
3xd 2x over k(zz,, x1].
3.2 HIERARCHICAL ATTENTION LAYER
The hierarchical attention mechanism operates in two successive phases. The first phase involves the computation local attention at each hierarchical level, with information being propagated in bottom-up manner through the hierarchy.
Let denote X (0) f x--xX, x input tensor representing n-level tree. For instance, a system of lg equations, padded so that each equation has & terms which have lo symbols each. In the example above,
& = lo 4.
We fix a set of embedding dimensions (dg, di...., ... d,-1) for each of the n levels. For simplicity, one could take d, d Vi. However, computationally, makes sense consider increasing
4

---

sequence (d,), move up the tree since upper levels need to encode more information and require much less compute.
The local self-attention at level 0 (leaf level) is computed as:
(()) Au(X ( x W x w! E xloxxx
with w (0) (e(,,)))) Ro trainable weight matrices.
Subsequently, a pooling operation p is applied aggregate information and reduce the dimension- ality, yielding the input for the next levels:
x p(Y(--1) E - xxxx-- 0
(3)
y Au( win x
w( E RR xxxxxxx Vi
(4)
with W()) (6)
Rdd-ixd;
(E(g.k,))
In the second phase, information is propagated in top-down fashion either via cross-attention mechanisms or simply additive aggregations. The latter is trivial and does not impact much the cost analysis, therefore we'll explain the former approach.
At each level, nodes refine their representations by extracting relevant contextual information from their corresponding parent node and their siblings the upper level:
Z (n- (
z
U(!) E R 1x-xXXx;; vinn- (5) (1) (0) A u( Z 2
(1)
where the queries, keys, and values are viewed as having X x... x l442 leading dimensions with the remaining fii+1 sequence length for the queries y()) whereas the keys and values (parents) are of sequence length f++1- The matrices (
(E(K,)) E Rd11 are trainable projection weights.
3.3 COST ANALYSIS
To simplify the analysis, omit the batch size and the notion of multi-head attention, and neglect some lightweight operations such as pooling in the following calculations. We also use the notation L, [Tk= & with L = Lu total length of the flattened input.
At level of the bottom-up phase, the projections w-(6) (g.k.o) have a complexity of 3L, x d= and the attention is (,),) The total complexity per level is:
Cip = 3L + 2L,4d
For the second phase at level i, the projections cost 21 x d;-id; and the cross attention call is O(2li+2 ((411)) x de 1)-1). Which adds up to:
Coown 2L di+ 1d + I1
Let l, = for convenience. The total computational complexity
カ-1
R-2
C=
Ci
Ci down
with d_1 d
n-1
n-2 1 + 2L,
Σ 3L,d,- id 2L,,dd Σ
n-1
n-1
3Lnd ddg + Σ 5L + Σ 2Lyd(( + +(41)
(8)
1=0
By choosing (d.); appropriately, we can control the complexity to be dominated either by the pro- jections or the attention mechanism. It also allows to choose the distribution of the compute over the tree, either allocate most of the compute to the lower level (e.g. d; S 1d4-1d=11) or to the top
level (e.g. d, 2 4-11d-1) distribute the compute the tree 14-1dh-1 & dg S 1-1d-11-
5

---

Case where d, d: Complexity is overwhelmingly dominated by the terms 3Ld 2Ld(Lo (1) while the flat attention's s cost is 3Ld + dL'd. When the sequence lengths are larger than the embedding dimension d. when scaling up the inputs, the dominating factors are (6o + ld versus ddd. For regular tree (i.e. an Ł-ary tree with & L the cost I à d only.
3.4 POLYNOMIAL ENCODING AND TOKENIZATION
We consider systems over finite fields k Fg (we chose q 7 for all of our experiments) and we adopt the standard lokenization of Kera et al. (2024) without the hybrid embedding. Ed} U
Namely, the vocabulary consists of the union of the sets {CI, ".......q-l.. Cq- U (E0,E1,... { <bos>,,,, <sep>> with d maximum degree in the dataset. A polynomial Σg ay 42 Ix
then tokenized by joining the encodings of each term by the plus token, each term is tokenized by encoding the coefficient and the powers of the variables in each term as Ca, Euj Eug Eu, including the null powers u, =0
Example: Consider the polynomial system: P1 = xgpe + +52/a1, P2 3.g + +z1 The tokenization produces the sequence:
<bos>
E2 <+> E2
<sep> C3 EO
E1 E0
3.5 POSITIONAL EMBEDDING
To encode positional information in multi-dimensional sequential data, propose learnable em- bedding scheme that generalizes standard positional encodings to arbitrary tensor shapes. Given an input of shape 1,,-1,.-- ... lo, d), we associate each axis j with dedicated embedding table E(J) E mamaa_ Xd The positional embedding for a token at index (in-1, 1 ......., in) is then con- structed as the sum of the corresponding embeddings from each dimension, i.e., PE(in-I,.... io) =
Σj=o E( This approach enables the model capture hierarchical and multi-axis positional de- pendencies in parameter-effie manner, and can be extended concatenation followed by linear projection if desired.
4 EXPERIMENTS
4.1
MODEL. COMPARISON
2 presents a comparison of different model architectures on the task of predicting Gröbner bases. All models were trained for 52 hours 8 V100 GPUs using backward-generated dataset consisting of million multivariate polynomial systems over Ey, with n variables and density p 0.33.
Figure
The baseline corresponds to the model from Kera et al. 2024). which uses encoder and de- coder layers with an embedding dimension of 1024. In contrast, our proposed HATSolven and HATSolver-3 models employ the same architectural hyperparameters as the baseline, but replace the standard attention mechanism with our hierarchical attention layer.
In HATSolver-2 2, we use two hierarchy levels. At the lowest level (level 0), the model attends to the individual tokens within a term. At the next level (level 1), it aggregates across all terms within all equations.
HATSolver-3 introduces third hierarchy level. Here, level again corresponds the tokens of term. Level 1 groups tokens into complete terms within each polynomial, while level 2 aggregates across the set of polynomials.
As shown in the figure, both BaTSolver-1 and HATSolver-3 demonstrate faster convergence and achieve higher accuracy compared to the baseline throughout training. HATSolvess SasSovve-222 was faster in training and performed about 450K training steps while the baseline model did less than 300K steps during the trainig period. The case HATSolver- is particularly noteworthy: al- though it requires padding forcing all polynomials to be represented with the same length- -the method remains faster than the baseline. The cost of padding is significant here because polynomial lengths vary widely: in this setting (n 6.p 0.33), polynomials range from only 12 terms up

---

Training Loes
Evaluation Loss
Rwokinn
3x
Taceline
2EAT Selvec 2
HATS 2
2x
25A7 Solvcc
HEAT SotSer 3
Twining Loss
Evaluation Loss
10
10-
6x
6x
40
1x
2000
4000
5000
2000
2000
4000
6000
8000
Training Steps (x 50)
Training Steps 50)
Token Accuracy (%)
Sequence Accuracy (%)
Dbwaline RATSeeer 2
HAT Subver 3
94
45
Toloin
10
80
Suacine
00
SEA Solvet 3
2000
4000
6000
2000
-5
2000
40000
6000
8000
Training Steps (x 501
Training Steps 50)
Figure 2: Training dynamics comparison; training models on predicting Gröbner bases for 52 hours on 8 V100 GPUs using the backward generated dataset of size million multivariate systems over F7 with n 6 variables and density p = 0.33. Baseline is the Kera al. (2024) base model with 6/6 encoder decoder layers and 1024 embedding dimensions. The HATSol ver-2/-3 models are our models with the same hyperparameters as the baseline, with the attention layer replaced by our hierarchical attention layer with 2 and 3 levels respectively. Sequence accuracy refers to the exact match accuracy of the model.
238 terms, with an average of 63 terms. This variability means that shorter sequences are heavily padded, effectively doubling the number of training tokens. Despite this overhead, HATSO 3 still maintains a clear advantage over the baseline in both speed and accuracy. More results for n = 7 and n = 10 including other finite fields such as F17 and Ezi are reported in the appendix figs. andd 5 and appendix C.4
4.2 SCALING UP HATSolver-3 (n = 13)
The instances solved in previous experiments proved to be relatively tractable for classical algo- rithms such as STD- FGLM Greuel et al. (2009); Faugère et al. (1993), as demonstrated in table [I] Hence we now focus on larger and more challenging problem instances. In this section we scale up our model architecture by expanding the embedding dimension to d' 1408 while keeping the number of layers to 4/4 encoder/decoder layers. The HATSolver-3 model is trained on a diverse dataset of 33-variable polynomial systems with various sparsity levels using the curriculum learning schedule discussed in appendix B with a = 2 and v 2. We cap the number of terms per equation in the datasets to 400 to avoid memory spikes. We give some statistics about the datasets in table 7 We use STD-FGIA algorithm provided in SageMath with the libSingular backend and Msolve as comparison basclines. These algorithms compute the Gröbner basis in the graded reverse lcxico- graphic order (grevlex) which is efficient, and followed by the FGLM/Fauger et al. (1993) algorithm for the change of term order to the lexicographic (lex) order.
Table I presents the performance of HATSolver= evaluated across range of systems of equa- tions over F,. Overall, the results indicate that HATSolver-3 is capable of learning to compute Gröbner bases and beats the classical algorithms STD-FGLA and Msolve ve. It is important to note
7

---

Density (%)
30
40
50
70
80
100
Success (%)
52.5
49.4
47.1
49.5
55.2
56.1
61.2
60.8
Support Acc.
65.0
63.0
59.0
61.0
68.0
70.0
94.0
91.0
Ours
Per Token Acc.
99.8
99.7
99.8
99.8
99.8
99.8
99.9
99.8
Runtime (s)
224
241
273
277
280
287
292
300
STD-
Success (%)
33.5
19.6
10.7
8.5
8.2
7.4
6.1
6.5
FGLM
Runtime (8)
652
936
1025
995
1068
687
1012
1 1129
Success (%)
32.8
17.0
11.7
6.9
7.1
4.9
4.5
4.6
Msolve
Runtime (s)
787
773
1274
766
1010
852
1232
588
Performance HATSolver-3 on computing Gröbner bases for polynomial systems with 13 variables over F7, across varying system densities with comparison to traditional algorithms STD-FGLM Greuel al. (2009) : Faugere et al. (1993) and Msolve ve Berthomieu et al., (202T). The density (%) indicates the proportion of nonzero terms in the matrix U2 in backward generation 2.4 controlling the sparsity of the systems. Success denotes the exact match test accuracy, i.e. the percentage of the test set instances for which the model generated the exact correct Gröbner basis. Support Acc. measures the accuracy when only the support (i.e., the set of monomials) of the polynomials is considered, treating two polynomials monomials regardless coefficients. Per Token Ace. reports the average token-level accuracy over all outputs. The reported runtime corresponds to the output generation phase, noting that optimization efforts were implemented (e.g. key-value caching. inference engines, etc.). For the STD-FGLM and Msolve algorithms, run is considered failed after 2 hour time limit.
Table 1:
equal if they have the same set of
that the runtime reported for our model refers to inference time only. Training is performed offline once and does not depend the number of evaluation samples. The model's exact match accu- racy fluctuates with density, exhibiting no simple monotonic trend, but reaches a peak of 61% at 90% density, with support accuracy (see the caption of table [ reaching 94%. This indicates that challenging instances exist across all levels of sparsity, and that the difficulty of the problem is not solely determined by the density of the system. Notably, when the same training experiment was conducted using the base model, it failed to learn any meaningful solution. The STD-FGLM and Msolve ve algorithms on the other hand do terminate after running for hours for most cases, with up to 93.5% timeout rate for the full density samples and average runtime of 119s (resp. 1434s) for the remaining 6.5% (resp 6%) completed runs.
4.3 ISOLATING ARCHITECTURA, IMPACT: HATSolver ON n = 13 WITHOUT CURRICULUM
To disentangle the contribution of the hierarchical attention architecture from the benefits of currieu- lum learning, we conducted an ablation study where HATSolver- was trained on a single dataset configuration without any staged progression. Specifically, we trained directly on dataset of size million samples with n 13 variables over the same finite field at density 0.9.
Density (%)
10
20
30
40
50
60
70
80
90
100
Accuracy (%)
14,40
14.31
13.45
14.72
18.36
20.11
20.10
26.42
33.85
28.57
(Exact Match) Accuracy of HATSol ver-3v trained without curriculum ables over F, at Prain 0.9, evaluated across densities Prest E 0.2, 1.0). Each test set contains 1000 systems. Model architecture: 4 encoder layers, 4 decoder layers, embedding dimen- sion 1408. Bold indicates training density.
Table 2:
13 vari-
Results and Analysis. revle three key findings. First, HATSolver-2 achieves 14-34% accuracy across test densities even without curriculum, demonstrating that the hierarchical architec- ture alone captures meaningful polynomial structure. Second, performance peaks at the training density 0.9 (33.85%) and degrades with distance from this regime, particularly low densities (p < 0.4: 13-15%). Meaning that the sparse regime is too far from the training distribution. Third, comparing curriculum-tranee results (Table shows curriculum learning provides a substantial
8

---

boost: from 33.85% to 61.2% 0.9. This confirms that while hierarchical attention provides the representational capacity for n = 13 systems, curriculum learning is a complementary technique for better performance. Another factor that could contribute to the performance gap is the amount of training data, as the curriculum-trained model was exposed to more data points than the model in this experiment.
4.4 TRAINING ON THE NEW BACKWARD-GENERRATED DATA.
We further evaluate HATSOl ver on datasets generated via the backward method of Kera et al. (202).. Note that while this method was designed for Border basis pairs, we utilize it here for Gröbner bases, incidentally validating its use for this task. By sampling random matrices, this approach offers a simpler alternative to the unimodular transformations used previously. It is worth noting, however, that the method is probabilistic: its correctness depends on the field size, which can be limiting factor on smaller fields.
We train HATSolver models with the same architecture on systems with 5 variables across
three different finite fields: F7, 16 and 1777 Results are presented in table 3 The model achieves comparable performance across all three fields, with accuracies ranging from 43-52% on low-density systems (p 0.2) and gracefully degrading to 22-33% on full-density systems (p 1.0).
Table 3: Exact Match Accuracy (%) of model on Gröbner basis prediction of multivariate poly- nomial systems with n = variables across different densities over multiple finite fields. Training is done on 1 A100 GPU per field for 24 hours using the backward data generation method from Kera et al. 2025). Model parameters are 2 encoder layers, 6 decoder layers, 1024 embedding dimensions.
Field
Density (%)
20
40
60
80
100
F7
52.45
48.34
44.32
37.28
33.63
F
43.35
36.86
35.31
25.96
24.01
F17
44.01
43.78
32.08
29.39
22.82
5 RELATED WORK
Yang et al. (016)) were the first to introduce Hierarchical Attention Networks (HAN) for classifying
Documents-modelie them as sequences of sentences, which turn are sequences of tokens. This
approach has inspired numerous subsequent works Chalkidis et al. (2019); Wu et al. (202 I); Liu
ct al. 20022; : Chalkidis et al. (2022); Yu et al.] 2 Liu et al. (2024); Slagle (022)) Ho et al.
22024 For instance, Wu al. (2021) introduced Hi- Transfforrrrr hierarchical interactive archi-
tecture for long document processing through a three-stage design. Their approach first employs a sentence Transformer to learn contextual representations for words within each sentence, generating sentence-level representations using special [CLS] tokens appended to each sentence. Subsequently, document Transformer processes these sentence representations with added positional embeddings to capture global document context and produce document context-aware sentence representations. third sentence Transformer stage enhances word-level modeling by propagating the global doc- ument context back to individual sentences. This is achieved by simply concatenating the global token to the local tokens and applying the transformer on this combined sequence-a process distinct from our second phase. Their complexity analysis demonstrates significant efficiency gains.
The recent work by Videau et al. (2025) introduces AU-Net, an autoregressive U-Net model that integrates tokenization and representation learning into multi-stage hierarchy operating directly raw bytes. Unlike traditional fixed tokenization methods such as Byte Pair Encoding (BPE), AU-Net dynamically pools bytes into words and multi-wordd chunks. This approach eliminates the need for predefined vocabularies and large embedding tables, and allows the model to handle rare or unseen tokens more efficiently.
While these hierarchical Transformer architectures focus on natural language processing tasks, simi- lar hierarchical attention principles have also been adapted to other domains such computer vision

---

Liu et. al. 2021 Liu et al. 2024), where computational efficiency is critical. Liu al. (2024) ad- dress the computational and memory inefficiencies of standard Multi-Head Aetfntttenioon (MHSA) in vision transformers by introducing the Hierarchical Aultii-ead Self ooff-Attetiion (H-MHSA) mech- anism. In their approach, the input image is initially partitioned into small patches, each treated as token. H-MHSA first computes self-attention locally within small grids of patches significantly reduce the computational burden. These local features are then merged into larger patches, and global attention computed over the reduced set of tokens allow for the modeling of long-range dependencies.
Our model generalizes these prior works, though with distinct structural changes tailored our problem setting. First, we introduce a top-down cross-attention phase that is absent in standard hierarchical models. For example, where Wu et al. (2021) propagates global context by simply concatenating global tokens to local sequences, we use cross-attention to redistribute context to lower levels. Second, while most existing architectures are restricted fixed two-level hierarchy (e.g., local/global patch/image), our design supports arbitrary tree depths. This is necessary for symbolic computation, where polynomial systems exhibit recursive nesting deeper than typical document image structures. Finally, implement this hierarchy within single, self-contained layer. This allows it to act as a drop-in replacement for standard attention, rather than requiring a specialized network backbone.
Another important direction for introducing inductive bias and improving computational efficiency is through input encoding and tokenization strategies that directly shorten sequence lengths. For instance, Kera et al. (2025) propose detailed scheme for efficient input sequence representation in polynomial systems. Instead of the standard approach-where each monomial is tokenized into n + 1 tokens (one for the coefficient and n for the exponents) introduce a "monomial embed- ding" method, representing each monomial (along with its follow-up token) as a single vector. This reduces the input size by removing the (n + 1) factor present in traditional representations, which is especially significant as the number of variables and degree increases.
6 CONCLUSION
In this work, we successfully design and implement a Hierarchical Attention Transformer (HAT) for the task of computing Gröbner bases for multivariate polynomial systems. Our results demonstrate substantial computational improvements over previous results, as well as superior scalability of the HATSolver compared to the standard Transformer baseline. Transformer-basse solvers may prove to be good replacement for solving multivariate polynomial equations in the absence of more effi- cient traditional algorithms. We leave it as future work to investigate whether model can be trained on any finite field, including non-prime fields such as power-of-twi fields (e.g. IF IF(e) used in cryptog- raphy, and whether the model can generalize to unseen fields. Additionally, some of the techniques explored in this work may offer useful insights for improving HAT models in natural language processing. Further research is needed to assess their practical impact in this domain. A natural extension is to align the training objective with the computational steps of the Gröbner basis algo- rithm, intersecting with the growing field of Neural Algorithmic Reasoning Veličković et (2020); Georgiev et al. (2024); Hashemi al. (2025). Future work should investigate whether implicitly learns algorithmic primitives-sun as S-polynomial construction and reduction-on if explicit supervision on these intermediate steps required to achieve robust -fittt-iiittibutiin gen- cralization.
vee
7 REPRODUCIBILITY STATEMENT
To ensure reproducibility, generated all datasets using the publicly available scripts from the repository associated with the prior work we build upon: Kera et al. (2024) This repository provides detailed instructions and code for dataset creation, enabling other researchers to replicate our data generation process in alignment with previous studies.
We have included comprehensive list of hyperparameters used in our experiments in Appendix tables 6 and 12 to facilitate exact replication of our training and evaluation procedures. Our own
10

---

codebase is not publicly available yet. However, plan to open source it in the future enable the research community to validate and build upon our results.
REFERENCES
Martin R. Albrecht, Carlos Cid, Jean-Charles Faugère, Robert Fitzpatrick, and Ludovic Perret. Algebraic algorithms for LWE problems. IACR Cryptol. ePrint Arch., pp. 1018, 2014. URL
http://epring pemprrnnttaarr........mii 02..r.oo//2014/1011
Jérémy Berthomieu, Christian Eder, and Mohab Safey El Din. msolve: A Library for Solving Poly- nomial Systems. In 202/ International Symposium on Symbolic and Algebraic Computation, 46th International Symposium Symbolic and Algebraic Computation, pp. 51-58, Saint Petersburg, Russia, July 2021. ACM. doi: 10. 0021145/345214334655555
François Boulier, François Lemaire, Michel Petitot, and Alexandre Sedoglavic. Chemical reac- tion systems, computer algebra and systems biology (invited talk). In Vladimir Gerdt, Wolfram Koepf, Ernst W. Mayr, and Evgenii V. Vorozhtsov (eds.). Computer Algebra in Sci- entific Computing 13th International Workshop, CASC 2011, Kassel, Germany, Septem- ber 5-9, 2011. Proceedings, volume 6885 of Lecture Notes in Computer Science, pp. 73-87. Springer, 2011. doi: 10. 00.1077998--3-642225566 978 3 642 23568-22 9_
URL /ttps::/dci.orgg/
Bruno Buchherger. Ein algorithmus auffinden der basiselemente des restklassenringes nach einem nulldimensionalen polynomideal. Ph. D. Thesis, Math. Inst., University of Innsbruck, 1965.
Bruno Buchberger. Bruno buchberger's phd thesis 1965: An algorithm for finding the basis elements of the residue class ring of zero dimensional polynomial ideal. J. Symb. Comput., 4173-4):475- 511. 2006. doi: 10.1016/..SC.2205.00.. 2005. 0.05.09.007 007
URL wttp:://///....001.0..11100.1
jsc.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural legal judgment prediction in english. arXiv preprint arXiv: 190X:::900600055) 2019.
Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. An exploration of hicrarchical attention transformers for efficient long document classification. arXiv preprint arXiv:2210.0553 2022.
Carlos Cid, Sean Murphy, and Matthew J. B. Robshaw. Algebraic aspects the advanced encryp- doi: 10. 0011077978--0-88--368222299
tion standard. Springer. 2006. ISBN 978-0-387-24363-4, URL https:://oi.orgg 1ttp://111110007000000011
Alessandro Colotti, Jorge Gareía Fontán, Alexandre Goldsztejn, Séhastien Briot, François Chaumette, Olivier Kermorgant, and Mohab Safey El Din. Determination of all stable and un- stable equilibria for image- 2024. doi: 10. 0019//RO.2024.42220 3422050
image-point-best point- visual servoing. IEEE Trans. Robotics, 40:3406-3424
URLhttts: //ooi//iii//// aeroi.org//00.0 110./ TRO. 2024.
David A. Cox, John Little, and Donal O'Shea. Ideals, Varieties, and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Algehra, 3/e (Undergraduate Texts in Mathematics). Springer- Verlag, Berlin, Heidelberg, 2007. ISBN 0387356509.
J.-C. Faugère. A new efficient algorithm for computing gröbner bases (F4). Journal of Pure and Applied Algebra, (39(1-3)661-88 1999.
J. Faugère. new efficient algorithm for computing gröbner bases without reduction to zero F5. In ISSAC'O2, pp. 75-83. ACM press, 2002.
J.C. Faugère, P. Gianni, D. Lazard, and T. Mora. Efficient computation of crro-dimensional gröbner bases by change of ordering. Journal of Symbolic Computation, 16(4):329 344, 1993. ISSN 0747-7171. doi: http:///oi.org// http://bii.og/1...000//s....o.. 19tp:///sco..939.1100. 1051, URL https: /www. sciencedio rect com/sct http://c..cc...a.aaaaaaaaaaaaaaa

---

Jorge García Fontán, Alessandro Colotti, Sébastien Briot, Alexandre Goldsztejn, and Mohab Safey EI Din. Computer algebra methods for polynomial system solving at the service of image-based visual servoing. ACM Commun. Comput. Algebra, 56()):3 40, 2022. doi: 10. 00.11445/572867.3377993 URL htR.:ttps://det org/oogg/11o 111.1..111111111111 3572871
M. R. Garey and David Johnson. Computers and Intractabilityy A Guide to the Theory of NP- Completeness. W. H. Freeman, 1979. ISBN 0-7167-1044-7.
Dobrik Georgiev Danilo Numeroso, Davide Bacciu, and Pietro Liò. Neural algorithmic reasoning for combinatorial optimisation, 2024. URL http::tts//.t..... wttp:://rxvvv.oo h
06064
G. M. Greuel, G. Pfister, and H. Schönemann. Singular: computer algebra system for polynomial computations. ACM Commun. Comput. Algebra, 1203):880-181, February 2009. ISSN 1932-
2232. doi: (1144//154434775444777 URL pettps://doi.oog wtt..o/g/....111111.11 1145//5503114 1504377
Baran Hashemi, Kurt Pasque, Chris Teska, and Ruriko Yoshida. Tropical attention: Neural algorith- mic reasoning for combinatorial algorithms, 2025. URLhttts://axxiv.ooom http://ww...rrra/b//5555555
71
Namgyu Ho, Sangmin Bae, Taehyeon Kim, Hyunjik Jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Se- Young Yun. Block transformer: Global-to-local language modeling for fast inference. Advances Neural Information Processing Systems, 37:48740-48783, 2024.
Yuta Kambe, Yota Maeda, and Tristan Vaccon. Geometric generality of transformer-based gr\," obner basis computation. arXiv preprint arXiv:2504. /2465, 2025.
Achim Kehrein and Martin Kreuzer. Characterizations of border bases. Journal of Pure and Applied Algebra, 196(2-3):251-270 2005.
Hiroshi Kera, Yuki Ishihara, Yuta Kambe, Tristan Vaccon, and Kazuhiro Yokoyama. Learning to compute gröbner bases. Advances in Neural Information Processing Systems, 37:33141-33187, 2024.
Hiroshi Kera, Nico Pelleriti, Yuki Ishihara, Max Zimmer, and Sebastian Pokutta. Computa- tional algebra with attention: Transformer oracles for border basis algorithms. arXiv preprint N:Xi::2505.23696 2025.
Yang Liu, Jiaxiang Liu, Li Chen, Yuxiang Lu, Shikun Feng, Zhida Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Emiiesppase: Learning hierarchical efficient transformer through regularized self-attention. arXiv preprint arXiv:2203. 12776, 2022.
Yun Liu, Yu-Huan Wu, Guolei Sun, Le Zhang, Ajad Chhatkuli, and Luc Van Gool. Vision trans- formers with hierarchical attention. Machine intelligence research, 21(4):670-683, 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Ernst W Mayr and Albert R Meyer. The complexity of the word problems for commutative semi- groups and polynomial ideals. Advances in mathematics, 46(3):305-329, 1982.
Dylan Peifer, Michael Stillman, and Daniel Halperm-ccistner, Leamnig selection strategies in buch- 01917
berger's 's algorithm, 2020.
Ludovic Perret. Bases de Gröbner en Cryptographie Post-Quantique. (Gröbner bases techniques in Shantum-Safe Cryptography). 2016. URL.itppss/rtaal.acceeess-oeosee
tel fal-01417803
Kevin Slagle. Spacebyte: Towards deleting tokenization from large language model- ing. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tom- czak, and C. Zhang (eds.), Advances Neural Information Processing Systems, vol-
37, pp. 124925-12442 Curran Associates, Inc., 2024. doi: 10.52202/ Dtt.:pee//pppoedis..eeerrrpoop.p http://eccccccppeeeeeeeeeeeerrr 079017-3967.
12

---

Matthias Johann Steiner. The complexity of algebraic algorithms for LWE. In Mare Joye and Gregor Leander (eds.), Advances in Cryptology- EUROCRYPT 2024 43rd Annual International Conference on the Theory and Applications of Cryptographic Techniques, Zurich, Switzerland, May 26-30, 2024, Proceedings, Part III, volume 14653 of Lecture Notes in Computer Science, URL https://doi:
pp- 375 403. Springer, 2024. doi:
org/10. 1007/978 3- 031 58734-28 13
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Petar Veličković, Rex Ying, Matilde Padovano, Raia Hadsell, and Charles Blundell. Neural execu- tion of graph algorithms, 2020. /tLLhttpee//rrxtr..rr/baaas/////
Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite, Marc Schoenauer, Olivier Teytaud, and David Lopez-Paz. From bytes to ideas: Language modeling with autoregressive u-nets. arXiv preprint 251.::2600..4671,, 2025.
Dongming Wang and Bican Xia. Stability analysis of biological systems with real solution classifi- cation. In Manuel Kauers (ed.), Symbolic and Algebraic Computation, International Symposium ISSAC 2005, Beijing, China, July 24-27, 2005, Proceedings, pp. 354-361, ACM, 2005. doi: 10.1 10.1145/1073884. 1077110 URL UtLhttss: //tp:://doo.... /oii www.oog///.. 11e11111111388 1073933
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. Hi Hi-transformer: Hierarchical interactive transformer for efficient and effective long document modeling. arXiv preprint arXiv:2 106.01040, 2021.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hicrarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies,
pp. 1480-1489, 2016.
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 36:78808-78823, 2023.
A MATHEMATICAL FOUNDATIONS
A.1 MONOMIAL ORDERS
Let k be field and k[x] k be the polynomial ring in n variables over k. A monomial in k(x] is an expression of the form a: ...rff where a; 2 0 are non-negative integers. We denote monomials as x" where
(a1..... ax) E N'
monomial order on k k[x] is total order on the set of monomials satisfying:
1. 1 < for all a # O
2. xf <x x". then x x for all E N
The lexicographic order
-tex with 21 > 322 > >... In defined by x" -jex x² if the leftmost nonzero entry of 3 is positive. For any nonzero polynomial f E k[x], the leading term It(f) is the largest monomial appearing in of with respect the chosen monomial order.
A.2 ZERO-DIMENSIONAL SYSTEMS
Let 1 C. k[x] be an ideal. The variety of I is defined as: V(T) (a E.n ((a 0 for all f E 1}
where Te is the algebraic closure k.
An ideal I is called zero-dimensional if its variety V(I) is finite, i.e., [V(I) co. Equivalently, is pero-dimensional and only if the quotient ring k[x]/I is mnite-dimensionall as a vector space over k.
13

---

Evaluation Protocol We evaluate on held-out test sets To...., Ta-1 using uniform sampling across all complexity levels: (i) = for = o,..., performance metrics reflect the model's 's ability across the full range of problem complexities and B CURRICULUM LEARNING SCHEDULERS
We use a structured curriculum learning approach to progressively train our model on systems of increasing complexity. Therefore, we generate datasets Do.... Pn of increasing number of vari- ables and increasing densities. During training, we sample from dataset Di with probability:
(i=n(t))² exp
2x
i(t,
with
p(t) =
Ej=0 exp
()-((()² 2
steps per epoch
where the curriculum center (t) advances with
The hyperparameters (learning pace) and a (curriculum width) control the progression speed and overlap between difficulty levels. Setting advances the curriculum focus by one variable count epoch, while determines how much probability mass is distributed to neighboring complexity levels.
Gaussian scheduler ensures smooth and stable transitions between complexity levels while maintain- ing exposure to simpler problems throughout training. Early training focuses on low-dimensional systems where the model can learn basic reduction patterns, while later stages emphasize higher- dimensional systems that require more sophisticated elimination strategies. The probabilistic sam- pling prevents abrupt difficulty jumps that could destabilize training.
This uniform evaluation ensures that
not being biased toward the current curriculum focus. At the end of training, we evaluate the model
on each dataset separately to characterize the model's performance and scaling behavior.
B.1 IMPROVING BASELINE MODEL WITH CURRICULUM LEARNING
We trained the base transformer model from Kera al. 2024) using the curriculum learning ap- proach explained in appendix B over polynomial systems over finite fields. In this experiment, used a model architecture with 4 encoder and 4 decoder layers and an embedding dimension of d = 1024, which differs from the 6-layer encoder/decoder and d 512 configuration used in Kera et al. (2024). This is motivated by our ablation experiments in appendix C.T which indicate that wider models perform better and train faster than deeper models. The curriculum gradually in- creased both the number of variables (n 2 to n = 7) and the system density (p). with each training task representing 1 million samples. Unlike the original Kera et al. paper, which only considered up to n = 5 and p = 0.2 and trained one model per configuration, our curriculum enabled the model to learn and generalize to much larger and denser systems once.
Table 4 reports the model's 's accuracy in predicting Gröbner bases for multivariate polynomial sys- tems over F7, evaluated on 1,000 test samples per setting. The curriculum learning approach enables the model to learn beyond the settings considered in prior work, achieving an accuracy of (0.41) for at full density p significant improvement over the baseline in Kera al. (0224). which only solved up to at a much lower density of p= =0 0.2.
C COMPLEMENTARY EXPERIMENTS
C.1 THE CHOICE WIDE VERSUS DEEP BASE TRANSFORMER
We present here an ablation experiment on the choice of embedding dimension and number of layers for the base Transformer model. We train wide versus deep models with the following configurations (encoder layers/ / decoder layers embedding dimensions): (3/3/ 1024), (4/4/ 1024), (6/6 512), and (8 / 512) on multivariate datasets with n = 3 variables and density p = 0.5. The training metrics shown in fig. which indicates that wider models perform better than deeper models for a fixed parameter budget. Based on these results, we choose to use wider base Transformers in our HATSol 1 ver architecture to maximize learning efficiency.
14

---

Table 4: (Exact Match) Accuracy (%) Support Accuracy (%) of the base transformer model in predicting Gröbner bases for multivariate polynomial systems over Ey. Each cell reports accuracy over 1,000 test samples, after training with I million samples per training setting (values shown in red: (n, *p) E((2, 1), (3, 0.5, 1), (4, 0.5, 1), (5, 0.33, 0.67, 1), (6. 0.33, 0.67, 1), (7, 0.25, 0.5, 0.75, 1)}. Non-colored cells correspond to interpolated evaluations. n is the number of variables, P the density of the randomly generated system (backward generation method). The curriculum learning approach enables scaling to larger and denser systems compared to prior work.
P (%)
2
3
4
5
6
7
57. 17.1/63.0
65.0/71.2
53.7/58.7
59.0/63.7
63.5/69.0
50.7/56.3
25
55.6/61.3
65.5/72.4
61.5/67.2
59.9/65.4
63.4/69.4
46.7/51.2
33
57.0/62.3
64.4/71.7
59.1/63.6
62.3/68.4
62.7/67.9
50.3/56.1
50
54.0/60.1
62.0/69.7
57.6/63.8
50.5/56.5
58.4/62.7
43.0/49.3
67
56.0/61.7
62.4/69.1
51.5/58.3
55.3/61.0
59.4/66.1
38.9/46.7
75
52.4/58.3
58.8/66.0
55.0/61.1
56.9/63.9
58.1/66.9
42.3/50.9
100
51.3/57.5
55.4/63.9
53.0/60.6
50.4/56.6
53.7/61.9
42.3/51.3
Training Loss
Evaluation Loss
2 10-
base 2024
bose 8/8 512
base 8.8 512
buse 5/6 512
beust 6.76 512
10
Trainiso
Evaluation
6x 10-
4 10-
Tralning Stage
Training Steps
Token Accuracy (%)
Sequence Accuracy (%)
70
Tokenh Accutnec 650
07
Sequance Acouracy
85
80
94
381 1524
-30
base 1024
1024
1024
92
512
20
base
1080
2000
4200
5000
6000
1.000
2000
3000
4000
5080
6000
Traising Staga 50)
Training Steps Ox 50)
Figure 3: Training base models with varying embedding dimensions and number of layers for 48 hours on multivariate datasets of 3 variables over finite field F7 and density p 0.5. Each configuration is run with 3 seeds and the plotted lines are smoothened averages. Hyperparameters: batch size- = 32, learning rate 10- with 1000 warm up steps. Training on V1000 per experiment for 48 hours.
C.2 EXTENDED COMPARISON WITH THE BASE MODEL FROM KERA ET AL. 2024
We present here more results on the comparison (section 4. of HATSolves with the base model from from/Kera al. (2024). We use the same model size and dataset budget in section 4.1 but train the models on larger instances, namely the configurations with n = 7 variables and density p = 0.25 and with = 10 variables and p 0.1. As reminder, the model has 6/6 encoder/decoder layers
15

---

with d = 1024 embedding dimensions. The training was performed on 8x VX00 GPUs for 72 hours. The training metrics are shown in figs. 4 25 which confirm our findings in section 4.1 that HATSolver learns much faster and is computationally more efficient; in fig. the HATSolver-3 performed more than 400K training steps during the 72h training period while the baseline model only did less than 120K training steps during the same period. More importantly, the accuracy of the base model is stuck at 0% while HATSolver-2 2 and HATSolver-3 3 achieve 9% and 8% accuracies respectively.
Training Loss
Evaluation Loss
Saseline
Busoline ohre- H675obee-2
3 3
MATSebowr
2x10
10-4
Training
Evaltaaton
20"2
6k10
Training Stegs (x
Training Staps
Token Accuracy (%)
Sequence Accuracy
Bescline
Buscline
HATSSohrse33
HATSoheee
HATS 2
Tiken Anturacy (16
Sequence Accuracy
155
94
10
2000
4000
0000
8000
10000
12000
14000
2000
4000
6000
8000
10000
12000
14030
Training Stepa OX 50)
Trilning Stops (x 50)
Figure 4: Training dynamics comparison; training models for 72 hours multivariate datasets of n variables and density p = 0.25
C.3 COMPARING WITH HI- TRANSFORMMMRR WU ET AL. 2021
We implement a version of the Hi- Hi-TTranfformer introduced by Wu et al. (2021) originally designed for long document modeling. Figure 6 shows the architecture of this model. The hierarchy two levels deep, where local transformer processes each equation separately, then the first token of each equation chosen as representation of the whole equation and passed to the system transformer. The output tokens of the system transformer are put back into their positions within equations. To propagate the global information into local tokens, another series of local transformers is applied to each equation separately.
We ran two experiments over the same datasets from Kera et al. (2024). The first one is over systems of n. 5 variables and density p 0.67. Another is with n 13 variables and density over F,. The Hi- Hi-Transformer show promessing results over the first dataset where achieved 75% exact match accuracy although at slower learning pace. For the n 13 case, the model never learned to predict a correct groebner basis. Further more, it's much slower than our 3-level hierarchy HATSol 1 v-r both in compute efficiency and learning speed.
0.4
16

---

Training
Evaluation
Taoxiin 2EAT Solvec
20AT Sofvuce
30-1
Twiming Less
Evalution Looa
Staelinee
10-1
2000
6000
5000
8000
2000
4000
5000
8000
Training Steps (x 50)
Training Steps 50)
Token Accuracy (%)
Sequence Accuracy (%)
90.5
Beswine
lhwwinne
EATSolver 2
RATSeiver
$5.0
KAlSoörer 3
HM4 Subver-3 3
Sequunore Accusrer (99)
Tokuin Accuracy (80
04.5
04.0
93.5
93.0
92.5
2000
6000
5000
8000
2000
4000
5000
8000
Training Steps (x 501
Training Steps 50)
Figure 5: Training dynamics comparison; training models for 72 hours multivariate datasets of n 10 variables and density = 0.1
C.4 TRAINING HATSolver ON OTHER FINITE FIELDS
While our main experiments focus on the prime field F7, we investigate the generalizability of HATSolvex to other finite fields, including both prime and non-prime fields. We train our model on polynomial systems over F17 (a prime field) and Fz = Fe (a non-prime extension field).
For this experiment, we systems with n 7 variables and evaluate performance across different polynomial densities e {0.2, 0.4, 0.6, 0.8, 1.0]. The model architecture consists of 2 encoder layers and 6 decoder layers with embedding dimension single A100 GPU for 24 hours per field, using the same curriculum learning strategy described in appendix B
1024. Training performed on
Results are presented in table 5 We observe that HATSolver successfully learns to compute Gröbner bases over both prime and non-prime finite fields, achieving comparable performance across both field types. The model reaches approximately 40% exact match accuracy on low-density systems (p 0.2) for both fields, demonstrating that the hierarchical attention mechanism general- izes effectively across different field characteristics.
Performance degrades gracefully as density increases, dropping to approximately 27-28% accuracy at full density (p- 1.0). This trend is consistent across both fields, and probably due to the fact that we stopped training before one epoch using curriculum which assigned most of the sampling prob- ability to the sparsest data bucket (p 0.2). The similar performance on F (characteristic 2) and F17 (characteristic 17) indicates that HATSolva ver's learned representations capture fundamental algebraic structure that transcends specific field properties.
17

---

Hi- Ti-Trranffomer for Multivariate Polynomial Systems
N
Equation
Equation Transformer
Equation Transformer
2N X
System
P3
PM
N X
Equation Transformer
Equation Transformer
Equation Transformer
Syeiel & Pas Frsh
Syrival Pas Fesh
Gaep
Koep?
Equation
Equation 2
Equation M
Figure 6: Architectural adaptation of the Hi- Hi-Transformer Wu et al. (2021) for multivariate polyno- mial equation systems. We modified the original layer Eeerinng-riginnlllyy a repeated sequence of (Sentence, Document, Sentence)-to grouped configuration of N Equation layers, followed by 2N System layers, and finally Equation layers. This modification ensures that an equal network depth is allocated to both system- wide and cquation-level information processing.
Table 5: Exact Match Accuracy (%) of our model on Gröbner basis prediction multivariate polynomial systems with n 7 variables across different densities over the prime and non-prime finite fields F17 and F2.. Training is done on A100 GPU per field for 24 hours. Model parameters 2 encoder layers, decoder layers, 1024 embedding dimensions.
Field
Density (%)
20
40
60
80
100
IF 16
40.50
35.74
30.09
28.31
28.07
Fi7
39.10
38.94
31.34
28.78
26.99
D DATASETS
D.I DATA GENERATION CONFIG
Table 6 lists the parameters used to generate the datasets using the script from Kera et al. (0244) We use the same configuration for fair comparison.
18

---

Hyperparameter
Value
Explanation
degree_sane sampllig
(empty)
Method for sampling polynomial degrees (not specified) Proportion of nonzero terms in generated polynomials Finite field used for computations (Galois Field of order Type of Gröbner basis generated (e.g., shape position) Maximum degree for polynomials matrices U1. U1.U2
density
field
GF7
gb_type
shape
max_degree_
max_degree_G
5
Maximum degree for polynomials in set G
max num terms F
Maximum number of polynomial terms in matrices U1, U2
www.muutmmemmm..mmmmmmm max wwm terms G
5
Maximum number of terms in polynomials in set G Maximum size (number polynomials) Number of duplicate samples generated Number of samples the test set Number of samples in the training set
max rax_size_F num .uppliicss num uum_samples.cce rest
F
1000
sampsammpes.trr .rrain
1,000,000
num var term sampling
Number of variables in the system
uniform
Method for sampling the number of polynomial terms (uniform over [1, max max_mmmmmmmmmmmm pat___ummmeeet juum_trms_{{E G)).
Table 6: Data Generation Configuration and explanations.
D.2 STATISTICS OF THE GENERATED DATA
statistics about the generated datasets for 13 variables finite field F,. Notably, we report the max lengthh (max sequence length at level 1) which corresponds to the maximum number of terms per equation within polynomial system (we report the max, mean, and std of this statistics over our dataset) for HATSo PorSolver-3 When using HATSolves instead, max lenghh the total number of terms in system which also report.
Table 7 provides
Density
Max # Monoms in Sys. std
Total # Monoms in Sys. std
Max Degree max
max
mean
mean
0.1
304.0
112.3
33.0
795.0
252.2
11.0
0.2
422.0
167.6
47.8
1188.0
371.6
11.0
0.3
568.0
222.7
61.6
1581.7
484.2
11.0
0.4
658.0
277.4
75.0
1973.1
593.0
11.0
0.5
763.0
331.9
87.7
2361.9
696.2
11.0
0.6
899.0
386.3
100.0
2752.4
798.5
11.0
0.7
987.0
440.9
112.6
3143.9
901.3
11.0
0.8
1099.0
495.0
125.1
3531.2
1002.9
11.0
0.9
1215.0
549.0
137.4
3918.4
1104.0
11.0
1.0
1269.0
602.9
149.3
4305.4
1203.2
11.0
Table 7: Statistics about the 13-variable datasets over Fy. Max # Monoms in Sys. is the maximum number of monomials per equation in each system, for which we report max/meea/sst over the whole dataset. This is the metric cap to 400 for training. This means that for p 0.4 for example, we trained on most of the dataset as mean+std = 277 + 75 < 400 while for 1. we considered less than 10% of the dataset mean std > 400. We also report the total number of monomials in each system Total # Monoms in Sys. which would be relevant for HATSolver-2 which considers this entity instead.
D.3 COMPARISON OF BACKWARD GENERATION AND FORWARD GENERATION DATASETS
We represent each polynomial vector in 364-dimensional space, where each dimension corre- sponds to a monomial up to degree 11. This embedding enables quantitative analysis of the intrinsic complexity and structure of the polynomial systems.
19

---

Forward Generation
Backward Generation
Metric
Mean
Std
Median
Mean
Std
Median
Sequence Length
237.51
109.62
229.00
441.69
236.62
399.00
Max Equation Size
27.36
12.06
27.00
34.03
13.96
32.00
Terms per Equation
15.90
12.85
14.00
22.07
14.24
19.00
Term Degree
3.81
2.32
3.00
4.89
2.29
5.00
Equations per Sample
3.00
0.00
3.00
4.01
0.81
4.00
Table 8: Statistical Comparison of Forward and Backward Generation Datasets for = 3 over 17,
D.3.1 MATRIX RANK ANALYSIS
The matrix rank provides insight into the linear independence and intrinsic dimensionality of the polynomial embeddings. Forward generation achieves a rank of 364, while backward generation achieves 216, indicating that forward generation spans the full embedding space while backward generation operates more constrained subspace.
D.3.2 PRINCIPAL COMPONENT ANALYSIS
Table presents the results of our PCA analysis. The number of components required explain different levels of variance differs between the two methodologies, suggesting distinct underlying structures in the generated polynomial systems.
Table 9: Intrinsic Dimensionality Analysis
Metric
Forward Generation
Backward Generation
Matrix Rank
364
216
90%
Variance (dims)
115
104
95%
Variance (dims)
163
128
99%
Variance (dims)
266
173
Sparsity Ratio
0.956
0.939
Avg Non-zero Elements
15.8
22.1
D.3.3 TRAINING RESULTS
Table 10: BART Model Performance Gröbner Basis Prediction over Ff
n
Density (p)
Method
Token Acc.
Acc.
Support Acc.
Samples
2
1.0
BG
96.67%
62.40%
69.90%
1000
3
0.5
BG
98.50%
67.38%
76.
987
3
1.0
BG
98.50%
67.00%
78.74%
903
3
FG
88.04%
0.10%
35.80%
1000
E CLASSICAL ALGORITHM RUNTIME
To verify the computational complexity claims presented in the Kera al. (024)) we conducted an independent timing analysis of the STD-F FLM algorithm using 1000 traals per configuration for polynomial systems with perimental setup utilized Intel(R) Xeon(R) Gold 6230 CPU @ 2. 10GGHz (80 cores) and used the algo- rithm via SageMath's 's interface to Singular (ideal.groebin) The results reveal substantial discrepancies with the published benchmarks, with our measure- ments consistently faster than the reported values across all tested configurations. Most notably, for p 0.2 variables, mean execution time of 0.008 0.028 seconds differs significantly from the claimed 7.46 seconds. The observed timing variations suggest either different experimental conditions, alternative algorithm implementations outlier effect the mean.
to 5 variables using the same backward generated datasets. Our ex-
')).
20

---

n
Density
Matched
Mean Time Std (s)
Median (8)
Max (8)
Paper Claim Mean (s)
2
1.0
100%
0.005 = 0.010
0.002
0.052
8.02
3
0.6
100%
0.005 0.010
0.002
0.050
7.50
4
0.3
100%
0.006 = 0.010
0.003
0.054
7.25
5
0.2
100%
0.008 = 0.028
0.004
0.755
7.46
Table 11: Comparison of STD-FGGII algorithm execution times between reported literature values in Kera et al. (024) and experimental verification. Our results show mean + standard deviation and median execution times across 1000 trials per configuration (n 2 variables, finite field F7) using 'lbssingular:sdfiglmm implementation via SageMath. Matched column indicate that the found groebner basis matches the one used in backward generation as sanity check metric. Our measurements (in seconds) are consistently faster than reported values. All trials were conducted on Intel(R) XR Gold 6230 CPU 2.10GHz (80 cores).
F TRAINING HYPERPARAMETERS
Hyperparameter
Value
Task
multivariat--crricllum
Model
hatsolver.3
Eval samples
1000
Train samples
1,000,000
Num train epochs
15
Optimizer
linear warmup, 1r=0.00001, 1rmmmuppuddatss11000 weight warrm..aaaaarpppp.
Num encoder heads
Timescale
40
Clip grad norm
1.0
Workers
4
Dtype
float16
Max sequence length
15,400,15
Max output sequence length
900
Num variables
13
Density
-1 # all available densities
Train batch size
Val batch size
Field
GF7
Curriculum scheduler ramp
Curriculum scheduler sigma Curriculum min num variables Max coefficient
13
100
Max degree
20
Auto find batch size
False
Dim expansion per level
Top down cross attn
True
Positional encoding combination
concat # versus sum
Pad to max length
True
Num encoder layers
4
Num decoder layers
Encoder embedding dim
1408
Table 12: Hyperparameters used in our table training experiment.
21


Reviews:
[{"review_id": "l2MMN6udJa", "reviewer": "Reviewer_7Nix", "summary": "This paper focuses on the recent line of work on learning to compute Gröbner bases with Transformers and proposes to replace flat attention with a hierarchical attention transformer (HAT) that is tailored to the tree shape structure of multivariate polynomial equations. The starting point is the limitation in Kera et al. that the usual Transformers could handle only up to about 5 variables since, in part, the tokenized sequences become extremely long and flat attention is quadratic in sequence length.  The authors propose a HAT layer that is based on hierarchal attention, similar in spirit to U-net based architectures.  The idea is a 2 phase computation where first attention is computed at each layer, pooling to reduce the dimensionality, and in the second phase, information is propagated in a top-down manner via cross addition or residual adds.", "questions": "(See above)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The motivation compared to Kera et al. is well stated.  The example given to illustrate how their hierarchical model works e.g., in fig 1 is pretty clear and the authors convincingly show that the hierarchical formulation is natural for this data.  They do go beyond the earlier baseline quoted from Kera et al. They claim to reach 13 variables, degree 11, whereas Kera et al. stopped earlier. If those numbers are correct and all other factors are equal, then this is a sizable improvement.  \n\nAt the same time I should note that I am not a domain expert in learned Gröbner basis computation, so I cannot fully judge how substantial this improvement is from the perspective of the Gröbner basis community, or whether this is mainly a better serialization and positional scheme applied to the same underlying idea.", "weaknesses": "The main part of the argument in Section 3 of this paper is that the problem is really the explosion in sequence length and that we should reflect the inherent tree structure in the model. This is reasonable, but it is also very close to what people have been doing with structured tokenization and with position functions that respect the underlying structure in the data, for example with hard ALiBi-like schemes for organized inputs (see eg [1]). The paper does not make it clear why this must be a new hierarchical attention block rather than a careful tokenization and positional encoding strategy that bakes in this inductive bias from the outset.\n\nSecond, I am not clear about which part is doing the heavy lifting for the improved performance over Kera et al.  As far as I can tell, there isn't any ablations reported in the paper of whether the performance boost is due to the hierarchical attention, more careful curriculum, different serialization, or training the model longer.\n\nGiven how hierarchical the data is, I would have expected at least one baseline that uses grouping of tokens into fixed length blocks with attention only inside blocks plus a cross block layer. Right now the paper is arguing that the custom HAT is needed without showing that a simpler approach fails. This makes the contribution somewhat difficult to judge.  However, I am willing to defer to other reviewers on this point.\n\nFinally, some of the writing is quite informal, particularly in the introduction (such as the first paragraph).  I did not penalize the authors for this.\n\n[1] https://arxiv.org/abs/2402.01032", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YLC9ogTJdq", "reviewer": "Reviewer_YTxd", "summary": "This study presents a new attention-based architecture for computing polynomial systems, particularly Gröbner bases. The prior study, Kera et al. (2024) used a standard architecture and suffered from prohibitively long input sequences and attention memory cost for large polynomial systems. The proposed study introduced a hierarchical attention mechanism that applies attention within terms, polynomials, and systems, accordingly. The experiments show a drastic training cost reduction and extension to systems with a larger number of variables than in the prior study.", "questions": "Besides the comments in the Weaknesses, I'd like the authors to clarify several points. \n- **Dimension design.** [l.263-265] suggests several designs of $(d_i)_i$. Are there any ablation studies and insights into this point?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- A clear focus on scaling up Transformer models for polynomial system computation. \n- New hierarchical attention module, which is particularly advantageous for the case where input sequence length surpasses the embedding dimension. \n- Experimental results show Transformer models with the proposed attention module scale up the problem size to the size where mathematical algorithms require long computation, which was not the case with the prior work.", "weaknesses": "I generally appreciate the achievement of this work - scaling up the problem size of learning-based Gröbner basis computation. The major weaknesses are two-fold: separation from prior studies and clarity/depth of experiments. I suppose that these are manageable weaknesses and expect the rebuttal process will resolve this. \n\n---\n**Vs. Prior Studies** \n\nWhile the proposed concept is clear and aligns with processing polynomial systems, hierarchical attention mechanism has been used in a wide range of applications, as the authors summarize in the Related Work section. It is unclear from the authors' description whether the proposed module is technically novel. For example, MEGABYTE [1], SpaceByte [2], and Block Transformer [3] are general hierarchical attention models. None of them are cited in the paper. \n\nI'd like the authors to present the technical novelty of the proposed method over such works. I understand that the focus of this work is more specialized for polynomial systems. In this context, the proposed method can be regarded as a generalization of monomial embedding [4]. The main text should clarify this as well. \n\n[1] Yu et al., \"MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers,\" NeurIPS'23\n\n[2] Sagel, \"SpaceByte: Towards deleting tokenization from large language modeling,\" NeurIPS'24\n\n[3] Ho et al., \"Block transformer: Global-to-local language modeling for fast inference,\" arXiv'24\n\n[4] Kera & Pelleriti et al., \"Computational Algebra with Attention: Transformer Oracles for Border Basis Algorithms,\" arXiv'24\n\n---\n**Unclear Experimental Setup**\n\nThe training setup is somewhat unclear to me. \n- In Section 4.1, why was the 4/4 model used rather than the 6/6 one as in the prior study? \n- According to the paper, the curriculum learning used 1M samples for each ($n$, $\\rho$) combination. The range and step size of $\\rho$ are not provided, but the number of training samples seems significant. \n- The caption of Table 1 mentions the out-of-distribution evaluations. However, no detailed description can be found. Why do colored texts suggest out-of-distribution? Is that because the corresponding values of $\\rho$ were not used in training? Again, the range and step size of $\\rho$ are not clarified, but I suspect the \"out-of-distribution\" $\\rho$ here is still interpolation rather than extrapolation. \n- I feel the same metric concept is referred to by different names, which made me confused in reading. Specifically, what is the difference between \"Accuracy\" in Table 1, \"Sequence Accuracy\" in Figure 2, and \"Success\" in Table 2? \n\n---\n\n**Experiment Depth**\n\nI suggest a few more experiments that might strengthen the value of the proposed model further. \n- [4] proposes a new backward transform method that is more general than the one used in the experiments (adapted from Kera et al.'24). It would also be interesting to evaluate the impact on this setup. \n- The visualization of the embedding space of terms, polynomials, and systems would be valuable. For example, are \"similar\" systems (e.g., having the same leading term set, or Gröbner basis) mapped close together? \n- The experiments target polynomial system solving, so the experiments only tested HAT-2 and HAT-3. Setting up an artificial problem/dataset and testing HAT-n with n > 3 could demonstrate the broader utility of this work.  \n\n---\n\nI feel the potential practical impact of this work is great, but the machine learning (and algebraic) technical novelty seems limited. My score is temporal, and if the authors can address particularly the first concern (**Vs. Prior Studies**), I'm happy to increase the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rAAcHNEuHE", "reviewer": "Reviewer_mXdc", "summary": "The paper proposes HATSolver, that uses a transformer with hierarchical attention mechanism to capture the hierarchy of he natural tree structure of polynomial systems (tokens  $\\rightarrow$ terms $\\rightarrow$ polynomials $\\rightarrow$ system). The model performs a bottom-up local self-attention followed by a top-down refinement via cross-level attention, and the authors provide a complexity analysis showing how hierarchical factorization reduces the quadratic attention bottleneck as lengths scale. The approach is trained on “backward-generated” datasets that pair non-Gröbner input systems with their reduced lexicographic Gröbner bases in shape position. The paper further adds a curriculum learning schedule to improve scalability. Empirically, the authors show that curriculum learning alone scales the baseline transformer beyond the n≤5 limitation, and demonstrate that their model converge faster and reach higher accuracy than the baseline under equal training budgets.", "questions": "1. Questions discussed in Weaknesses.  \n2. In Table 2, it’s unclear to me if those numbers are per-instance, per-batch, or total, and for how many instances. Please clarify them.\n3. What pooling function you ultimately used (mean? learned? top-k?)\n4. Table 4 reports observed max degree 11, while Table 6 lists “Max degree 20” for the training hyperparameters. Clarify the degree cutoffs effective during training vs. evaluation. \n5. Please add multiple seeds runs error bars to the reported numbers.\n6. Since HATSolver-3 suffers from heavy padding, did you try bucketing by length to mitigate doubled token counts? If so, what was the effect on throughput/accuracy?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I think adapting hierarchical attention to polynomial systems is a thoughtful inductive bias. Moreover, the results are fairly explorative and include a larger-scale experiment at $n=13$ with a broad density sweep and baselines. On the clarity side, I can say that the two-phase hierarchical mechanism is described clearly with consistent notation. I also believe that scaling and even a partial success in neural Gröbner basis (GB) computation from n≤5 to n=13 is a meaningful step, especially since GB computation is a backbone in many symbolic tasks.", "weaknesses": "1. I think the actual success in this task is the exact string equality to the ground-truth reduced lex basis from backward generation. While reduced lex GB is unique, the paper does not report algebraic verification (e.g., reduce every ($f\\in F$) by predicted ($\\widehat{G}$), check ($\\mathrm{lt}(\\langle\\widehat{G}\\rangle)=\\mathrm{lt}(I)$), minimality/monicity) as a post-hoc correctness test. The study grades ``exact string equality'', near-misses caused by non-monic scaling, ordering, or tails not fully reduced will be counted as failures, even if the ideal and leading-term ideal are correct. A CAS pass would quantify how many such string failures are algebraically valid solutions. Please add a CAS-based verifier in evaluation. \n2. I think the test set is somewhat biased. The scope is restricted to shape position, 0-dimensional radical ideals over ($\\mathbb{F}_7$). That matches prior work, but the actual algorithmic alignment requires tests on other fields (incl. non-prime), other monomial orders, and non-shape zero-dimensional ideals. The paper acknowledges this as future work; however, even small OOD tests ($F(5)/F({11}$), random orderings, non-shape samples) would not only strengthen claims, but also would calibrate sensitivity to the data generator and support the beyond ($\\mathbb{F}_7$) claim. \n3. Since PoSSo and Gröbner basis computation are NP-hard in the worst case, it would strengthen the paper’s soundness and positioning to discuss connections to Neural Algorithmic Reasoning as a neural alignment work (see below). What I really want to see is a comprehensive discussion on what algorithmic primitives (e.g., S-polynomial selection/reduction, Buchberger’s criteria, elimination structure) does your architecture implicitly learn, and how does the hierarchical attention help align with those primitives? A brief discussion would clarify whether HATSolver is merely pattern-matching the training distribution or actually capturing algorithmic invariants that might transfer OOD (e.g., to different fields/orders/densities). Relevant references are:\n\n- Veličković et al. (ICLR 2020), Neural execution of graph algorithms.\n- Georgiev et al. (LoG 2023), Neural algorithmic reasoning for combinatorial optimisation.\n- Dudzik et al. (LoG 2024), Asynchronous algorithmic alignment with cocycles.\n- Hashemi et al. (NeurIPS 2025), Tropical Attention: Neural Algorithmic Reasoning for Combinatorial Algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)