Title: Learning to compute Gröbner bases
Venue: NeurIPS 2024
Accepted: yes
Keywords: Transformer; Gröbner bases; Computational algebra
Abstract: Solving a polynomial system, or computing an associated Gröbner basis, has been a fundamental task in computational algebra. However, it is also known for its notorious doubly exponential time complexity in the number of variables in the worst case. This paper is the first to address the learning of Gröbner basis computation with Transformers. The training requires many pairs of a polynomial system and the associated Gröbner basis, raising two novel algebraic problems: random generation of Gröbner bases and transforming them into non-Gröbner ones, termed as backward Gröbner problem. We resolve these problems with 0-dimensional radical ideals, the ideals appearing in various applications. Further, we propose a hybrid input embedding to handle coefficient tokens with continuity bias and avoid the growth of the vocabulary set. The experiments show that our dataset generation method is a few orders of magnitude faster than a naive approach, overcoming a crucial challenge in learning to compute Gröbner bases, and Gröbner computation is learnable in a particular class.

Learning to compute Gröbner bases

Abstract

Solving a polynomial system, or computing an associated Gröbner basis, has been a fundamental task in computational algebra. However, it is also known for its notorious doubly exponential time complexity in the number of variables in the worst case. This paper is the first to address the learning of Gröbner basis computation with Transformers. The training requires many pairs of a polynomial system and the associated Gröbner basis, raising two novel algebraic problems: random generation of Gröbner bases and transforming them into non-Gröbner ones, termed as backward Gröbner problem. We resolve these problems with 0-dimensional radical ideals, the ideals appearing in various applications. Further, we propose a hybrid input embedding to handle coefficient tokens with continuity bias and avoid the growth of the vocabulary set. The experiments show that our dataset generation method is a few orders of magnitude faster than a naive approach, overcoming a crucial challenge in learning to compute Gröbner bases, and Gröbner computation is learnable in a particular class.

1 Introduction

Understanding the properties of polynomial systems and solving them have been a fundamental problem in computational algebra and algebraic geometry with vast applications in cryptography [8, 93], control theory [71], statistics [27, 41], computer vision [78], systems biology [60], and so forth. Special sets of polynomials called Gröbner bases [16] play a key role to this end. In linear algebra, the Gaussian elimination simplifies or solves a system of linear equations by transforming its coefficient matrix into the reduced row echelon form. Similarly, a Gröbner basis can be regarded as a reduced form of a given polynomial system, and its computation is a generalization of the Gaussian elimination to general polynomial systems. However, computing a Gröbner basis is known for its notoriously bad computational cost in theory and practice. It is an NP-hard problem with the doubly exponential worst-case time complexity in the number of variables [29, 67]. Nevertheless, because of its importance, various algorithms have been proposed in computational algebra to obtain Gröbner bases in better runtime. Examples include Faugère's F4/F5 algorithms [33, 34] and M4GB [66].

In this study, we investigate Gröbner basis computation from a learning perspective, envisioning it as a practical compromise to address large-scale polynomial system solving and understanding when mathematical algorithms are computationally intractable. The learning approach does not require explicit design of computational procedures, and we only need to train a model using a large amount of (non-Gröbner set, Gröbner basis) pairs. Further, if we restrict ourselves to a particular class of Gröbner bases (or associated ideals ), the model may internally find some patterns useful for prediction. The success of learning indicates the existence of such patterns, which encourages the improvement of mathematical algorithms and heuristics. Several recent studies have already addressed mathematical tasks via learning, particularly using Transformers [14, 19, 58]. For example, [58] showed that Transformers can learn symbolic integration simply by observing many (d f /d x , f ) pairs in training. The training samples are generated by first randomly generating f and computing its derivative d f /d x and/or by the reverse process.

However, a crucial challenge in the learning of Gröbner basis computation is that it is mathematically unknown how to efficiently generate many (non-Gröbner set, Gröbner basis) pairs. We need an efficient backward approach (i.e., solution-to-problem computation) because, as discussed above, the forward approach (i.e., problem-to-solution computation) is prohibitively expensive. To this end, we frame two problems: (i) a random generation of Gröbner bases and (ii) a backward transformation from a Gröbner basis to an associated non-Gröbner set. To our knowledge, neither of them has been addressed in the study of Gröbner bases because of the lack of motivations; all the efforts have been dedicated to the forward computation from a non-Gröbner set to Gröbner basis.

Another challenge in the learning approach using Transformers lies in the tokenization of polynomials on infinite fields, such as R and Q . To cover a wide range of coefficients, one has to either prepare numerous number tokens or split a number into digits. The former requires a large embedding matrix, and the latter incurs large attention matrices due to the lengthy sequences. To resolve this, we introduce a continuous embedding scheme, which embeds coefficient tokens by a small network and avoids the tradeoff between vocabulary size and sequence length. The continuity of the function realized by the network naturally implements the continuity of the numbers in the embedding.

We summarize the contributions as follows.

We investigate the first learning approach to the Gröbner computation using Transformers and experimentally show its learnability. Unlike most prior studies, our results indicate that training a Transformer may be a compromise to NP-hard problems to which no efficient (even approximate or probabilistic) algorithms have been designed.

We uncovered two unexplored algebraic problems-random generation of Gröbner bases and backward Gröbner problem and propose efficient methods to address them in the 0dimensional case. The problems are essential to the learning approach but also algebraically interesting and need interaction between computational algebra and machine learning.

We propose a new input embedding to efficiently handle a large range of coefficients without the tradeoff between the size of the embedding matrix and attention maps.

Our experiments show that the proposed dataset generation is highly efficient and faster than a baseline method by a few orders of magnitude. Further, we observe a learnability gap between polynomials on finite fields and infinite fields while predicting polynomial supports are more tractable.

2 Related Work

Gröbner basis computation. Gröbner basis is one of the fundamental concepts in algebraic geometry and commutative ring theory [24, 39]. By its computational aspect, Gröbner basis is a very useful tool for analyzing the mathematical structures of solutions of algebraic constraints. Notably, the form of Gröbner bases is suited for finding solutions and allows parametric coefficients, and thus, it is vital to make Gröbner basis computation efficient and practical in applications. Following the definition of Gröbner bases in [16], the original algorithm to compute them can be presented as (i) create potential new leading terms by constructing S-polynomials , (ii) reduce them either to zero or to new polynomials for the Gröbner basis, and (iii) repeat until no new S-polynomials can be constructed. Plenty of work has been developed to surpass this algorithm. There are four main strategies: (a) avoiding unnecessary S-polynomials based on the F5 algorithm and the more general signature-based algorithms [9, 34]. Machine learning appeared for this task in [73]. (b) More efficient

reduction using efficient linear algebraic computations using [33] and the very recent GPU-using [12]. (c) Performing modular computations , following [6, 70], to prevent coefficient growth during the computation. (d) Using the structure of the ideal, e.g., [13, 35] for change of term ordering for 0-dimensional ideals or [83] when the Hilbert function is known. In this study, we present the fifth strategy: (e) Gröbner basis computation fully via learning without specifying any mathematical procedures.

Transformers for mathematics. Recent studies have revealed that Transformers can be used for mathematical reasoning and symbolic computation. The training only requires samples (i.e., problem-solution pairs), and no explicit mathematical procedures need to be specified. In [58], the first study that uses Transformers for mathematical problems is presented. It showed that Transformers can learn symbolic integration and differential equation solving with training with sufficiently many and diverse samples. Since then, Transformers have been applied to checking local stability and controllability of differential equations [22], polynomial simplification [3], linear algebra [19, 20], symbolic regression [14, 26, 44, 45], Lyapunov function design [4] and attacking the LWE cryptography [61, 89]. In [75], comprehensive experiments over various mathematical tasks are provided. In contrast to the aforementioned studies, we found that Gröbner basis computation, an NP-hard problem, even has an algebraic challenge in the dataset generation. This paper introduces unexplored algebraic problems and provides an efficient algorithm for a special but important case (i.e., 0-dimensional ideals), thereby realizing an experimental validation of the learning of Gröbner basis computation.

3 Notations and Definitions

We introduce the necessary notations and definitions in this Section. The reader interested in a gentle introduction to Gröbner basis theory can refer to the classical book [25]. For their comfort, we have moreover compiled most elementary additional definitions and notations in App. A.

We consider a polynomial ring k [ x 1 , . . . , x n ] with a field k and variables x 1 , . . . , x n . For a set F ⊂ k [ x 1 , . . . , x n ] , the ideal generated by F is denoted by ⟨ F ⟩ . Once a term order on the terms of k [ x 1 , . . . , x n ] is fixed, one can define leading terms and Gröbner bases.

Definition 3.1 (Leading term) . Let F = { f 1 , . . . , f s } ⊂ k [ x 1 , . . . , x n ] and let ≺ be a term order. The leading term LT( f i ) of f i is the largest term in f i in ordering ≺ . The leading term set of F is LT( F ) = { LT( f 1 ) , . . . , LT( f s ) } .

Definition 3.2 (Gröbner basis) . Fix a term order ≺ . A finite subset G of an ideal I is said to be a ≺ -Gröbner basis of I if ⟨ LT( G ) ⟩ = ⟨ LT( I ) ⟩ .

The condition ⟨ LT( G ) ⟩ = ⟨ LT( I ) ⟩ means that for any element h ∈ I , the leading term LT( h ) is divided by the leading term LT( g ) of an element g ∈ G . It gives a complete test whether a given polynomial h is in I or not by polynomial division with G , similar to Gaussian elimination by a basis of a vector space. The remainder is 0 means that h ∈ I , and otherwise means that h ̸∈ I . This is related to finding solutions. Roughly speaking, if h ∈ I = ⟨ f 1 , . . . , f s ⟩ , then we have a form h = ∑ s i =1 h i f i and the system f 1 ( x 1 , . . . , x n ) = · · · = f s ( x 1 , . . . , x n ) = 0 shares solutions with the equation h ( x 1 , . . . , x n ) = 0 .

̸

Note that ⟨ LT( G ) ⟩ ⊂ ⟨ LT( I ) ⟩ is trivial from G ⊂ I . The nontriviality of the Gröbner basis lies in ⟨ LT( G ) ⟩ ⊃ ⟨ LT( I ) ⟩ ; that is, a finite number of leading terms can generate the leading term of any polynomial in the infinite set I . The Hilbert basis theorem [25] guarantees that every ideal I = { 0 } has a Gröbner basis. Moreover, using the multivariate division algorithm, one gets that any Gröbner basis G of an ideal I generates I . We are particularly interested in the reduced Gröbner basis G of I = ⟨ F ⟩ , which is unique once the term order is fixed.

Intuition of Gröbner bases and system solving. Let G = { g 1 , . . . , g t } be a Gröbner basis of an ideal ⟨ F ⟩ = ⟨ f 1 , . . . , f s ⟩ . The polynomial system g 1 ( x 1 , . . . , x n ) = · · · = g t ( x 1 , . . . , x n ) = 0 is a simplified form of f 1 ( x 1 , . . . , x n ) = · · · = f s ( x 1 , . . . , x n ) = 0 with the same solution set. With the term order ≺ lex , G has a form g 1 ∈ k [ x n 1 , . . . , x n ] , g 2 ∈ k [ x n 2 , . . . , x n ] , . . . , g t ∈ k [ x n t , . . . , x n ] with n 1 ≤ n 2 ≤ . . . ≤ n t , which may be regarded as the 'reduced row echelon form' of a polynomial system. In our particular case (i.e., 0-dimensional ideals in shape position; cf. Sec. 4.2), we have ( n 1 , n 2 , . . . , n t ) = (1 , 2 , . . . , n ) . Thus, one can obtain the solutions of the polynomial system using

a backward substitution, i.e., by first solving a univariate polynomial g t , next solving bivariate polynomial g t -1 , which becomes univariate after substituting the solutions of g t , and so forth.

Other notations. The subset k [ x 1 , . . . , x n ] ≤ d ⊂ k [ x 1 , . . . , x n ] denotes the set of all polynomials of total degree at most d . For a polynomial matrix A ∈ k [ x 1 , . . . , x n ] s × s , its determinant is given by det( A ) ∈ k [ x 1 , . . . , x n ] . The set F p with a prime number p denotes the finite field of order p . The set ST( n, k [ x 1 , . . . , x n ]) denotes the set of upper-triangular matrices with all-one diagonal entries (i.e., unimodular upper-triangular matrices) with entries in k [ x 1 , . . . , x n ] . The total degree of f ∈ k [ x 1 , . . . , x n ] is denoted by deg( f ) .

4 New Algebraic Problems

Our goal is to realize Gröbner basis computation through a machine learning model. To this end, we need a large training set { ( F i , G i ) } i m =1 with finite polynomial set F i ⊂ k [ x 1 , . . . , x n ] and Gröbner basis G i of ⟨ F i ⟩ . As the computation from F i to G i is computationally expensive in general, we instead resort to backward generation (i.e., solution-to-problem process); that is, we generate a Gröbner basis G i randomly and transform it to non-Gröbner set F i .

What makes the learning of Gröbner basis computation hard is that, to our knowledge, neither (i) a random generation of Gröbner basis nor (ii) the backward transform from Gröbner basis to non-Gröbner set has been considered in computational algebra. Its primary interest has been instead posed on Gröbner basis computation (i.e., forward generation), and nothing motivates the random generation of Gröbner basis nor the backward transform. Interestingly, machine learning now sheds light on them. Formally, we address the following problems for dataset generation.

Problem 4.1 (Random generation of Gröbner bases) . Find a collection G = { G i } i m =1 with the reduced Gröbner basis G i ⊂ k [ x 1 , . . . , x n ] of ⟨ G i ⟩ , i = 1 , . . . , m . The collection should contain diverse bases, and we need an efficient algorithm for constructing them.

Problem 4.2 (Backward Gröbner problem) . Given a Gröbner basis G ⊂ k [ x 1 , . . . , x n ] , find a collection F = { F i } µ i =1 of polynomial sets that are not Gröbner bases but ⟨ F i ⟩ = ⟨ G ⟩ for i = 1 , . . . , µ . The collection should contain diverse sets, and we need an efficient algorithm for constructing them.

Problems 4.1 and 4.2 require the collections G , F to contain diverse polynomial sets. Thus, the algorithms for these problems should not be deterministic but should have some controllable randomness. Several studies reported that the distribution of samples in a training set determines the generalization ability of models trained on it [19, 58]. However, the distribution of non-Gröbner sets and Gröbner bases is an unexplored and challenging object of study. It can be another challenging topic and goes beyond the scope of the present study.

4.1 Scope of this study

Non-Gröbner sets have various forms across applications. For example, in cryptography (particularly post-quantum cryptosystems), polynomials are restricted to dense degree-2 polynomials and generated by an encryption scheme [93]. On the other hand, in systems biology (particularly, reconstruction of gene regulatory networks), they are typically assumed to be sparse [59]. In statistics (particularly algebraic statistics), they are restricted to binomials, i.e., polynomials with two monomials [41, 79].

As the first study of Gröbner basis computation using Transformers, we do not focus on a particular application and instead address a generic case reflecting a motivation shared by various applications of computing Gröbner basis: solving polynomial systems or understanding ideals associated with polynomial systems having solutions. Particularly, we focus on 0-dimensional radical ideals, a special but fundamental class of ideals.

Definition 4.3 (0-dimensional ideal) . Let F be a set of polynomials in k [ x 1 , . . . , x n ] . An ideal ⟨ F ⟩ is called a 0-dimensional ideal if all but a finite number of terms belong to LT( ⟨ F ⟩ ) .

In fact, the number of terms not belong to LT( ⟨ f 1 , . . . , f s ⟩ ) is an upper bound of the number of solutions of the system f 1 ( x 1 , . . . , x n ) = · · · = f s ( x 1 , . . . , x n ) = 0 . In particular, the finiteness of the number of terms not belong to LT( ⟨ f 1 , . . . , f s ⟩ ) implies the finiteness of the number of solutions. This is the reason why we call such ideals ' 0 -dimensional' ideals in Def. 4.3.

0 -dimensional ideals are the fundamental ideals in the study of pure algebra. This is partly because of the ease of analysis. As Def. A.5 shows, 0 -dimensional ideals relate to finite-dimensional vector spaces, and thus, analysis and algorithm design can be essentially addressed by matrices and linear algebra.

Also ideals in most practical scenarios are known to be 0-dimensional. For example, a multivariate public-key encrypted communication (a candidate of post-quantum cryptosystems) with a public polynomial system F over a finite field F p will be broken if one finds any root of the system F ∪ { x p 1 -x 1 , . . . , x p n -x n } ) . One should note that the ideal ⟨ F ∪ { x p 1 -x 1 , . . . , x p n -x n }⟩ is 0-dimensional [84, Sec. 2.2]. Generically, 0-dimensional ideals defined from polynomial systems having solutions are radical 2 (i.e., non-radical ideals are in a zero-measure set in the Zariski topology). The proofs of the results in the following sections can be found in App. C. Hereinafter, the sampling of polynomials is done by a uniform sampling of coefficients from a prescribed range.

It is also worth noting that Transformers cannot be an efficient tool for general Gröbner basis computation, and thus, we should focus on a particular class of ideals and pursue in-distribution accuracy. This is evident from the facts that Gröbner basis computation is NP-hard and that machine learning models perform best on in-distribution samples and do not generalize perfectly. Fortunately, unlike standard machine learning tasks (e.g., image classification task), users can frame their problems beforehand (i.e., they know what types of polynomials they want to handle), and they can collect as many training samples as they want if an efficient algorithm exists. As mentioned above, the form of non-Gröbner sets varies across applications, and thus, we focus on the generic case and leave the specialization to future work.

4.2 Random generation of Gröbner bases

We address Prob. 4.1 using the fact that 0-dimensional radical ideals are generally in shape position .

Definition 4.4 (Shape position) . Ideal I ⊂ k [ x 1 , . . . , x n ] is called in shape position if some univariate polynomials h, g 1 , . . . , g n -1 ∈ k [ x n ] form the reduced ≺ lex -Gröbner basis of I as follows.

As can be seen, the ≺ lex -Gröbner basis consists of a univariate polynomial in x n and the difference of univariate polynomials in x n and a leading term x i for i < n . While not all ideals are in shape position, 0-dimensional radical ideals are almost always in shape position: if an ⟨ f 1 , . . . , f s ⟩ ⊂ k [ x 1 , . . . , x n ] is a 0-dimensional and radical ideal, a random coordinate change ( y 1 , . . . , y n ) = ( x 1 , . . . , x n ) R with a regular (i.e., invertible) matrix R ∈ k n yields ˜ f 1 , · · · , ˜ f s ∈ k [ y 1 , . . . , y n ] , and the ideal ⟨ y 1 , . . . , y n ⟩ generally has the reduced ≺ lex -Gröbner basis in the form of Eq. (4.1) (cf. Prop. A.14).

̸

With this fact, an efficient sampling of Gröbner bases of 0-dimensional radical ideals can be realized by sampling n polynomials in k [ x n ] , i.e., h, g 1 , . . . , g n -1 with h = 0 . We have to make sure that the degree of h is always greater than that of g 1 , . . . , g n -1 , which is necessary and sufficient for G to be a reduced Gröbner basis. This approach involves efficiency and randomness, and thus resolving Prob. 4.1. Note that while our approach assumes term order ≺ lex , if necessary, one can use an efficient change-of-ordering algorithm, e.g., the FGLM algorithm [35]. The cost of the FGLM algorithm is O ( n · deg( h ) 3 ) based on the number of arithmetic operations over k . Besides the ideals in shape position, we also consider the Cauchy module in App. B, which defines another class of 0-dimensional ideals.

4.3 Backward Gröbner problem

To address Prob. 4.2, we consider the following problem.

Problem 4.5. Let I ⊂ k [ x 1 , . . . , x n ] be a 0-dimensional ideal, and let G = ( g 1 , . . . , g t ) ⊤ ∈ k [ x 1 , . . . , x n ] t be its ≺ -Gröbner basis with respect to term order ≺ . 3 Find a polynomial matrix A ∈ k [ x 1 , . . . , x n ] s × t giving a non-Gröbner set F = ( f 1 , . . . , f s ) ⊤ = AG such that ⟨ F ⟩ = ⟨ G ⟩ .

Namely, we generate a set of polynomials F = ( f 1 , . . . , f s ) ⊤ from G = ( g 1 , . . . , g t ) ⊤ by f i = ∑ t j =1 a ij g j for i = 1 , . . . , s , where a ij ∈ k [ x 1 , . . . , x n ] denotes the ( i, j ) -th entry of A . Note that ⟨ F ⟩ and ⟨ G ⟩ are generally not identical, and the design of A such that ⟨ F ⟩ = ⟨ G ⟩ is of our question.

A similar question was studied without the Gröbner condition in [17, 18]. They provided an algebraic necessary and sufficient condition for the polynomial system of F to have a solution outside the variety defined by G . This condition is expressed explicitly by multivariate resultants. However, strong additional assumptions are required: A,F,G are homogeneous, G is a regular sequence, and in the end, ⟨ F ⟩ = ⟨ G ⟩ is only satisfied up to saturation. Thus, they are not compatible with our setting and method for Prob. 4.1.

Our analysis gives the following results for the design A to achieve ⟨ F ⟩ = ⟨ G ⟩ for the 0-dimensional case (without radicality or shape position assumption).

Theorem 4.6. Let G = ( g 1 , . . . , g t ) ⊤ be a Gröbner basis of a 0-dimensional ideal in k [ x 1 , . . . , x n ] . Let F = ( f 1 , . . . , f s ) ⊤ = AG with A ∈ k [ x 1 , . . . , x n ] s × t .

If ⟨ F ⟩ = ⟨ G ⟩ , it implies s ≥ n .

If A has a left-inverse in k [ x 1 , . . . , x n ] t × s , ⟨ F ⟩ = ⟨ G ⟩ holds.

The equality ⟨ F ⟩ = ⟨ G ⟩ holds if and only if there exists a matrix B ∈ k [ x 1 , . . . , x n ] t × s such that each row of BA -E t is a syzygy 4 of G , where E t is the identity matrix of size t .

The first statement of Thm. 4.6 argues that polynomial matrix A should have at least n rows. For an ideal in shape position, we have a ≺ lex -Gröbner basis G of size n , and thus, A is a square or tall matrix. The second statement shows a sufficient condition. The third statement provides a necessary and sufficient condition. Using the second statement, we design a simple random transform of a Gröbner basis to a non-Gröbner set without changing the ideal.

We now assume ≺ = ≺ lex and 0-dimensional ideals in shape position. Then, G has exactly n generators. When s = n , we have the following.

Proposition 4.7. For any A ∈ k [ x 1 , . . . , x n ] n × n with det( A ) ∈ k \ { 0 } , we have ⟨ F ⟩ = ⟨ G ⟩ .

As non-zero constant scaling does not change the ideal, we focus on A with det( A ) = ± 1 without loss of generality. Such A can be constructed using the Bruhat decomposition:

where U 1 , U 2 ∈ ST( n, k [ x 1 , . . . , x n ]) are upper-triangular matrices with all-one diagonal entries (i.e., unimodular upper-triangular matrices) and P ∈ { 0 , 1 } n × n denotes a permutation matrix. Noting that A -1 satisfies A -1 A = E n , we have ⟨ AG ⟩ = ⟨ G ⟩ from Thm. 4.6. Therefore, random sampling ( U 1 , U 2 , P ) of unimodular upper-triangular matrices U 1 , U 2 and a permutation matrix P resolves the backward Gröbner problem for s = n .

We extend this idea to the case of s > n using a rectangular unimodular upper-triangular matrix:

where U ′ 2 ∈ ST( n, k [ x 1 , . . . , x n ]) and O s -n,n ∈ k [ x 1 , . . . , x n ] ( s -n ) × n is the zero matrix. The permutation matrix is now P ∈ { 0 , 1 } s × s . Note that U 2 G already gives a non-Gröbner set such that ⟨ U 2 G ⟩ = ⟨ G ⟩ ; however, the polynomials in the last s -n entries of U 2 G are all zero by its construction. To avoid this, the permutation matrix P shuffles the rows and also U 1 to exclude the zero polynomial from the final polynomial set.

To summarize, our strategy is to compute F = U 1 PU 2 G , which only requires a sampling of O ( s 2 ) polynomials in k [ x 1 , . . . , x n ] , and O ( n 2 + s 2 ) -times multiplications of polynomials. Note that even in the large polynomial systems in the MQ challenge, a post-quantum cryptography challenge, we have n < 100 and s < 200 [93].

4.4 Dataset generation algorithm

The combination of the discussion in the previous sections gives an efficient dataset generation algorithm (see Alg. 1 for a pseudocode).

Theorem 4.8. Consider a polynomial ring k [ x 1 , . . . , x n ] . Given the dataset size m , maximum degrees d, d ′ > 0 , maximum size of non-Gröbner set s max ≥ n , and term order ≺ , Alg. 1 returns a collection D = { ( F i , G i ) } i m =1 with the following properties: For all i = 1 , . . . , m ,

| G i | = n and | F i | ≤ s max .

The set G i is the reduced ≺ -Gröbner basis of ⟨ F i ⟩ . The set F i is not, unless G i , U 1 , U ′ 2 , P are all sampled in a non-trivial Zariski closed subset. 5

The ideal ⟨ F i ⟩ = ⟨ G i ⟩ is a 0-dimensional ideal in shape position.

The time complexity is O ( m ( nS 1 ,d + s 2 S n,d ′ +( n 2 + s 2 ) M n, 2 d ′ + d )) when ≺ = ≺ lex , where S n,d denotes the complexity of sampling an n -variate polynomial with total degree at most d , and M n,d denotes that of multiplying two n -variate polynomials with total degree at most d . If ⊀ = ≺ lex , O ( mnd 3 ) is additionally needed.

The proposed dataset generation method is a backward approach, which first generates solutions and then transforms them into problems. In this case, we have control over the complexity of the Gröbner bases and can add some intrinsic structure if any prior information is available. For example, a multi-variate encryption scheme encapsulates secret key information in the solution of a polynomial system with a single solution in a base field [93]. The ideal associated with such a system is 0-dimensional and in shape position; namely, its Gröbner basis has the following form: G = ⟨ x n -a n , x 1 -a 1 , . . . , x n -1 -a n -1 ⟩ , where a 1 , a 2 , . . . , a n are constants [84]. The backward approach allows one to restrict the Gröbner bases in a dataset into such a class.

This is not the case with forward approaches. While they may include prior information into nonGröbner sets, it is computationally expensive to obtain the corresponding Gröbner bases. It is also worth noting that a naive forward approach, which randomly generates non-Gröbner sets and computes their Gröbner bases, should be avoided even if it were computationally tractable because, for example, if the Gröbner basis of such F is generally { 1 } when | F | > n .

5 Hybrid Input Embedding

Transformers are an efficient learner of sequence-to-sequence functions. They receive and generate a sequence of tokens . In our context, for example, { x 2 -10 , y } can be tokenized as [x, ^, 2, +, -10, <sep>, y ], a sequence of tokens. The vocabulary set V is the collection of all possible tokens. Each token s ∈ V has a predesignated token ID i ( s ) ∈ N , and the input embedding layer of Transformer associates them with the i ( s ) -th row of the embedding matrix W E ∈ R |V|× D , where D is the embedding dimension. The embedding matrix is updated during training, and eventually, we have a nice vector representation for each token.

However, this approach requires a large embedding matrix to handle a wide range of number tokens. 6 The number tokens dominate the vocabulary set, and Transformers have to learn the relationship of the number tokens from scratch during the training. At the inference, if the given numbers are out of range, the model cannot work. In our case, this is inconvenient, particularly when the coefficient field is k = Q . For example, let F = {-5 / 3 y 3 -y -1 / 2 , 7 / 2 xy 2 -5 / 3 x -2 } , which we obtained from a random sampling with some upper bounds on the degree, number of terms, and integers appearing in numerators and denominators. Even the simplicity of F , its reduced ≺ lex -Gröbner basis has a polynomial of g 1 = x +569520 / 427411 y 2 -158760 / 427411 y +612912 / 427411 . If we tokenize a/b as [ a, /, b ] , we need to prepare more than a million integer tokens. Noting that g 1 ≈ x +1 . 33 y 2 -0 . 37 y +1 . 43 , it is more reasonable to handle coefficients as real values and also implement the inductive bias on the continuity of numbers in the embedding.

We propose a hybrid input embedding that accepts both discrete token IDs and continuous values. Let s = [ s 1 , . . . , s L ] to be a sequence of tokens. Some of these tokens are in V and otherwise in R . For those in V , the standard input embedding based on the embedding matrix is applied. For the others, a small feed-forward network f E : R → R D is applied. A Transformer with the proposed embedding should equip a regression head for these continuous tokens. This allows us to handle any number as a single token without the explosion of the vocabulary set (i.e., embedding matrix). As feed-forward networks are a continuous function, they naturally implement the continuity of numbers; two close values s 1 , s 2 ∈ R are expected to be embedded in similar vectors. The hybrid input embedding has two advantages. First, as claimed above, we are no longer suffering from the large embedding matrix for registering many number tokens and can naturally implement the continuity bias. Second, We do not have the 'out-of-range' issue. Further, we can scale the coefficients of given polynomials globally so that they match our training coefficient range. 7 Refer to App. D for the details.

6 Experiments

We now present the efficiency of the proposed dataset generation method and the learnability of Gröbner basis computation. 8 All the experiments were conducted with 48-core CPUs, 768GB RAM, and NVIDIA RTX A6000ada GPUs. The training of a model takes less than a day on a single GPU. More information on the profile of generated datasets, the training setup, and additional experimental results are given in Apps. E and F.

6.1 Dataset generation

First, we demonstrate the efficiency of the proposed dataset generation framework. We constructed 16 datasets D n ( k ) for n ∈ { 2 , 3 , 4 , 5 } and k ∈ { F 7 , F 31 , Q , R } and measured the runtime of the forward generation and our backward generation. The dataset D n ( k ) consists of 1,000 pairs of non-Gröbner set and Gröbner basis in k [ x 1 , . . . , x n ] of ideals in shape position. Each sample ( F, G ) ∈ D n ( k ) was prepared using Alg. 1 with ( d, d ′ , s max , ≺ ) = (5 , 3 , n + 2 , ≺ lex ) . The number of terms of univariate polynomials and n -variate polynomials is uniformly determined from [1 , 5] and [1 , 2] , respectively. When k = Q , the coefficient a/b are restricted to those with a, b ∈ {-5 , . . . , 5 } for random polynomials and a, b ∈ {-100 , . . . , 100 } for polynomials in F . In the forward generation, one may first generate random polynomial sets and then compute their Gröbner bases. However, this leads to a dataset with a totally different complexity from that constructed by the backward generation, leading to an unfair runtime comparison between the two generation processes. As such, the forward generation instead computes Gröbner bases of the non-Gröbner sets given by the backward generation, leading to the identical dataset. We used SageMath [82] with the libSingular backend. As Tab. 1 shows, our backward generation is a few orders of magnitude faster than the forward generation. A sharp runtime growth is observed in the forward generation as the number of variables increases. Note that these numbers only show the runtime on 1,000 samples, while training typically requires millions of samples. Therefore, the forward generation is almost infeasible, and the proposed method resolves a bottleneck in the learning of Gröbner basis computation.

6.2 Learnability of Gröbner basis computation

We now demonstrate that Transformers can learn to compute Gröbner bases. To examine the general Transformer's ability, we focus on a standard architecture (e.g., 6 encoder/decoder layers and 8 attention heads) and a standard training setup (e.g., the AdamW optimizer [65] with ( β 1 , β 2 ) = (0 . 9 , 0 . 999) and a linear decay of learning rate from 10 -4 ). The batch size was set to 16, and models were trained for 8 epochs. We also tested the hybrid input embedding. Refer to App. E for the complete information. Each polynomial set in the datasets is converted into a sequence using the prefix representation and the separator tokens. Unlike natural language processing, our task does not allow the truncation of an input sequence because the first term of the first polynomial in F certainly relates to the last term of the last polynomial. To make the input sequence length manageable for vanilla Transformers, we used simpler datasets D -n ( k ) using U 1 , U ′ 2 in Alg. 1 of a moderate density σ ∈ (0 , 1] . This makes the maximum sequence length less than 5,000. Specifically, we used σ = 1 . 0 , 0 . 6 , 0 . 3 , 0 . 2 for n = 2 , 3 , 4 , 5 , respectively. The training set has one million samples, and the test set has one thousand samples. With hybrid input embedding, coefficients are predicted by regression, and we quantized them for F p and otherwise regarded them correct when the mean squared error is less than 0.1.

Table 2 shows that trained Transformers successfully compute Gröbner bases with moderate/high accuracy. Several intriguing observations below are obtained. See App. F for more results. Particularly, App. F.3 presents several examples found in the datasets for which Transformer successfully computed Gröbner bases significantly faster than math algorithms. Table 2 also includes the results on Cauchy module datasets on which Transformers are trained and tested. The dataset generation starts with sampling the roots in k n , and the other parts follow the generation of D -n ( k ) . The results on ( Q , n = 3) with standard embedding is not shown as it requires too many number tokens.

The performance gap across the rings. The accuracy shows that the learning is more successful on infinite field coefficients k ∈ { Q , R } than finite field ones k = F p . This may be a counter-intuitive observation because there are more possible coefficients in G and F for Q than F p . Specifically, for G , the coefficient a/b ∈ Q is restricted to those with a, b ∈ {-5 , . . . , 5 } (i.e., roughly 50 choices), and a, b ∈ {-100 , . . . , 100 } (i.e., roughly 20,000 choices) for F . In contrast, there are only p choices for F p . The performance even degrades for the larger order p = 31 . Interestingly, the support accuracy shows that the terms forming the polynomial (i.e., the support of polynomial) are correctly identified well. Thus, Transformers have difficulty determining the coefficients in finite fields. Several studies have also reported that learning to solve a problem involving modular arithmetic may encounter some difficulties [21, 38, 74], but no critical workaround is known.

Incorrect yet reasonable failures. We observed that the predictions by a Transformer are mostly reasonable even when they are incorrect. For example, only several coefficients may be incorrect, and the support can be correct as suggested by the relatively high support accuracy in Tab. 2. In such a case, one can use a Gröbner basis computation algorithm that works efficiently given the leading terms of the target unknown Gröbner basis [83]. Refer to App. F.2 for extensive lists of examples.

Number token

の埋め込みの可視化

(a) Embedding of     -coefficients

Distance

N

orm

N

ormalized

dot product

Slice at 0

(Unnorm.)

Slice at 0

1 layer

2 layers

(b) Embedding of        -coefficients

Distance

N

orm

N

ormalized

dot product

Slice at 0

Slice at 0

Slice at 0

Slice at 0

(Unnorm.)

F

31

R

(i)

(ii)

(iii)

(iv)

(v)

(vi)

(i)

(ii)

(iii)

(iv)

(v)

(vi)

Hybrid embedding. Table 2 shows that determining coefficients by regression is less successful than classifications. For infinite field k , this may be because of the accumulation of coefficient errors during the auto-regressive generation. Thus, the current best practice would be to prepare many number tokens in the vocabulary set, or a sophisticated regression-by-classification approach may be helpful [76]. Note that the results for k = F p are shown for reference as the finite field elements do not have ordering. Figure 1 shows a contrast between the embedding functions learned in infinite field and finite field. Particularly, the slice of distance matrix (ii) and that of the dot-product matrix (vi) show that these metrics align well with the difference between numbers in R . However, we cannot observe convincing patterns in the embedding in F 31 . For the two-layer case in Fig. 1(a), we observe sharp changes around ± 5 of the horizontal axis. This may be because of the gap in the coefficient range in the input and output space. The coefficients of F ranges between [ -100 , 100] , while that of G does between [ -5 , 5] . In Tab. 3, we show that the increase of hidden layers of f E does not lead to improvement.

7 Conclusion

This study proposed the first learning approach to a fundamental algebraic task, the Gröbner basis computation. While various recent studies have reported the learnability of mathematical problems by Transformers, we addressed the first problem with nontriviality in the dataset generation. Ultimately, the learning approach may be useful to address large-scale problems that cannot be approached by Gröbner basis computation algorithms because of their computational complexity. Transformers can output predictions in moderate runtime. The outputs may be incorrect, but there is a chance of obtaining a hint of a solution, as shown in our experiments. We believe that our study reveals many interesting open questions to achieve Gröbner basis computation learning. Some are algebraic problems, and others are machine learning challenges, further discussed in Sec. H.

Acknowledgement. We would like to thank Masayuki Noro (Rikkyo University) for his fruitful comments on our dataset construction algorithm and Noriki Nishida (RIKEN Center for Advanced Intelligence Project) for his help in the implementation. Hiroshi Kera was supported by JST PRESTO Grant Number JPMJPR24K4, JST ACT-X Grant Number JPMJAX23C8, Mitsubishi Electric Information Technology R&D Center, and the Chiba University IAAR Research Support Program and the Program for Forming Japan's Peak Research Universities (J-PEAKS). Yuki Ishihara was supported by JSPS KAKENHI Grant Number JP22K13901 and Institute of Mathematics for Industry, Joint Usage/Research Center in Kyushu University (FY2023 Short-term Joint Research 'Speeding up of symbolic computation and its application to solving industrial problems' (2023a006)). Yuta Kambe was supported by Mitsubishi Electric Research Associate Program.

<latexi sh

1_b

64="

W29n k5CISurZD

NFMw/z

ET0

>A

X

cf

d

L

H

vg

p

o

Q

V

q

U

P

J3

O

j7

+

Y

B

m

y

R

G

K

8

<latexi sh

1_b

64="dAwpoyI9

78

0

5jVr/

k

>

E

H

cfZR

M

C

L

B

Wq

Dm

T

2

Q

zY

S

K

P

3g

v

u

n

F

O

J

X

U

1

References

J. Abbott, C. Fassino, and M.-L. Torrente. Stable border bases for ideals of points. Journal of Symbolic Computation , 43(12):883-894, 2008.

J. Abbott, M. Kreuzer, and L. Robbiano. Computing zero-dimensional schemes. Journal of Symbolic Computation , 39(1):31-49, 2005.

V. Agarwal, S. Aditya, and N. Goyal. Analyzing the nuances of transformers' polynomial simplification abilities. In The First Mathematical Reasoning in General Artificial Intelligence Workshop at International Conference on Learning Representations , 2021.

A. Alfarano, F. Charton, and A. Hayat. Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

R. Antonova, M. Maydanskiy, D. Kragic, S. Devlin, and K. Hofmann. Analytic manifold learning: Unifying and evaluating representations for continuous control. arXiv , abs/2006.08718, 2020.

E. Arnold. Modular algorithms for computing Gröbner bases. Journal of Symbolic Computation , 35:403-419, 2003.

M. F. Atiyah and I. G. MacDonald. Introduction To Commutative Algebra . Addison-Wesley series in mathematics. Avalon Publishing, 1994.

G. V. Bard. Algorithms for Solving Polynomial Systems . Springer US, 2009.

M. Bardet, J.-C. Faugère, and B. Salvy. On the complexity of the F5 Gröbner basis algorithm. Journal of Symbolic Computation , 70:49-70, September 2015.

T. Becker, V. Weispfenning, and H. Kredel. Gröbner Bases: A Computational Approach to Commutative Algebra . Graduate texts in mathematics. Springer-Verlag, 1993.

I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv , 2020.

J. Berthomieu, S. Graillat, D. Lesnoff, and T. Mary. Modular matrix multiplication on GPU for polynomial system solving. ACM Commun. Comput. Algebra , 57(2):35-38, August 2023.

J. Berthomieu, V. Neiger, and M. Safey El Din. Faster change of order algorithm for Gröbner bases under shape and stability assumptions. In Proceedings of the 2022 International Symposium on Symbolic and Algebraic Computation , pages 409-418, 2022.

L. Biggio, T. Bendinelli, A. Neitz, A. Lucchi, and G. Parascandolo. Neural symbolic regression that scales. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 936-945, 18-24 Jul 2021.

M. Brickenstein. Slimgb: Gröbner bases with slim polynomials. Revista Matemática Complutense , 23:453-466, 2010.

B. Buchberger. Ein Algorithmus zum Auffinden der Basiselemente des Restklassenringes nach einem nulldimensionalen Polynomideal (An Algorithm for Finding the Basis Elements in the Residue Class Ring Modulo a Zero Dimensional Polynomial Ideal) . PhD thesis, Mathematical Institute, University of Innsbruck, Austria, 1965. English translation in J. of Symbolic Computation, Special Issue on Logic, Mathematics, and Computer Science: Interactions. Vol. 41, Number 3-4, Pages 475-511, 2006.

L. Busé. Étude du résultant sur une variété algébrique . Theses, Université Nice Sophia Antipolis, 2001.

L. Busé, M. Elkadi, and B. Mourrain. Resultant over the residual of a complete intersection. Journal of Pure and Applied Algebra , 164(1):35-57, 2001.

F. Charton. Linear algebra with transformers. Transactions on Machine Learning Research , 2022.

F. Charton. What is my math transformer doing? - three results on interpretability and generalization. arXiv , abs/2211.00170, 2022.

F. Charton. Learning the greatest common divisor: explaining transformer predictions. In The Twelfth International Conference on Learning Representations , 2024.

F. Charton, A. Hayat, and G. Lample. Learning advanced mathematical computations from examples. In International Conference on Learning Representations , 2021.

Y. Chen, Q. Tao, F. Tonin, and J. A. Suykens. Primal-Attention: Self-attention through asymmetric kernel SVD in primal representation. In Advances in Neural Information Processing Systems , 2023.

D. A. Cox. Solving equations via algebras. In Solving Polynomial Equations , pages 63-124. Springer-Verlag, Berlin, 2005.

D. A. Cox, J. Little, and D. O'Shea. Ideals, Varieties, and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Algebra . Undergraduate Texts in Mathematics. Springer International Publishing, 2015.

S. D'Ascoli, P.-A. Kamienny, G. Lample, and F. Charton. Deep symbolic regression for recurrence prediction. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 4520-4536, 17-23 Jul 2022.

P. Diaconis and B. Sturmfels. Algebraic algorithms for sampling from conditional distributions. The Annals of Statistics , 26(1):363 - 397, 1998.

J. Ding, S. Ma, L. Dong, X. Zhang, S. Huang, W. Wang, N. Zheng, and F. Wei. LongNet: Scaling Transformers to 1,000,000,000 tokens. arXiv , 2307.02486, 2023.

T. W. Dubé. The structure of polynomial ideals and Gröbner bases. SIAM Journal on Computing , 19(4):750-773, 1990.

N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, S. Welleck, P. West, C. Bhagavatula, R. L. Bras, J. D. Hwang, S. Sanyal, X. Ren, A. Ettinger, Z. Harchaoui, and Y. Choi. Faith and fate: Limits of transformers on compositionality. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

D. Eisenbud. Commutative algebra: with a view toward algebraic geometry . Springer Science &Business Media, 2013.

C. Fassino. Almost vanishing polynomials for sets of limited precision points. Journal of Symbolic Computation , 45(1):19-37, 2010.

J.-C. Faugère. A new efficient algorithm for computing Gröbner bases (F4). Journal of Pure and Applied Algebra , 139(1):61-88, 1999.

J.-C. Faugère. A new efficient algorithm for computing Gröbner bases without reduction to zero (F5). In Proceedings of the 2002 International Symposium on Symbolic and Algebraic Computation , ISSAC '02, page 75-83, New York, NY, USA, 2002. Association for Computing Machinery.

J.-C. Faugère, P. M. Gianni, D. Lazard, and T. Mora. Efficient computation of zero-dimensional Gröbner bases by change of ordering. Journal of Symbolic Computation , 16(4):329-344, 1993.

P. Gianni and T. Mora. Algebraic solution of systems of polynomial equations using Groebner bases. In Applied Algebra, Algebraic Algorithms and Error-Correcting Codes , pages 247-257, Berlin, Heidelberg, 1989. Springer Berlin Heidelberg.

S. Golkar, M. Pettee, M. Eickenberg, A. Bietti, M. Cranmer, G. Krawezik, F. Lanusse, M. McCabe, R. Ohana, L. Parker, B. R.-S. Blancard, T. Tesileanu, K. Cho, and S. Ho. xVal: A continuous number encoding for large language models, 2023.

A. Gromov. Grokking modular arithmetic. arXiv , abs/2301.02679, 2023.

G.-M. Gruel and G. Pfister. A Singular Introduction to Commutative Algebra, 2nd Edition . Sringer Verlag, 2008.

D. Heldt, M. Kreuzer, S. Pokutta, and H. Poulisse. Approximate computation of zerodimensional polynomial ideals. Journal of Symbolic Computation , 44(11):1566-1591, 2009.

T. Hibi. Gröbner bases. Statistics and software systems. Springer Tokyo, March 2014.

C. Hou, F. Nie, and D. Tao. Discriminative vanishing component analysis. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , pages 1666-1672, Palo Alto, California, 2016. AAAI Press.

R. Iraji and H. Chitsaz. Principal variety analysis. In Proceedings of the 1st Annual Conference on Robot Learning (ACRL) , pages 97-108. PMLR, 2017.

P.-A. Kamienny, S. d'Ascoli, G. Lample, and F. Charton. End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems , 2022.

P.-A. Kamienny, G. Lample, S. Lamprier, and M. Virgolin. Deep generative symbolic regression with Monte-Carlo-Tree-Search. arXiv , abs/2302.11223, 2023.

A. Karimov, E. G. Nepomuceno, A. Tutueva, and D. Butusov. Algebraic method for the reconstruction of partially observed nonlinear systems using differential and integral embedding. Mathematics , 8(2):300-321, February 2020.

A. Karimov, V. Rybin, E. Kopets, T. Karimov, E. Nepomuceno, and D. Butusov. Identifying empirical equations of chaotic circuit from data. Nonlinear Dynamics , 111(1):871-886, 2023.

A. Kehrein and M. Kreuzer. Computing border bases. Journal of Pure and Applied Algebra , 205(2):279-295, 2006.

H. Kera. Border basis computation with gradient-weighted normalization. In Proceedings of the 2022 International Symposium on Symbolic and Algebraic Computation , pages 225-234, New York, 2022. Association for Computing Machinery.

H. Kera and Y. Hasegawa. Noise-tolerant algebraic method for reconstruction of nonlinear dynamical systems. Nonlinear Dynamics , 85(1):675-692, 2016.

H. Kera and Y. Hasegawa. Approximate vanishing ideal via data knotting. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , pages 3399-3406, Palo Alto, California, 2018. AAAI Press.

H. Kera and Y. Hasegawa. Spurious vanishing problem in approximate vanishing ideal. IEEE Access , 7:178961-178976, 2019.

H. Kera and Y. Hasegawa. Gradient boosts the approximate vanishing ideal. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence , pages 4428-4425, Palo Alto, California, 2020. AAAI Press.

H. Kera and Y. Hasegawa. Monomial-agnostic computation of vanishing ideals. Journal of Computational Algebra , 11:100022, 2024.

H. Kera and H. Iba. Vanishing ideal genetic programming. In Proceedings of the 2016 IEEE Congress on Evolutionary Computation (CEC) , pages 5018-5025, Piscataway, NJ, 2016. IEEE.

F. J. Király, M. Kreuzer, and L. Theran. Dual-to-kernel learning with ideals. arXiv , abs/1402.0099, 2014.

N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient Transformer. In International Conference on Learning Representations , 2020.

G. Lample and F. Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations , 2020.

R. Laubenbacher and B. Stigler. A computational algebra approach to the reverse engineering of gene regulatory networks. Journal of Theoretical Biology , 229(4):523-537, 2004.

R. Laubenbacher and B. Sturmfels. Computer algebra in systems biology. American Mathematical Monthly , 116(10):882-891, 2009.

C. Y. Li, E. Wenger, Z. Allen-Zhu, F. Charton, and K. E. Lauter. SALSA VERDE: a machine learning attack on LWE with sparse small secrets. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

J. Limbeck. Computation of approximate border bases and applications . PhD thesis, Passau, Universität Passau, 2013.

R. Livni, D. Lehavi, S. Schein, H. Nachliely, S. Shalev-Shwartz, and A. Globerson. Vanishing component analysis. In Proceedings of the 30th International Conference on Machine Learning , volume 28(1) of Proceedings of Machine Learning Research , pages 597-605, Atlanta, Georgia, USA, June 2013. PMLR.

H. Lombardi and I. Yengui. Suslin's algorithms for reduction of unimodular rows. Journal of Symbolic Computation , 39(6):707-717, 2005.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

R. H. Makarim and M. Stevens. M4GB: An efficient Gröbner-basis algorithm. In Proceedings of the 2017 ACM on International Symposium on Symbolic and Algebraic Computation , ISSAC'17, pages 293-300, New York, NY, USA, 2017. Association for Computing Machinery.

E. W. Mayr and A. R. Meyer. The complexity of the word problems for commutative semigroups and polynomial ideals. Advances in Mathematics , 46(3):305-329, 1982.

H. M. Möller and B. Buchberger. The construction of multivariate polynomials with preassigned zeros. In Computer Algebra. EUROCAM 1982. Lecture Notes in Computer Science , pages 24-31. Springer Berlin Heidelberg, 1982.

M. Noro and K. Yokoyama. A modular method to compute the rational univariate representation of zero-dimensional ideals. Journal of Symbolic Computation , 28(1):243-263, 1999.

M. Noro and K. Yokoyama. Usage of modular techniques for efficient computation of ideal operations. Math. Comput. Sci. , 12(1):1-32, 2018.

H. Park and G. Regensburger, editors. Gröbner Bases in Control Theory and Signal Processing . De Gruyter, 2007.

H. Park and C. Woodburn. An algorithmic proof of Suslin's stability theorem for polynomial rings. Journal of Algebra , 178(1):277-298, 1995.

D. Peifer, M. Stillman, and D. Halpern-Leistner. Learning selection strategies in buchberger's algorithm. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv , abs/2201.02177, 2022.

D. Saxton, E. Grefenstette, F. Hill, and P. Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations , 2019.

D. Shah and T. M. Aamodt. Learning label encodings for deep regression. In The Eleventh International Conference on Learning Representations , 2023.

Y. Shao, G. Gao, and C. Wang. Nonlinear discriminant analysis based on vanishing component analysis. Neurocomputing , 218:172-184, 2016.

H. Stewenius. Gröbner Basis Methods for Minimal Problems in Computer Vision . PhD thesis, Mathematics (Faculty of Engineering), 2005.

B. Sturmfels. Solving systems of polynomial equations . Number 97. American Mathematical Soc., 2002.

Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive network: A successor to Transformer for large language models. arXiv , 2023.

A. Suslin. On the structure of the special linear group over polynomial rings. Mathematics of the USSR - Izvestija , 11(2):221-238, April 1977.

The Sage Developers. SageMath, the Sage Mathematics Software System (Version 10.0) , 2023. https://www.sagemath.org .

C. Traverso. Hilbert functions and the Buchberger algorithm. Journal of Symbolic Computation , 6:287-304, 1997.

E. Ullah. New Techniques for Polynomial System Solving . PhD thesis, Universität Passau, 2012.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

L. Wang and T. Ohtsuki. Nonlinear blind source separation unifying vanishing component analysis and temporal structure. IEEE Access , 6:42837-42850, 2018.

M. Wang and J. Deng. Learning to prove theorems by learning to generate theorems. In Advances in Neural Information Processing Systems , volume 33, pages 18146-18157. Curran Associates, Inc., 2020.

Z. Wang, Q. Li, G. Li, and G. Xu. Polynomial representation for persistence diagram. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6116-6125, 2019.

E. Wenger, M. Chen, F. Charton, and K. E. Lauter. SALSA: Attacking lattice cryptography with Transformers. In Advances in Neural Information Processing Systems , volume 35, pages 34981-34994, 2022.

E. S. Wirth, H. Kera, and S. Pokutta. Approximate vanishing ideal computations at scale. In International Conference on Learning Representations , 2023.

E. S. Wirth and S. Pokutta. Conditional gradients for the approximately vanishing ideal. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151, pages 2191-2209, 28-30 Mar 2022.

H. Yan, Z. Yan, G. Xiao, W. Wang, and W. Zuo. Deep vanishing component analysis network for pattern classification. Neurocomputing , 316:240-250, 2018.

T. Yasuda, X. Dahan, Y.-J. Huang, T. Takagi, and K. Sakurai. MQ challenge: hardness evaluation of solving multivariate quadratic problems. Cryptology ePrint Archive , 2015.

A Basic Definitions in Algebra

Definition A.1 (Ring, Field ([7], Chap. 1 §1)) . A set R with an additive operation + and a multiplicative operation ∗ is called a (commutative) ring if it satisfies the following conditions:

a +( b + c ) = ( a + b ) + c for any a, b, c ∈ R ,

there exists 0 ∈ R such that a +0 = 0 + a = a for any a ∈ R ,

for any a ∈ R , there exists -a such that a +( -a ) = ( -a ) + a = 0 ,

a + b = b + a for any a, b ∈ R ,

a ∗ ( b ∗ c ) = ( a ∗ b ) ∗ c for any a, b, c ∈ R ,

there exists 1 ∈ R such that a ∗ 1 = 1 ∗ a = a for any a ∈ R ,

a ∗ ( b + c ) = a ∗ b + a ∗ c for any a, b, c ∈ R ,

( a + b ) ∗ c = a ∗ c + b ∗ c for any a, b, c ∈ R ,

a ∗ b = b ∗ a for any a, b ∈ R .

A commutative ring R is called a field if it satisfies the following condition

for any a ∈ R \ { 0 } , there exists a -1 such that a ∗ a -1 = a -1 ∗ a = 1 .

Definition A.2 (Polynomial Ring ([7], Chap. 1 §1)) . In Definition A.1, k [ x 1 , . . . , x n ] , the set of all n -variate polynomials with coefficients in k , satisfies all conditions (1)-(9). Thus, k [ x 1 , . . . , x n ] is called a polynomial ring.

Definition A.3 (Quotient Ring ([7], Chap. 1 §1)) . Let R be a ring and I an ideal of R . For each f ∈ R , we set [ f ] = { g ∈ R | f -g ∈ I } . Then, the set { [ f ] | f ∈ R } is called the quotient ring of R modulo I and denoted by R/I . Indeed, R/I is a ring with an additive operation + and a multiplicative operation ∗ , where [ f ] + [ g ] = [ f + g ] and [ f ] ∗ [ g ] = [ f ∗ g ] for f, g ∈ R respectively.

Definition A.4 (Generators) . For F = { f 1 , . . . , f s } ⊂ k [ x 1 , . . . , x n ] , the following set

is an ideal and said to be generated by F , and f 1 , . . . , f s are called generators .

Definition A.5 (0-dimensional ideal ([25], Chap. 5 §3, Thm. 6)) . Let F be a set of polynomials in k [ x 1 , . . . , x n ] . An ideal ⟨ F ⟩ is called a 0-dimensional ideal if the k -linear space k [ x 1 , . . . , x n ] / ⟨ F ⟩ is finite-dimensional, where k [ x 1 , . . . , x n ] / ⟨ F ⟩ is the quotient ring of k [ x 1 , . . . , x n ] modulo ⟨ F ⟩ .

Definition A.6 (Radical ideal ([7], Chap. 1 §1)) . For an ideal I of k [ x 1 , . . . , x n ] , the set { f ∈ k [ x 1 , . . . , x n ] | f m ∈ I for a positive integer m } is called the radical of I and denoted by √ I . Also, I is called a radical ideal if I = √ I .

Definition A.7 (Syzygy ([10], Chap. 3, §3)) . Let F = { f 1 , . . . , f s } ⊂ k [ x 1 , . . . , x n ] . A syzygy of F is an s -tuple of polynomials ( q 1 , . . . , q s ) ∈ k [ x 1 , . . . , x n ] s such that q 1 f 1 + · · · + q s f s = 0 .

Definition A.8 (Term ([10], Chap. 2, §1)) . For a polynomial f = ∑ α 1 ,...,α n c α 1 ,...,α n x α 1 1 · · · x α n n with c α 1 ,...,α n ∈ K and α 1 , . . . , α n ∈ Z ≥ 0 , each x α 1 1 · · · x α n n is called a term in f .

Definition A.9 (Total Degree ([25], Chap. 1 §1, Def. 3)) . For a term x α 1 1 · · · x α n n , its total degree is the sum of indices α 1 + · · · + α n . For a polynomial f , the total degree of f is the maximal total degree of terms in f .

Definition A.10 (Term order ([10], Definition 5.3)) . A term order ≺ is a relation between terms such that

(comparability) for different terms x α 1 1 · · · x α n n and x β 1 1 · · · x β n n , either x α 1 1 · · · x α n n ≺ x β 1 1 · · · x β n n or x β 1 1 · · · x β n n ≺ x α 1 1 · · · x α n n holds,

̸

̸

̸

(minimality of 1) the term 1 is the smallest term i.e. 1 ≺ x α 1 1 · · · x α n n for any term x α 1 1 · · · x α n n = 1 .

Example A.11. The lexicographic order ≺ lex prioritizes terms with larger exponents for the variables of small indices, e.g.,

Two terms are first compared in terms of the exponent in x 1 (larger one is prioritized), and if a tie-break is needed, the next variable x 2 is considered, and so forth.

Example A.12. The graded lexicographic order ≺ grlex prioritizes terms with higher total degree. 9 For tie-break, the lexicographic order is used, e.g.,

Term orders prioritizing lower total degree terms as ≺ grlex are called graded term orders.

Definition A.13 (Reduced Gröbner basis) . A ≺ -Gröbner basis G = { g 1 , . . . , g t } of I is called the reduced Gröbner basis of I if

the leading coefficient of g i with respect to ≺ is 1 for all i = 1 , . . . , t ,

no terms of g i lies on ⟨ LT( G \ { g i } ) ⟩ for any i = 1 , . . . , t

.

Proposition A.14 ([36], Prop. 1.6; [69], Lem. 4.4) . Let I be a 0-dimensional radical ideal. If k is of characteristic 0 or a finite field of large enough order, then a random linear coordinate change puts I in shape position.

B Cauchy module

We here provide the definition of the Cauchy module, which defines another class of 0-dimensional ideals.

Definition B.1 (Elementary symmetric polynomials) . The elementary symmetric polynomials s 1 , . . . , s n in n variables x 1 , . . . , x n are

̸

Definition B.2 (Cauchy module) . Let S n be the symmetric group on a finite set of size n . Let k be an algebraically closed field, and let α = ( α 1 , · · · , α n ) ∈ k n be a generic point (i.e., α i = α j for i = j ). Let the finite subset A ⊂ k n

Let f 1 ( t ) be a polynomial of t ,

Let us introduce indeterminates z 1 , . . . , z n and

The set of polynomials C = { f 1 , . . . , f s } is called the Cauchy module.

Remark B.3. The Cauchy module is the reduced ≺ lex -Gröbner basis of ⟨ C ⟩ with z 1 ≺ lex · · · ≺ lex z n .

C Dataset generation algorithm

Theorem 4.6. Let G = ( g 1 , . . . , g t ) ⊤ be a Gröbner basis of a 0-dimensional ideal in k [ x 1 , . . . , x n ] . Let F = ( f 1 , . . . , f s ) ⊤ = AG with A ∈ k [ x 1 , . . . , x n ] s × t .

If ⟨ F ⟩ = ⟨ G ⟩ , it implies s ≥ n .

If A has a left-inverse in k [ x 1 , . . . , x n ] t × s , ⟨ F ⟩ = ⟨ G ⟩ holds.

The equality ⟨ F ⟩ = ⟨ G ⟩ holds if and only if there exists a matrix B ∈ k [ x 1 , . . . , x n ] t × s such that each row of BA -E t is a syzygy 10 of G , where E t is the identity matrix of size t .

Proof.

In general, if an ideal I is generated by s elements and s < n , then the Krull dimension of k [ x 1 , . . . , x n ] /I satisfies that dim k [ x 1 , . . . , x n ] /I ≥ n -s > 0 (Krull's principal ideal theorem [31, §10]). Since the Krull dimension of k [ x 1 , . . . , x n ] / ⟨ G ⟩ is 0, we have s ≥ n .

(2) From

BF

=

F

=

BAG

AG

=

G

, we have

, indicating

⟨

F

⟨

F

⟩ ⊂ ⟨

⟩ ⊃ ⟨

G

⟩

⟩

G

.

If

A

has a left-inverse

. Therefore, we have

⟨

F

⟩

B

=

∈

⟨

k

[

⟩

G

x

.

(3) If the equality ⟨ F ⟩ = ⟨ G ⟩ holds, then there exists a t × s matrix B ∈ k [ x 1 , . . . , x n ] t × s such that G = BF . Since F is defined as F = AG , we have G = BF = BAG and G = E t G in k [ x 1 , . . . , x n ] t . Therefore we obtain ( BA -E t ) G = 0 . In particular, each row of BA -E t is a syzygy of G . Conversely, if there exists a t × s matrix B ∈ k [ x 1 , . . . , x n ] t × s such that each row of BA -E t is a syzygy of G , then we have ( BA -E t ) G = 0 in k [ x 1 , . . . , x n ] t , therefore the equality ⟨ F ⟩ = ⟨ G ⟩ holds since we have G = E t G = BAG = BF .

Proposition 4.7. For any A ∈ k [ x 1 , . . . , x n ] n × n with det( A ) ∈ k \ { 0 } , we have ⟨ F ⟩ = ⟨ G ⟩ .

1

, . . . , x

n

]

×

t

s

, we have

Proof. From the Cramer's rule, there exists B ∈ k [ x 1 , . . . , x n ] n × n such that BA = det( A ) E n , where E n denotes the n -byn identity matrix. Indeed, the i -th row B i of B satisfies for i = 1 , . . . , n ,

where ˜ A ( i ) j is the matrix A with the j -th column replaced by the i -th canonical basis (0 , ..., 1 , ..., 0) ⊤ . Since det( A ) is a non-zero constant, A has the left-inverse B in k [ x 1 , . . . , x n ] n × n . Thus ⟨ F ⟩ = ⟨ G ⟩ from Thm. 4.6.

Theorem 4.8. Consider a polynomial ring k [ x 1 , . . . , x n ] . Given the dataset size m , maximum degrees d, d ′ > 0 , maximum size of non-Gröbner set s max ≥ n , and term order ≺ , Alg. 1 returns a collection D = { ( F i , G i ) } i m =1 with the following properties: For all i = 1 , . . . , m ,

| G i | = n and | F i | ≤ s max .

The set G i is the reduced ≺ -Gröbner basis of ⟨ F i ⟩ . The set F i is not, unless G i , U 1 , U ′ 2 , P are all sampled in a non-trivial Zariski closed subset. 11

The ideal ⟨ F i ⟩ = ⟨ G i ⟩ is a 0-dimensional ideal in shape position.

The time complexity is O ( m ( nS 1 ,d + s 2 S n,d ′ +( n 2 + s 2 ) M n, 2 d ′ + d )) when ≺ = ≺ lex , where S n,d denotes the complexity of sampling an n -variate polynomial with total degree at most d , and M n,d denotes that of multiplying two n -variate polynomials with total degree at most d . If ⊀ = ≺ lex , O ( mnd 3 ) is additionally needed.

Proof. Outside of the Zariski subset part, statements 1-3 are trivial from Alg. 1 and the discussion in Sec.s 4.2 and 4.3. To obtain the desired Zariski subsets, we consider the vector space of polynomials of degree d +2 d ′ or less. We remark that if F i is a ≺ -Gröbner basis, its leading terms must belong to a finite amount of possibilities. For a polynomial to have a given term as its leading term, zero conditions on terms greater than this term are needed, defining a closed Zariski subset condition. By considering the finite union of all these conditions, we obtain the desired result.

To obtain one pair ( F, G ) , the random generation of G needs O ( nS 1 ,d ) , and the backward transform from G to F needs O ( s 2 S n,d ′ ) to get U 1 , U 2 and ( n 2 + s 2 ) M n, 2 d ′ + d ) for the multiplication F = U 1 PU 2 G . Note that the maximum total degree of polynomials in F is 2 d ′ + d .

D Hybrid Input Embedding

We here present the supplemental information of Transformers with hybrid input embedding. Let s = [ s 1 , . . . , s L ] to be a sequence of tokens. Some of these tokens are in V and otherwise in R . We call the former discrete tokens and the latter continuous tokens. For discrete tokens, the standard input embedding based on the embedding matrix is applied. For continuous tokens, a small feed-forward network f E : R → R D is applied. Unlike discrete tokens, continuous tokens are predicted by regression. For this sake, Transformers should equip a regression head, and they solve a classification task and a regression task simultaneously. In the classification task, continuous tokens are all replaced with a single coefficient token. In other words, the classification head predicts the support of the polynomials, while the regression head predicts the coefficients to be filled in the coefficient tokens. The auto-regressive generation process is naturally induced by a standard method.

In our experiments, we implemented f E by one-hidden layer ReLU Network, i.e.,

where w 1 , b 1 ∈ R D , W 2 ∈ R D × D , b 2 ∈ R D and φ is the ReLU function applied entry-wise. We also tried f E with one more hidden layer. However, this only has a minor improvement on the R case; see Tab. 3.

E Training Setup

This section provides the supplemental information of our experiments presented in Sec. 6.

E.1 Gröbner basis computation algorithms

In Tab. 1, we tested three algorithms provided in SageMath with the libSingular backend for forward generation.

STD ( libsingular:std ): The standard Buchberger algorithm.

SLIMGB ( libsingular:slimgb ): A variant of the Faugère's F4 algorithm. Refer to [15].

STDFGLM ( libsingular:stdfglm ): Fast computation using STD with the graded reverse lexicographic order followed by the FGLM for the change of term orders. Only for 0-dimensional cases.

E.2 Training setup

Dataset. Both training and test samples are generated using our method. It involves sampling of random polynomials. The degree and the support size (the number of terms) of them as well as coefficients are restricted by user-defined bounds (see Sec. 6.1). Let d max , µ max be the maximum degree and the maximum support size. A random polynomial is obtained as the sum of µ ∼ U [1 , µ max ] monomials uniformly and randomly sampled from k [ x 1 , . . . , x n ] ≤ d max . When the samples are fed to a Transformer, polynomials are tokenized into an infix representation. For example, { x 2 -1 / 2 y, y } ⊂ Q [ x, y ] is tokenized to [ C1, E2, E0, +, C-1, /, C2, E0, E1, <sep>, C1, E0, E1 ] .

Training. We used a Transformer model [85] with a standard architecture: 6 encoder/decoder layers, 8 attention heads, token embedding dimension of 512 dimensions, and feed-forward networks with 2048 inner dimensions. The absolute positional embedding is learned from scratch. The dropout rate was set to 0.1. We used the AdamW optimizer [65] with ( β 1 , β 2 ) = (0 . 9 , 0 . 999) with no weight decay. The learning rate was initially set to 10 -4 and then linearly decayed over training steps. All training samples are visited in a single epoch, and the total number of epochs was set to 8. The batch size was set to 16. At the inference time, output sequences are generated using a beam search with width 1. For the hybrid input embedding, we used a ReLU network with one hidden layer (cf. Sec. D). A model with this embedding predicts coefficients as continuous values, and the mean-squared loss with weight 0.01 is additionally used for the training. Note that while the exponents are also numbers, we treat them as discrete tokens because they are always discrete and their range is moderate.

F Additional Experimental Results.

We provide the additional experimental results with Transformers with the standard input embedding.

F.1 Dataset generation

The runtime comparison for datasets with and without density control is given in Tab. 4, and the success rate (i.e., not encountering the timeout) is given in Tab. 5. The generation of density-controlled datasets D -n ( k ) (1,000 samples) requires less runtime, and the proposed method is not always the fastest. However, it is important to remember that the forward method is still not feasible because of the difficulty in sampling overdetermined non-Gröbner sets (i.e., F s). Generally, such F only leads to a trivial ideal with Gröbner basis { 1 } . The profile of datasets is given in Tab. 6.

Table 4: Runtime comparison (in seconds) of forward generation (F.) and backward generation (B.) of dataset D n ( k ) of size 1,000. The forward generation used either of the three algorithms provided in SageMath with the libSingular backend. For D -n ( k ) , the proposed method is not necessarily the fastest. Note that the runtime of the forward methods does not include the sampling of F , and F is given from the datasets constructed by the backward method. The sampling step roughly consists of 30% of the runtime in the backward method. It is also worth noting that sampling of overdetermined F generally leads to a trivial ideal with the Gröbner basis { 1 } .

F.2 Success and failure cases

Tables 7-18 show examples of success cases. One can see that Transformers can solve many nontrivial instances. Tables 19 and 21 show examples of failure cases. One can see that, interestingly, the incorrect predictions appear reasonable. Examples are all taken according to their order in each dataset (i.e., no cherry-picking).

F.3 Superiority of Transformer in several cases.

Approaching Gröbner basis computation using a Transformer has a potential advantage in the runtime because the computational cost has less dependency on the problem difficulty than mathematical algorithms do. However, currently, mathematical algorithms run faster than Transformers because of our naive input scheme. Nevertheless, we observed several examples in our D -n ( k ) datasets for which Transformers generate the solutions efficiently, while mathematical algorithms take significantly longer time or encounter a timeout.

Particularly, we examined the examples in D -n ( k ) where several forward methods encounter a timeout with the five-second budget, see Tab. 5. We fed these examples to Transformer and the three forward algorithms again, but now with a 100-second budget. For such examples, as shown in Tab. 22, Transformers completed the computation in less than a second, while the two forward algorithms, STD and SLIMGB, often used longer computation time or encountered a timeout.

It is worth noting that STD and SLIMGB are general-purpose algorithms, while STDFGLM is specially designed for the zero-dimension ideals. To summarize, Transformers successfully computed Gröbner bases with much less runtime than general-purpose algorithms for several examples. This shows a

potential advantage of using a Transformer in Gröbner basis computation, particularly for large-scale problems.

F.4 Generalization to out-distribution samples

Handling out-distribution samples is beyond the scope of the current work. Several studies of using a Transformer for math problems (e.g., integer multiplication [30] and linear algebra [19]) addressed out-distribution generalization by controlling the training sample distribution. This is because these problems are of moderate difficulty, and naive training sample generation methods exist. However, this is not the case for our task, and we thus focused on the problem of efficient generation of training samples in this paper. Nevertheless, we consider that presenting a limitation of our work by experiments is helpful for future work. Thus, we here present that the Transformers trained on our datasets certainly fail on out-distribution samples through several cases.

Out-distribution samples. We generate additional datasets D u n ( k ) for k ∈ { Q , F 7 , F 31 } . These datasets are generated as D n ( k ) with a slight difference in sampling of random polynomials. Originally, a (degree-bounded) random polynomial is obtained using monomials randomly sampled from k [ x 1 , . . . , x n ] ≤ d max . Since there are more high-degree terms than low-degree ones, random polynomials are more likely to be high-degree. In D u n ( k ) , we instead uniformly sample the degree-bound d from U [1 , d max ] , and then conduct the sampling of monomials. As Tab. 23 shows, this change in the sampling strategy causes some distribution shifts. Table 24 shows the prediction accuracy and support accuracy on the new datasets. As can be seen, the accuracy drops when the base field is a finite field F p . When it is Q , the accuracy drop is moderate.

Katsura-n. Katsura-n is a typical benchmarking example for Gröbner basis computation algorithms ('-n' denotes the number of variables). Table 25 shows a list of examples for different n ∈ { 2 , 3 , 4 } and k ∈ { Q , F 7 , F 31 } . While the Gröbner bases in Katsura-n have a form of Eq. (4.1), one can readily find its qualitative difference in the non-Gröbner sets from those in our D n ( k ) datasets (cf. Tables 7-18). For example, non-Gröbner sets in Katsura-n consist of low-degree sparse polynomials, whereas those in D n ( k ) are not because of the generation processes (i.e., the product and sum of polynomials through the multiplication by polynomial matrices U 1 , U 2 ). Therefore, Katsura-n examples are greatly out-distributed samples for our Transformers. We fed these samples to the trained Transformers, but only obtained the output sequences that cannot be transformed back to polynomials because of their invalid prefix representation.

As Gröbner basis computation is an NP-hard problem, it may not be a good idea to peruse a general solver via learning. Instead, we should ultimately aim at a solver for large-scale but specialized cases. Thus, the generation of application-specific training samples and the pursuit of in-distribution accuracy will be a future direction. From this perspective, existing mathematical benchmark examples may not be practical because they are mostly artificial, empirically found difficult, and/or designed for easily generating variations in the number of variables n . 12 They are useful for math algorithms, i.e., the algorithms proved to work for all the cases (i.e., 100% in/out-distribution samples), but not for our current work because they are out-distribution samples.

G Buchberger-Möller Algorithm for Prob. 4.1

Here, we discuss another approach for Prob. 4.1 using the Buchberger-Möller (BM) algorithm [2, 68]. Although we did not adopt this approach, we include this for completeness as many variants have been recently developed and applied extensively in machine learning and other data-centric applications.

Given a set of points X ⊂ k n and a graded term order, the BM algorithm computes a Gröbner basis of its vanishing ideal I ( X ) = { g ∈ k [ x 1 , . . . , x n ] | g ( p ) = 0 , ∀ p ∈ X } . While several variants follow in computational algebra [1, 32, 40, 48, 49, 54, 62], interestingly, it is also recently tailored for machine learning [42, 51-53, 56, 63, 90, 91]. Various applications have followed such as machine learning [77, 92], signal processing [86-88], nonlinear dynamics [46, 47, 50], and more [5, 43, 55]. Such broad applications derived from the distinguishing design of the BM algorithm: unlike most computer-algebraic algorithms, it takes a set of points (i.e., dataset) as input, not a set of polynomials.

Therefore, to address Prob. 4.1, one may consider using the BM algorithm or its variants, e.g., by running the BM algorithm m times while sampling diverse sets of points. An important caveat is that Gröbner bases that can be given by the BM algorithm may be more restrictive than those considered in the main text (i.e., the Gröbner bases of ideals in shape position). For example, the former generates the largest ideals that have given k -rational points for their roots, whereas this is not the case for the latter. Another drawback of using the BM algorithm is its large computational cost. The time complexity of the BM algorithm is O ( n · | X | 3 ) . Furthermore, we need O ( n d ) points to obtain a Gröbner basis that includes a polynomial of degree d in the average case. Therefore, the BM algorithm does not fit our settings that a large number of Gröbner bases are needed (i.e., m ≈ 10 6 ). Accelerating the BM algorithm by reusing the results of runs instead of independently running the algorithm many times can be interesting for future work.

H Open Questions

Random generation of Gröbner bases To our knowledge, no prior studies addressed this problem. Our study focuses on generic 0-dimensional ideals. These ideals are generally in shape position, and a simple sampling strategy is available. However, some applications may be interested in other classes of ideals (e.g., positive dimensional or binomial ones) or a particular subset of 0-dimensional ideals (e.g., those associated with a single solution in the coefficient field). The former case is an open problem. The latter case may be addressed by the Buchberger-Möller algorithms [68] (cf. App. G).

Backward Gröbner problem. Machine learning models perform better for in-distribution samples than out-distribution samples. Thus, it is essential to design a training sample distribution that is close to one's use case. As noted in Sec. 4.1, polynomial systems (either Gröbner basis or not) take a domain-specific form. Backward generation gives us control over Gröbner bases but not for non-Gröbner sets. Hence, we need a well-tailored backward generation method specialized to an application, as the specialized Gröbner basis computation algorithms in computational algebra. This paper addressed a generic case. Prop. 4.7 states that any matrix A ∈ SL n ( k [ x 1 , . . . , x n ]) satisfies Prob. 4.5. This raises two sets of open questions: (i) are there matrices outside SL n ( k [ x 1 , . . . , x n ]) satisfying Prob. 4.5? Can we sample them? and (ii) is it possible to efficiently sample matrices of SL n ( k [ x 1 , . . . , x n ]) ? To efficiently generate our dataset, we have restricted ourselves to sampling matrices having a Bruhat decomposition (see Eq. (4.2)), which is a strict subset of SL n ( k [ x 1 , . . . , x n ]) . Sampling matrices in SL n ( k [ x 1 , . . . , x n ]) remains an open question. Thanks to Suslin's stability theorem and its algorithmic proofs [64, 72, 81], SL n ( k [ x 1 , . . . , x n ]) is generated by elementary matrices and a decomposition into a product of elementary matrices can be computed algorithmically. One may hope to use sampling of elementary matrices to sample matrices of SL n ( k [ x 1 , . . . , x n ]) . It is unclear whether this can be efficient as many elementary matrices are needed [64].

Distribution analysis. Several studies have reported that careful design of a training set leads to a better generalization of Transformers [19, 21]. Algebraically, analyzing the distribution of Gröbner bases and the non-Gröbner sets is challenging, particularly when some additional structures (e.g., sparsity) are injected. Thus, the first step may be to investigate the generic case (i.e., dense polynomials). In this case, Thm. 4.6(3) is helpful to design an algorithm that is certified to be able to yield all possible G and F almost uniformly. While dataset generation algorithms should run efficiently for practicality, a solid analysis may be of independent interest in computational algebra.

Long mathematical expressions. As in most of the prior studies, we used a Transformer of a standard architecture to see the necessity of a specialized model. From the accuracy aspect, such a vanilla model may be sufficient for Q [ x 1 , . . . , x n ] but not for F p [ x 1 , . . . , x n ] . From the efficiency perspective, the quadratic cost of the attention mechanism prevents us from scaling up the problem size. For large n , both non-GB sets and Gröbner bases generally consist of many dense polynomials, leading to long input and output sequences. Unlike natural language processing tasks, the input sequence cannot be split as all the symbols are related in mathematical tasks. It is worth studying several attention mechanisms that work with sub-quadratic memory cost [11, 23, 28, 57, 80] can be introduced with a small degradation of performance even for mathematical sequences, which have a different nature from natural language (e.g., a single modification of the input sequence can completely change the solution of the problem).

ID

F

G

0

f

1

=

x

0

-

x

4

4

-

x

3

4

-

2

x

2

4

+1

/

4

g

1

=

x

0

-

x

4

4

-

x

3

4

-

2

x

2

4

+1

/

4

f

2

=

5

/

4

x

0

x

3

1

+ 1

/

2

x

0

x

2

1

-

5

/

4

x

3

1

x

4

4

-

5

/

4

x

3

1

x

3

4

-

5

/

2

x

3

1

x

2

4

+5

/

16

x

3

1

-

1

/

2

x

2

1

x

4

4

-

1

/

2

x

2

1

x

3

4

-

x

2

1

x

2

4

+ 1

/

8

x

2

1

-

1

/

3

x

1

x

2

x

2

3

+

3

x

1

x

3

x

4

+ 5

/

3

x

2

x

2

3

x

4

+

x

2

-

15

x

3

x

2

4

-

5

x

4

4

+

x

3

4

+2

/

5

x

2

4

+2

x

4

-

5

/

2

g

2

=

x

1

-

5

x

4

f

3

=

x

3

0

x

4

-

x

2

0

x

5

4

-

x

2

0

x

4

4

-

2

x

2

0

x

3

4

+

1

/

4

x

2

0

x

4

+

x

0

x

2

1

x

4

+

2

x

0

x

1

x

2

4

+

x

0

x

3

3

-

x

2

1

x

5

4

-

x

2

1

x

4

4

-

2

x

2

1

x

3

4

+

1

/

4

x

2

1

x

4

-

4

/

3

x

1

x

3

2

x

2

3

x

4

+1

/

15

x

1

x

2

2

x

3

3

+

12

x

1

x

2

2

x

3

x

2

4

-

3

/

5

x

1

x

2

x

2

3

x

4

-

2

x

1

x

6

4

-

2

x

1

x

5

4

-

4

x

1

x

4

4

+1

/

2

x

1

x

2

4

+20

/

3

x

3

2

x

2

3

x

2

4

+

4

x

3

2

x

4

-

1

/

3

x

2

2

x

3

3

x

4

-

60

x

2

2

x

3

x

3

4

-

1

/

5

x

2

2

x

3

-

20

x

2

2

x

5

4

+ 4

x

2

2

x

4

4

+ 8

/

5

x

2

2

x

3

4

+

8

x

2

2

x

2

4

-

10

x

2

2

x

4

+ 3

x

2

x

2

3

x

2

4

+

x

2

x

3

x

4

4

-

1

/

5

x

2

x

3

x

3

4

-

2

/

25

x

2

x

3

x

2

4

-

2

/

5

x

2

x

3

x

4

+

1

/

2

x

2

x

3

-

x

3

3

x

4

4

-

x

3

3

x

3

4

-

2

x

3

3

x

2

4

+1

/

4

x

3

3

+

x

5

4

+3

/

2

x

4

4

-

1

/

2

x

3

4

-

5

/

3

x

2

4

-

1

g

3

=

x

2

-

5

x

4

4

+

x

3

4

+2

/

5

x

2

4

+2

x

4

-

5

/

2

f

4

=

1

/

2

x

2

0

x

1

x

2

+

1

/

2

x

0

x

2

1

x

2

3

x

4

+

4

/

3

x

0

x

2

1

x

3

x

2

4

-

5

/

3

x

0

x

2

1

x

3

-

5

/

6

x

0

x

1

x

2

x

4

3

-

1

/

2

x

0

x

1

x

2

x

4

4

-

1

/

2

x

0

x

1

x

2

x

3

4

-

x

0

x

1

x

2

x

2

4

+1

/

8

x

0

x

1

x

2

+

15

/

2

x

0

x

1

x

3

3

x

4

+

25

/

6

x

0

x

2

x

4

3

x

4

+

5

/

2

x

0

x

2

x

2

3

+ 1

/

2

x

0

x

5

3

+ 4

/

3

x

0

x

4

3

x

4

-

75

/

2

x

0

x

3

3

x

2

4

-

25

/

2

x

0

x

2

3

x

4

4

+5

/

2

x

0

x

2

3

x

3

4

+

x

0

x

2

3

x

2

4

+

5

x

0

x

2

3

x

4

-

25

/

4

x

0

x

2

3

-

1

/

2

x

2

1

x

2

3

x

5

4

-

1

/

2

x

2

1

x

2

3

x

4

4

-

x

2

1

x

2

3

x

3

4

+

1

/

8

x

2

1

x

2

3

x

4

-

4

/

3

x

2

1

x

3

x

6

4

-

4

/

3

x

2

1

x

3

x

5

4

-

x

2

1

x

3

x

4

4

+ 5

/

3

x

2

1

x

3

x

3

4

+ 11

/

3

x

2

1

x

3

x

2

4

-

5

/

12

x

2

1

x

3

-

1

/

2

x

5

3

x

4

4

-

1

/

2

x

5

3

x

3

4

-

x

5

3

x

2

4

+

1

/

8

x

5

3

-

4

/

3

x

4

3

x

5

4

-

4

/

3

x

4

3

x

4

4

-

8

/

3

x

4

3

x

3

4

+

1

/

3

x

4

3

x

4

+1

/

2

x

2

3

x

5

4

+3

/

4

x

2

3

x

4

4

-

1

/

4

x

2

3

x

3

4

-

5

/

6

x

2

3

x

2

4

-

1

/

2

x

2

3

+ 4

/

3

x

3

x

6

4

+ 2

x

3

x

5

4

-

2

/

3

x

3

x

4

4

-

20

/

9

x

3

x

3

4

-

4

/

3

x

3

x

4

+

x

3

+

5

x

4

4

+1

/

5

x

2

4

+1

/

2

x

4

+2

/

5

g

4

=

x

3

+5

x

4

4

+1

/

5

x

2

4

+1

/

2

x

4

+2

/

5

f

5

=

-

x

2

0

x

2

3

+

3

/

2

x

0

x

2

1

x

2

x

2

3

x

4

+

3

/

2

x

0

x

2

x

5

3

+

x

0

x

2

3

x

4

4

+

x

0

x

2

3

x

3

4

+2

x

0

x

2

3

x

2

4

-

1

/

4

x

0

x

2

3

-

3

/

2

x

2

1

x

2

x

2

3

x

5

4

-

3

/

2

x

2

1

x

2

x

2

3

x

4

4

-

3

x

2

1

x

2

x

2

3

x

3

4

+3

/

8

x

2

1

x

2

x

2

3

x

4

+1

/

3

x

1

x

2

2

x

4

3

-

3

x

1

x

2

x

3

3

x

4

+

x

1

-

5

/

3

x

2

2

x

4

3

x

4

-

x

2

2

x

2

3

-

3

/

2

x

2

x

5

3

x

4

4

-

3

/

2

x

2

x

5

3

x

3

4

-

3

x

2

x

5

3

x

2

4

+

3

/

8

x

2

x

5

3

+ 15

x

2

x

3

3

x

2

4

+ 3

/

2

x

2

x

2

3

x

5

4

+

29

/

4

x

2

x

2

3

x

4

4

-

7

/

4

x

2

x

2

3

x

3

4

-

29

/

10

x

2

x

2

3

x

2

4

-

2

x

2

x

2

3

x

4

+

x

2

x

2

3

-

3

x

2

3

x

2

4

-

15

x

3

x

6

4

-

3

/

5

x

3

x

4

4

-

3

/

2

x

3

x

3

4

-

6

/

5

x

3

x

2

4

-

5

x

4

g

5

=

x

5

4

+3

/

2

x

4

4

-

1

/

2

x

3

4

-

5

/

3

x

2

4

-

1

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction present the scope of the study and its contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clarify the problem setting in this paper and potential future work in Section 4.1. Besides, several open questions are discussed in detail in the Appendix H.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: All the assumptions and complete proofs are provided in Sections 4 and C. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided detailed description of the experiment setups.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code to reproduce our experiments is released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and test details to follow the results are provided in Sections 6 and E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars were not measured due to computational costs. However, we conducted experiments on various setups and provide their details in Sections 6 and F.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information is given at the beginning of Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have confirmed the present study meets the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: This paper focuses on the learnability of a mathematical task, and we cannot see any urgent social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: This paper does not have an evident risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a machine learning algorithm to compute Gröbner bases of 0-dimensional ideals in shape position. \nTo address the backward Gröbner problem, the authors propose an algorithm based on 1) random generation of Gröbner bases by sampling univariate polynomials related to the shape position, then on 2) random generation of a polynomial matrix to generate non-Gröbner sets. \nThe associated algorithm running time is derived in Theorem 4.8. \nAfter generating tuples with the two above-mentioned steps, the authors rely on Transformers for learning to compute Gröbner bases.\n\nSTRENGTHS:\n- The benchmarks show that computing Gröbner bases with the proposed backward approach can be substantially more efficient than with the usual \"forward\" algorithms.\n\nWEAKNESSES:\n- The paper contains many typos, grammatical mistakes and missing articles. The authors are invited to carefully rewrite the paper to increase the overall quality. Two examples (among many) are in p6, l255: \"Consider polynomial ring\", \"Given dataset size\". \n\n- The training approach, via Transformers, does not come with any theoretical guarantees related to global/local convergence, running time, error bounds. No explanation is provided regarding the cases where the method works or fails. The ML algorithms related to transformers are mostly described in the appendix and only vaguely in the main text. \n\n- The paper does not address concrete benchmarks of target applications (e.g. in cryptography or biological systems) where computing Gröbner bases is out of reach or very time consuming. Providing such experiments would definitely make the approach credible. \n\n- Many important aspects (theorem proofs, limitations) are postponed to the appendix. Overall I believe that this work, possibly sound and surely interesting, would be worth publishing but the current conference format (coming together with a short allocated review time) and focus are certainly not the best fit.\n\nQUESTIONS:\n- The framework is limited to 0-dimensional ideals in shape position. It is well known that the shape position assumption is not so strong but the 0-dimensional ideal one is very strong. \nRandom search applied to higher-dimensional ideals would be more challenging and interesting. Could the authors provide more insights to address such problems?\n\n- Could the authors derive a killer application for which their framework would provide new results or results obtained in a more efficient way than with the forward approach?\n\nLIMITATIONS:\n- There is no section dedicated about the limitations of this work in the main text. Section H is not part of the main text.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article investigates the use of machine learning techniques to compute Gröbner basis of polynomial systems. This problem consists in, given a term order and a finite set of polynomials $f_1, \\cdots, f_m$, to compute another set of polynomials $g_1, \\cdots, g_m$ that have the following desirable properties $(P)$: \n\n$(P)$ the $g_i$'s all belong the ideal generated by $f_1, \\cdots, f_m$, and such that any leading term in the ideal generated by the $f_i$'s is generated by the $g_i$'s.\n\nTo do so, the authors propose a method to first sample training data, which was previously problematic. Then, after embedding the set of polynomials $f_i$'s and $g_i$'s that they embed (after tokenization and a hybrid embedding technique that they detail in the appendix), they make use of a transformer to output the $g_i$'s given the $f_i$'s.\n\nSTRENGTHS:\n- The article proposes a serious study, is well-written and very comprehensible for non-experts in computational algebra and Gröbner basis. \n\n- The introduction and comparison to existing work is clear.\n\n- The articles presents interesting ideas to generate training datasets, using a backward approach. Instead of first sampling the $f_i$'s (non-Gröbner set) and then an associated Gröbner basis, they first generate a set of polynomials formed by some gi's and then generate an associated system of polynomials fi's that thave property (P) (in other words, first generate the solutions, then the problem).\n\n- The authors included example of success for, but also examples of failure, which can be of use for future improvements.\n\n- The authors share in details the results and conclusion of their study, in particular their experiments. Therefore, this article may be of great interest beyond the computational algebra community, including those who have to solve polynomial systems of questions for various applications.\n\nWEAKNESSES:\nI do not see a major weakness of this article. It constitutes a serious study of the use of machine learning advancements for computational algebra. One comment (not a weakness): Section D, there is a reference to a Table that is missing (line 728).\n\nQUESTIONS:\n- Line 115, the authors mention the notion of reduced Gröbner basis. Do they mean normalized and in shape position?  The authors may want to clarify this. \n\n- The authors may want (if they think this is relevant) to introduce some short additional comment for non-experts about the Gröbner basis definition line 109. This would help to understand the motivation of the definition. Namely: Understanding and finding solutions of a polynomial system can be adressed via studying the ideal generated by the fi's. For example, if one is able to find if the constant 1 is in the ideal generated by the fi's, then there is no solution for the system. It turns out the condition on the leading terms in the defintion are equivalent to, given a polynomial h, that the remainder after multivariate division of h by the gi's to be zero if and only if the h is in the ideal generated by the fi's (see for instance Definition 1 in [70]). \nHence one can answer the above question (if 1 is in the ideal) -- as well as others-- by performing a multivariate division algorithm taking on 1 and the Gröbner basis. It the output is 0, then 1 belong to the ideal and there is no solution.\n\nLIMITATIONS:\nThe authors clearly explain the apparent limitation of their work, in particular to which extent the experiments they conducted are successful. They report extensive experimental results in the appendix, in particular failed examples.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper proposes to learn to compute Gröbner bases (as the title says). This includes two important problems. 1) Generating the dataset and 3) Finding an appropriate encoding of the problem to feed into a transformer architecture. That is finding an encoding for a system of polynomials to be solved.

The paper gives adequate solution for both of these issues and implements it,

STRENGTHS:
The paper tackles a really tough and important problem with a novel approach. This is done while also placing the work nicely in the existing literature.

I enjoyed reading the open questions section in the Appendix.

WEAKNESSES:
The method does not provide any guarantees. In this regard it is then not entirely clear what the benefit would be of having "hints at the right solution".

QUESTIONS:
1) Could the authors comment on my point raised in the "weakness". Specifically, how do you envisage to use learned Gröbner basis to be useful for any downstream tasks? Would you use them as a heuristic to exactly solve polynomial systems?

2) Why did you limit the rings to Q,R F_p. Why not N or p-adic numbers?

LIMITATIONS:
Exemplary!
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGröbner bases are a tool of fundamental importance in the field of computational algebra. Unfortunately known algorithms for computing Gröbner bases are very ineficient, having a running time that is double-exponential on the number of variables. In this work, the authors propose a machine-learning based approach for the computation problem. Instead of devising an algorithm that is always guaranteed to correctly output a Gröbner basis, the authors propose a learning algorithm based on transformers. \n\nTo address this learning problem, the authors address other interesting problems such as random generation of Gröbner bases and the Backward Gröbner problem.\n\nSTRENGTHS:\nThe paper provide an interesting experimental evaluation on how transformers can be used to learn Gröbner bases. The random generation of Gröbner bases is also interesting.\n\nWEAKNESSES:\nThe main weakness, in my opinion, is that the paper does not attempt to provide a characterization of a subclass of polynomial systems that can be efficiently solved using the transformed-based approach. Therefore it is difficult to have an idea of what properties the input polynomial system must satisfy in order for the approach to give reasonable results. \n\nThe number of variables considered in the experiments (n=2,3,4,5) is way to low to give an idea about the complexity of the problem.  For this number of variables it is not clear whether the machine learning method provides any advantage at all with respect to traditional computational algebraic approaches. \n\nThe theoretical part of the paper is much more concentrated on the issue of random generation of Gröbner bases than on the learning part. For this reason, it is my impression that the authors are putting too much emphasis on the part of the paper where results are not very satisfactory (the learning theory part). The paper would be more solid if concentrated on random generation of Gröbner bases with the learning part as an interesting application.\n\nQUESTIONS:\n1) What is the largest number of variables for which the machine learning method gave interesting results? \n\n2) How does the accuracy of the model decay with the number of variables? \n\n3) What is the relation between the number of variables and the time necessary to train the model?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Transformer based method to compute Gröbner basis, a known NP-hard problem.  The authors focus on polynomials with 0-dimensional radical ideals and propose efficient algorithms to generate training samples of polynomial systems and their corresponding Gröbner basis. The main novelties include (1) an efficient backward transformation algorithm from a Gröbner basis to an associated non-Gröbner set; (2) a hybrid input embedding for both discrete and continuous-value tokens. The transformer, which was trained on millions of Gröbner basis pairs generated by the authors, performed well on some types of rings.\n\nSTRENGTHS:\nThe paper introduced a novel method to compute a significant problem in computational algebra, demonstrating high accuracy for certain types of rings. The efficient backward sampling methods are less explored and may facilitate further ML studies in this field. Additionally, this backward approach helps restrict the Gröbner basis to applications of interest. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n* I have serval questions about the choice of the 0-dimensional ideal as the scope of the study : \n(1) the authors first mentioned “…, and thus, we should focus on a particular class of ideals and pursue in-distribution accuracy” and then claimed “… and thus, we focus on the generic case and leave the specialization to future work”.  Do you regard this choice as a “particular” or “general” case of the Gröbner basis computation? \n(2) What is motivation of focusing on 0-dimensional ideals (since it is already mentioned that this work is meant to be general)? Is it purely because of the sampling easiness? Also, what are the difficulties of working on the more general Gröbner basis computation problem? \n(3) The discussion of, whether Transformers, or more generally, ML methods, can help NP-hard problems generally or only in-distribution, is interesting. There are some results from the SAT field that end-to-end ML models may be able to generalize to out-of-distribution problems [1]. As you claim here “in-distribution accuracy” is what we are after, it is important to include some experiments across different applications, to support this claim. \n\n* The hybrid input embedding for both discrete and continuous-value tokens is claimed to be one of the paper’s contributions. However, this idea is not new, as other research has explored similar problems (e.g.,  [2, 3, 4]). The authors do not provide a review of existing methods or distinguish their approach from these works. \n\nIt is interesting to note that many incorrect results are reasonable. This suggests that incorporating the Transformer method with planning (5) or RL (6) may help the performance through feedbacks. \n\n\nReference\n[1] Cameron, Chris, et al. \"Predicting propositional satisfiability via end-to-end learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.\n[2] Charton, François. \"Linear algebra with transformers.\" arXiv preprint arXiv:2112.01898 (2021).\n[3] Golkar, Siavash, et al. \"xval: A continuous number encoding for large language models.\" arXiv preprint arXiv:2310.02989 (2023).\n[4] McLeish, Sean, et al. \"Transformers Can Do Arithmetic with the Right Embeddings.\" arXiv preprint arXiv:2405.17399 (2024).\n[5] Kamienny, Pierre-Alexandre, et al. \"Deep generative symbolic regression with Monte-Carlo-tree-search.\" International Conference on Machine Learning. PMLR, 2023.\n[6] Jha, Piyush, et al. \"RLSF: Reinforcement Learning via Symbolic Feedback.\" arXiv preprint arXiv:2405.16661 (2024).\n\nQUESTIONS:\n* With the proposed sampling methods based on shape position and Cauchy module, how diverse is this generated set? Does these generating methods further restrict the scope less than the 0-dimensional radical ideals?\n\nLIMITATIONS:\nIt is not clear whether this work can be generalized to beyond 0-dimensional ideals.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}