Title: Suggesting Variable Order for Cylindrical Algebraic Decomposition via Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Graph Neural Network

Cylindrical Algebraic Decomposition.
Abstract: Cylindrical Algebraic Decomposition (CAD) is one of the pillar algorithms of symbolic computation, and its worst-case complexity is double exponential to the number of variables. Researchers found that variable order dramatically affects efficiency and proposed various heuristics. 
The existing learning-based methods are all supervised learning methods that cannot cope with diverse polynomial sets.
This paper proposes two Reinforcement Learning (RL) approaches combined with Graph Neural Networks (GNN) for Suggesting Variable Order (SVO). One is GRL-SVO(UP), a branching heuristic integrated with CAD. The other is GRL-SVO(NUP), a fast heuristic providing a total order directly. We generate a random dataset and collect a real-world dataset from SMT-LIB. The experiments show that our approaches outperform state-of-the-art learning-based heuristics and are competitive with the best expert-based heuristics. Interestingly, our models show a strong generalization ability, working well on various datasets even if they are only trained on a 3-var random dataset. The source code and data are available at https://github.com/dongyuhang22/GRL-SVO.

Suggesting Variable Order for Cylindrical Algebraic Decomposition via Reinforcement Learning

Abstract

Cylindrical Algebraic Decomposition (CAD) is one of the pillar algorithms of symbolic computation, and its worst-case complexity is double exponential to the number of variables. Researchers found that variable order dramatically affects efficiency and proposed various heuristics. The existing learning-based methods are all supervised learning methods that cannot cope with diverse polynomial sets. This paper proposes two Reinforcement Learning (RL) approaches combined with Graph Neural Networks (GNN) for Suggesting Variable Order (SVO). One is GRL-SVO(UP), a branching heuristic integrated with CAD. The other is GRLSVO(NUP), a fast heuristic providing a total order directly. We generate a random dataset and collect a real-world dataset from SMT-LIB. The experiments show that our approaches outperform state-of-the-art learning-based heuristics and are competitive with the best expert-based heuristics. Interestingly, our models show a strong generalization ability, working well on various datasets even if they are only trained on a 3-var random dataset. The source code and data are available at https://github.com/dongyuhang22/GRL-SVO.

1 Introduction

As learned in school, we know how to answer the question of whether a quadratic equation for x has a real root. For example,

where b, c are unknowns. We can answer it by checking whether the discriminant is non-negative, i.e., b 2 -4 c ≥ 0 . What if the number and degree of variables increase, and the formula involves the combination of the universal quantifier ( ∀ ), existential quantifier ( ∃ ), and logical operators (and( ∧ ), or( ∨ ), not( ¬ ))? Checking whether polynomials satisfy some mathematical constraints is a difficult problem and has puzzled mathematicians since ancient times. Until the 1930s, Alfred Tarski [1] answered the question by proving that the theory of real closed fields admits the elimination of quantifiers and gives a quantifier elimination procedure. Unfortunately, the procedure was impractical due to its non-elementary complexity. In 1975, George Collins discovered the first relatively efficient algorithm, Cylindrical Algebraic Decomposition (CAD) [2]. Currently, CAD (and variants) has

3.75

3.25

2.75

2.25

1.75

1.25

0.75

0.25

0.25

2

1

0

1

2

x

y

y

=

x

2

(a)

x

≺

y

𝑝𝑝

: (

-

5,4)

(b)

x

≺

y

(c)

y

≺

x

become one of the most fundamental algorithms in symbolic computation and is widely used in computational geometry [3], robot motion planning [4], constraint programming [5, 6, 7].

More precisely, CAD is an algorithm that eliminates (technically called project ) variables one by one and finally results in a list of regions that are sign-invariant to the polynomials (technically called cells ). CAD provides an efficient quantifier elimination in real space, thereby enabling the solution of various problems related to polynomials. For example, the space of the discriminant of the quadratic equation (i.e., b 2 -4 c ≥ 0 ) can be the combination of satisfied cells. We provide a detailed description in Appendix A. Due to its powerful analytical ability and great versatility, it is also accompanied by huge limitations. The theoretical worst complexity is double exponential to the number of variables [2].

Researchers have conducted in-depth studies on improving its efficiency. According to theoretical and practical research, there lives a very important conclusion that 'variable order in CAD can be crucially important' [8, 9, 10, 11, 12, 13, 14, 15, 16]. The selection of variable orders has a great effect on the time, memory usage as well as the number of cells in CAD. As an example, [8] introduces a category of problems where one variable order leads to a result with double exponential complexity to the number of variables, while another order yields a constant-sized result.

In this paper, we present a Graph-based Reinforcement Learning for Suggesting Variable Order (GRL-SVO) approach for CAD. It has two variants: GRL-SVO(UP) (i.e., utilizing project ) and GRL-SVO(NUP) (i.e., not utilizing project ). GRL-SVO(UP) is integrated into the CAD and can select the 'best' next projecting variable. Considering the high cost of interacting with the symbolic computation tool, we also propose a fast approach GRL-SVO(NUP), which will simulate the state transition (i.e., project ) via two rules (update rule and delete rule). It can report a total variable order before the CAD process. To evaluate the effectiveness of the models, we conduct a dataset of random polynomial sets with 3 to 9 variables and collected instances from SMT-LIB [13, 17] to form a real-world dataset. Experimental results show that our approaches outperform state-of-the-art learning-based heuristics and are competitive with the best expert-based heuristics. GRL-SVO also exhibits a strong generalization capability. The models are only trained on a 3-var random dataset, but they still work well on other datasets.

2 Background and Related Work

In this section, we briefly introduce some basic definitions of CAD [2, 18]. We classify the previous works of SVO and give an overview of the techniques used.

2.1 Cylindrical Algebraic Decomposition (CAD)

A Cylindrical Algebraic Decomposition (CAD) is a decomposition algorithm of a set of polynomials in ordered R n space resulting in finite sign-invariant regions, named cells . As shown in Figure 1a, there are 3 cells with different colors (two infinite regions and the curve), and any points in the cell lead to the same sign of y -x 2 . Let R [ x ] be the ring of polynomials in the variable vector x = [ x 1 , · · · , x n ] with coefficients in R [9].

Definition 1 (Cell) . For any finite set Q ⊆ R [ x ] , a cell of Q is defined as a maximally connected set in R n where the sign of every polynomial in Q is constant.

CAD accepts a set of polynomials and a fixed variable order and mainly consists of three running phases: project , root isolate , and lift . The project phase eliminates a variable of a polynomial set once at a time. It will result in a new projected polynomial set that carries enough information to ensure possible decomposition. After repeating calls to project , CAD constitutes a step-like (from n-1 variables to 1 variable) set of projected polynomials. The root isolate procedure isolates all roots of the univariate polynomial set, and the roots split R into some segmentations. The lift phase samples in the segmentations and assigns the sampled value to the former projected polynomial set so that the former polynomial set will become a univariate polynomial set. After repeating root isolate and lift n -1 times, CAD reconstructs the entire space via a set of cells characterized by the sample points. Since the CAD process and the project operators are not prerequisites to understand our approach, we arrange more details in Appendix A. Here, we exemplify the process and the effect of different variable orders.

Example 2.1. Consider a polynomial set { x 3 y +4 x 2 + xy, -x 2 +2 xy -1 } as in Figure 1b and 1c.

CAD process. Assume that the variable order is x ≺ y . CAD eliminates y first and results in a polynomial set { x, x 2 + 1 , x 4 + 10 x 2 + 1 } (i.e., project phase). The polynomial set has only one root x = 0 (i.e., root isolation phase). Then R will be split into three segmentations: { x < 0 , x = 0 , x > 0 } . We sample { x = -5 , x = 0 , x = 5 } and result in three different polynomial sets: {-130 y +100 , -10 y -26 } , {-1 } , and { 130 y +100 , 10 y -26 } (i.e., lift phase). Let's take the first polynomial set as an example, and it has two roots, i.e., { 10 13 , -13 5 } (i.e., root isolation phase). Then R will be split into five segmentations: { y < -13 5 , y = -13 5 , -13 5 < y < 10 13 , y = 10 13 , y > 10 13 } . As shown in Figure 1b, the sample red point ( -5 , 4) can represent a sign-invariant region, the whole shaded area (i.e., x 3 y +4 x 2 + xy < 0 ∧ -x 2 +2 xy -1 < 0 ∧ x < 0 ).

Effect of different variable orders. Figure 1b first eliminates y then x and results in 13 cells, and Figure 1c first eliminates x then y and results in 89 cells, almost seven times that of the former.

2.2 Suggesting Variable Order for Cylindrical Algebraic Decomposition

An Expert-Based (EB) heuristic is a sequence of meticulous mechanized rules. It is mainly derived from theoretical analysis or a large number of observations on practical instances and summarized by experts. The heuristics can capture the human-readable characteristics of the problem. A LearningBased (LB) heuristic will suggest an order through the scoring function or a variable selection distribution given by the learning model. It can exploit features deep inside the problem statement via high-dimensional abstraction.

Another important indicator is whether invoking project , as the project phases are time-consuming for SVO heuristics in practice. In the following, UP denotes heuristics utilizing project , and NUP denotes heuristics not utilizing project .

EB & UP. The heuristics sotd (sum of total degree) [10], and ndrr (number of distinct real roots) [12] will project utilizing all different variable orders until the polynomial sets with only one variable. Then sotd will select the order with the smallest sum of total degrees for each monomial, while ndrr will select the order with the smallest number of distinct real roots. Because of the combinatorial explosion of orders, the heuristics projecting all orders only work on the case with a small number of variables. Based on CAD complexity analysis, gmods [13] selects the variable with the lowest degree sum in the polynomial set after each project phase.

EB & NUP. The heuristics brown [9], and triangular [11] introduced a series of rules about statistical features like degree, total degree, and occurrence to distinguish the importance of variables. The heuristic chord [14] also provides an efficient algorithm based on the associated graph. It makes a variable order via perfect elimination ordering on the graph. Note that chord heuristic only works on the chordal graph. It is a special case that, after each project phase, the graph only removes the linked edges of the projected variable without changing the other components.

LB & UP. To the best of our knowledge, no heuristic should be classified into this category.

LB & NUP. The approach EMLP [15] utilizes an MLP neural network. The network takes the selected statistics of the polynomial set as input and outputs a label for variable order directly. If the

polynomial set has 3 variables, then 3! = 6 output labels are necessary for the neural network. The approach PVO [16] combines neural network and EB & NUP heuristics like brown and triangular . The neural network is trained to predict the best first variable while the EB & NUP heuristics decide other parts of orders. These kinds of heuristics work on 6 variables at most in their experiments.

According to the classification, our proposed approaches, GRL-SVO(UP) and GRL-SVO(NUP), can be categorized as LB & UP and LB & NUP, respectively.

2.3 Graph Neural Network and Reinforcement Learning

Graph Neural Networks (GNNs) are a class of deep learning models for graph-structured data. GNNs include many variants according to the characteristics of the problems, such as GCN [19], GAT [20], superGAT [21], and so on. Reinforcement Learning (RL) is an advanced machine learning paradigm where an agent learns to make decisions by interacting with its environment. It includes various frameworks, such as REINFORCE [22], DQN [23], and so on. By leveraging the expressive power of GNNs to learn complex graph structures and the adaptability of Reinforcement Learning (RL) to find optimal decision-making policies, researchers have remarkably succeeded in combinatorial algorithm design [24, 25, 26]. GRL-SVO is based on an Advantage Actor-Critic (A2C) framework [24, 27] with a Graph Network [28].

3 Method

This section starts with the problem formulation for the framework, followed by an overview and description of the graph representation and architecture. Finally, we introduce the state transition without project , which is the key technique for GRL-SVO(NUP).

3.1 Problem Formulation

We now give the formulation of SVO for CAD. Our goal is to improve CAD efficiency by suggesting a better variable order. Computation time and memory usage are important indicators, but they will be affected by random factors, such as CPU clock, usage rate, etc. As the main output of CAD, cells can be the best candidate. In order to measure the quality of the result, the number of cells is an appropriate indicator that intuitively shows the effect of CAD [29]. In theory, a large number of cells means that the partitions are fragmented compared to a small number of cells. Usually, the polynomial set generated from project phase is complex and difficult for the next phases of CAD. In practice, the number of cells also strongly correlates to the computation time and memory usage. Figures of the relation are listed in Appendix B, and we found that the computation time and memory usage increase when the number of cells increases. The objective is to minimize the number of cells N ( Q,σ ) , where Q ⊆ R [ x ] is a polynomial set with coefficients in R and σ is the given variable order, i.e., min N ( Q,σ ) .

By analyzing the input polynomial set Q , we can derive the variable order so that we ought to minimize the objective:

The difficulties of this framework mainly come from two aspects:

Huge input space. The expression of a polynomial is compressed, and any slight change in form (such as power) will change the geometric space drastically. The EB heuristics may become inefficient when encountering characteristics beyond the summarized patterns.

Huge output space. The number of variable orders and the number of variables have a factorial relationship, i.e., n variables resulting in n ! different variable orders. For example, 10 variables lead to 3628800 candidate variable orders. sotd -like, ndrr -like, and EMLP -like heuristics become impractical due to the vast number of candidate variable orders.

3.2 GRL-SVO Overview

In this paper, considering the challenges mentioned above, we propose the GRL-SVO approach, and Figure 2 shows the overall architectures. For huge input space, we compress the polynomial

…

F: Polynomial set

𝑥𝑥

3

𝑦𝑦

+

⋯

, …

GNN

Actor

Critic

𝑥𝑥

action

𝑟𝑟

reward

(F, A, X)

Environment

Projection

F

⟵

Projected Polynomial Set

Update A,X

(a) GRL-SVO(UP)

F: Polynomial set

𝑥𝑥

3

𝑦𝑦

+

⋯

, …

GNN

Actor

Critic

𝑥𝑥

action

𝑟𝑟

reward

(F, A, X)

Environment

(without Symbolic Computation Tool)

Delete Rule

Update Rule:

𝜙𝜙

⋅

,

𝑥𝑥

…

(b) GRL-SVO(NUP)

information into an associated graph [30] with embeddings, which is simple and can depict the relationship between variables. For huge output space, we utilize the neural network to predict the next best variable, and by repeating until no variables are left, the trajectory corresponds to a variable order. In detail, the actor neural network provides a distribution of actions, i.e., the choice of variables. The critic neural network scores the total order and stabilizes the training process as a state-dependent baseline. GNN encodes each variable of the current state as a high-dimensional embedding, and additional neural network components transform them into our policy. As for state transformation, GRL-SVO(UP) and GRL-SVO(NUP) are different in utilizing project . The environment of GRL-SVO(UP) projects the selected variable and reorganizes the total graph, while that of GRL-SVO(NUP) updates the graph via the update rule and delete rule.

3.3 Graph Representation

The graph of a polynomial set can be different. We introduce a graph structure that can reflect the coupling relationship between variables.

Definition 2 (Associated Graph [30]) . Given a polynomial set F , and the variable set of F , V = var ( F ) , an associated graph G F ( V, E ) of F is an undirected graph, where E = { ( x i , x j ) |∃ f ∈ F, x i , x j ∈ var ( f ) } .

In other words, if two variables appear in the same polynomial, they will have an edge.

GNNs have invariance to permutations and awareness of input sparsity [31, 32, 33]. The strength similarly applies to our work. The associated graph is pure and simple, which only retains information related to variables and 'neighbors' in the same polynomial. Note that such a graph can easily become a complete graph. For example, x 1 + x 2 + x 3 + x 4 corresponds to a complete associated graph. So, we need to distinguish nodes via rich embeddings, detailed in Appendix B. The embeddings are proposed based on former research [16, 34] and our observations. The embedding vectors of variables will be first normalized via the z-score method, i.e.,

where F 0 is the corresponding polynomial set and E j [ x ] denotes the j -th scalar of the original embedding of variable x .

The graph representation is a tuple ( F 0 , A 0 , X 0 ) where A 0 ∈ { 0 , 1 } n × n is the adjacency matrix of the associated graph and X 0 ∈ R n × d is a normalized node embedding matrix for variables.

To encode the representation, we utilize a stack of k GNN layers, Formula (5.7) in [28]. The process encodes the representations X i +1 = [ X i +1 [0]; · · · ; X i +1 [ n -1]] via

where θ i g,self and θ i g,neigh are trainable parameter matrices in R H i × H i +1 , and σ denotes the activation function. θ i g,bias is the trainable bias term in R H i +1 , and N ( u ) is the set of neighbours of variable

u . H i is the dimension of hidden channels, and H 0 = d . The layer will aggregate the local information of variables and update the embedding sequentially. Finally, we obtain the intermediate tuple ( F k , A k , X k ) .

3.4 Architecture

3.4.1 Markov Decision Process (MDP)

State Space and Action Space. GRL-SVO(UP/NUP) will suggest a variable order for any polynomial set. The state space includes the graph representation of any polynomial set, i.e., S = { G = ( F 0 , A 0 , X 0 ) } . Although it is a very large space, the state s provides sufficient statistics to evaluate actions. Action corresponds to a candidate variable to project . For a given state s , the action space is A = var ( F s ) , where F s denotes the polynomial set of current state s .

Environment. At the time t , the environment of GRL-SVO(UP) takes current state s t and selected variable (action a t ) as input and outputs a new state s t +1 via processing the projected polynomial set of CAD. That of GRL-SVO(NUP) only removes the selected variable and linked edges from the current state s t and updates embeddings via neural networks, which is detailed in Section 3.5.

Reward. The number of cells is sufficient to reflect the impact of variable order on efficiency. R ( σ | s 0 ) = -N ( F s 0 , σ ) /M denotes the reward for a given variable order σ under the initial state s 0 , i.e., the negative number of cells divided by a static normalization factor M . If the order leads to running timeout and cannot obtain the number of cells, R ( σ | s 0 ) = -1 directly. The reward of agent policy will increase as the training progresses.

3.4.2 Neural Network Architecture

Neural Network. The actor network θ a : R n × d - → R n combines an MLP and softmax layer that transforms the X k t of s t to the action distribution at time t . The action obeys the distribution, i.e.,

The critic network θ c : R n × d - → R is a combination of MeanPool and MLP layers, where MeanPool : R n × d - → R d . The critic value is defined as

Training. The parameters of GRL-SVO θ = { θ g , θ a , θ c } will go through an end-to-end training process via stochastic gradient descent method. Given initial state s , we aim to learn the parameters of a stochastic policy p θ ( σ | s ) , which assigns high probabilities to order with a small number of cells and low probabilities to order with a large number of cells. Our neural network architecture uses the chain rule to factorize the probability of a variable order σ as p θ ( σ | s ) = ∏ n t =1 p θ ( σ t | s, σ <t ) , where p θ ( σ t | s, σ <t ) = θ a ( X k t ) is current action distribution. σ t is the t -th element in the variable order σ and σ <t is the partial order from σ 1 to σ t -1 . The training objective is the expected reward, which is defined as J ( θ | s ) = E σ ∼ p θ ( ·| s ) R ( σ | s ) .

Through the well-known REINFORCE algorithm [35], the gradient of the training objective is

where c ( s ) = θ c ( X k 0 ) is the predicting critic value.

Through Monte Carlo sampling, we obtain N i.i.d polynomial sets corresponding to states s 1 , s 2 , · · · , s N ∼ S , and N variable orders σ i ∼ p θ ( ·| s i ) . So we update the parameters of neural networks via

Inference. At inference time, we generate an order via a greedy selection. For the i -th element of the order σ , we select the variable x with the maximal probability, i.e., x = arg max x ∈ V t -1 ( p ( x | s, σ <t )) ,

where V t -1 = var ( F s t -1 ) is the set of variables that have not been projected at time t . After selecting variables n -1 times, we obtain the total order σ .

3.5 State Transition without Project

We provide an LB & NUP heuristic to free from interaction with the symbolic computation tools. It simulates the project via a neural network for embedding transformation and a delete rule.

As an example, Figure 3 shows a specific case of state transition. The polynomial set changes after project phase the variable x 4 , where 3 x 2 1 x 4 -4 x 3 4 -1 is reduced to a set { x 1 -1 , x 1 +1 , x 2 1 -x 1 +1 , x 2 1 + x 1 +1 } without x 4 while the polynomial x 3 1 -4 x 2 2 x 3 +12 x 2 +3 remains unchanged. So, the embedding of neighbors of x 4 , i.e., x 1 , will change greatly while

𝑥𝑥1

𝑥𝑥2

𝑥𝑥3

𝑥𝑥4

(a)

𝑥𝑥1

𝑥𝑥2

𝑥𝑥3

(b)

that of other variables will change slightly. Besides, the projected variable should also be removed from the associated graph. Based on the aforementioned inspiring situations, we propose two rules for approximately simulating project .

At time t -1 , assume x i is the next projecting variable. We mainly consider the projected variable's influence on its 1-hop neighbor variables.

Update Rule. We update the embedding X without project for other variables x j via

where ϕ ( a, b ) = MLP ( CONCAT ( a, b )) is the neural network that simulates the project for embedding transformation.

Delete Rule. It will trivially remove x i and edges linked to x i and update A,X in the state correspondingly.

A ← -RemoveRowColumn ( A,Map ( x i )) ,

where the function Map ( x ) : V - → N , maps the variable x to the index in matrix A and X , the operation RemoveRowColumn ( A,Map ( x i )) removes the row and column of variable x i from A , and the operation RemoveRow ( X,Map ( x i )) removes the row of variable x i from X .

After such transitions, the state will feed the model defined by Section 3.4 and obtain the next projecting variable. After calling the model n -1 times, the trajectory corresponds to a variable order suggested by GRL-SVO(NUP).

4 Experiments

4.1 Setup

Implementation and Environments. We utilized PyTorch Geometric [36] for implementations of our approach. The hyper-parameters are listed in Appendix B. We utilized the NVIDIA Tesla V100 GPU for training. After the heuristics output the variable order, all instances with the given variable order are run with MAPLE 2018 on an Intel Xeon Platinum 8153 CPU (2.00GHz). The run-time limit is 900 seconds, and the time to predict a variable order is also counted.

Dataset. We utilize two CAD datasets: random and SMT-LIB . The detailed parameters of random generation and collecting methods for SMT-LIB are presented in Appendix B.

The random dataset contains 7 categories from 3-var to 9-var. We generate 20000 3-var instances and split them into training, testing, and validation sets in a ratio of 8:1:1. We pre-run all 3! = 6 variable orders and remove non-conforming instances where some variables are eliminated due to random

0

20

40

60

80

100

Epochs

1000

2000

3000

4000

5000

Number of Timeout Instances

GRL-SVO(UP)

GRL-SVO(NUP)

gmods

triangular

sotd

ndrr

EMLP

(a)

0

20

40

60

80

100

Epochs

0.35

0.30

0.25

0.20

0.15

0.10

Average Rewards of All Instances

GRL-SVO(UP)

GRL-SVO(NUP)

gmods

triangular

sotd

ndrr

EMLP

(b)

3

4

5

6

7

8

9

Number of Variables

0

200

400

600

800

Average Prediction Time (s)

gmods

triangular

sotd

ndrr

GRL-SVO(UP)

GRL-SVO(NUP)

(c)

generation, for example, x -x = 0 ; we remove the instances that are all timeout under 6 orders. The rest sets have 14990, 1871, and 1887 instances, respectively. The other categories contain 1000 instances. They are generated by randpoly of MAPLE. We also collect a SMT-LIB dataset where there are from 3-var to 9-var instances with numbers { 5908 , 1371 , 131 , 123 , 318 , 41 , 24 } .

Baselines. We compare our approach with two kinds of state-of-the-art heuristics, UP or NUP . For UP, we compare our approaches with sotd and ndrr implemented in the ProjectionCAD package [37]. Besides, we also implement gmods [13] in MAPLE. For NUP, we compare our approaches with brown , and triangular implemented in the RegularChains package [38]. EMLP approach is proposed by England [15] and its implementation only supports 3-var instances. PVO(brown) and PVO(triangular) are PVO approaches combined with EB & NUP heuristics, brown and triangular , respectively. The provided implementations [16] only support 4,5,6-var instances.

Criterion. Assume T and N denote the running time and number of cells. AVG . T and AVG . N denote the average of T and N . If an instance runs timeout, we count the maximum time (900 seconds) and the maximum number of cells of this instance solved by other heuristics into the calculation. We remove the instances that all heuristics lead to CAD timeout, as such instances can not distinguish the ability of heuristics. PT and # SI denote the variable order prediction time used by heuristics and the number of solved instances within the time limit. For UP heuristics, PT also takes project time into consideration. Except for # SI , the criteria are the smaller, the better.

Experiments. We conduct three experiments to evaluate our approaches. First of all, we only train the models on the 3-var random dataset. Then, we generalize the trained models on random datasets with up to 9 variables and the SMT-LIB dataset. Finally, we compare GRL-SVO(UP) and GRL-SVO(NUP) and discuss different application scenarios. We list ablation experiments in Appendix C, investigating the effect of features (of the initial embedding), network size, network structure, reward normalization factor M , GNN architecture, and coefficient. We also list additional results in Appendix D, including results under other criteria and performance of fine-tuning.

4.2 Results

Training GRL-SVO. Figure 4a and Figure 4b show the performance during training with the instances in the training set. GRL-SVO outperforms the other heuristics through training and shows a rapid performance improvement. The UP heuristics are better than the NUP heuristics. The inference can be more accurate because the UP heuristics obtain more information than the NUP heuristics. In the beginning, GRL-SVO(NUP) with the random initial parameters is better than GRL-SVO(UP). After training, GRL-SVO(UP) shows better performance.

Generalization. We only train on the 3-var dataset and generalize the models to the other datasets. After removing all timeout instances, there are 1876, 651, 416, 354, 349, 409, and 383 instances for the random dataset from 3-var to 9-var; for the SMT-LIB dataset, 1777, 387, 17 instances for 3-var, 4-var to 6-var and 7-var to 9-var, respectively. Table 1 shows the performance of all heuristics, and the best scores are bolded. GRL-SVO(UP) is the only LB approach with UP heuristics, achieving the best performance among most UP and NUP heuristics except for the 4-var category. GRL-SVO(NUP) also achieves competitive performance compared to other NUP heuristics. GRL-SVO also shows

competitive performance in the real-world dataset, SMT-LIB dataset. Note that although the heuristics sotd perform better than GRL-SVO(UP) on the 4-var instances, they run timeout in most instances from 5-var due to the combinatorial explosion of the number of variable orders as shown in Figure 4c. The NUP heuristics take a short prediction time, and the polylines in Figure 4c overlap. For example, an instance with 7 variables leads to 7! = 5040 variable orders to project for sotd and ndrr .

4.3 Discussion on GRL-SVO(UP/NUP)

As in Table 1, GRL-SVO(UP) performs better in # SI , AVG . T , AVG . N compared to GRLSVO(NUP). It is understandable because GRL-SVO(UP) receives more real-world information from the projected polynomial set at each step. As in Figure 4c, GRL-SVO(NUP) is faster than GRLSVO(UP). The inference time of GRL-SVO(NUP) is almost unchanged with the increase of variables, but there is an obvious increase of GRL-SVO(UP). As the number of variables grows, the time of project and interactions between GRL-SVO(UP) and symbolic computation tools will be critical. It is also the bottleneck of UP heuristics like GRL-SVO(UP), gmods , sotd , and ndrr . Therefore, the application scenarios corresponding to the two models will be different.

Internalizing GRL-SVO(UP) into the CAD process is a promising option. As project is an algorithm component of CAD, internalization will help reuse the results of projection and reduce the interaction time. GRL-SVO(NUP) is cheap and can extract information directly from polynomial representations. It might be applied to other tools that do not use the entire CAD process. As a canonical example, the automated reasoning tools like Z3 [5], YICES2 [6], CVC5[39], utilize project partially only for generating lemmas when solving non-linear arithmetic problems. At the beginning of solving, they also require a fixed variable order. For tasks that are time-critical and do not utilize full CAD in the solving process, GRL-SVO(NUP) seems to be a better option.

5 Limitations

Our graph representation cannot embed complete information on polynomials. The associated graph is simple and only shows the relationship between variables. Through selection, we conduct the embedding of graph nodes, but they still ignore plenty of minutiae information, for example, the distribution of coefficients. Besides, the lack of sufficiently large datasets is also a matter of urgency. We can generate large amounts of random data but may lack practical instances for training. Another slight limitation is the prediction time. Python and PyTorch are both heavy techniques. The prediction time of GRL-SVO shown in Figure 4c is actually not much different from other heuristics.

6 Conclusion and Future Work

In the paper, we have proposed the first RL-based approach to suggest variable order for cylindrical algebraic decomposition. It has two variants: GRL-SVO(UP) for LB & UP and GRL-SVO(NUP) for LB & NUP. GRL-SVO(UP) can suggest branching variables in the CAD process; GRL-SVO(NUP) can suggest total variable order before the CAD process. Our approaches outperform state-of-the-art learning-based heuristics and are competitive with the best expert-based heuristics. Our RL-based approaches also show a strong learning and generalization ability. Future work is to deploy our approach to practical applications, such as constructing an RL-based package for MAPLE and an algorithm component for automated reasoning tools for non-linear arithmetic solving.

References

Alfred Tarski. A Decision Method for Elementary Algebra and Geometry. In Quantifier elimination and cylindrical algebraic decomposition , pages 24-84. Springer, 1998.

George E Collins. Quantifier elimination for real closed fields by cylindrical algebraic decompostion. In Automata theory and formal languages , pages 134-183. Springer, 1975.

Mark de Berg, Otfried Cheong, Marc J. van Kreveld, and Mark H. Overmars. Computational Geometry: Algorithms and Applications, 3rd Edition . Springer, 2008.

Steven M. LaValle. Planning Algorithms . Cambridge University Press, 2006.

Dejan Jovanovic and Leonardo Mendonça de Moura. Solving Non-linear Arithmetic. In IJCAR 2012 , volume 7364 of Lecture Notes in Computer Science , pages 339-354. Springer, 2012.

Dejan Jovanovic. Solving Nonlinear Integer Arithmetic with MCSAT. In VMCAI 2017 , volume 10145 of Lecture Notes in Computer Science , pages 330-346. Springer, 2017.

Fuqi Jia, Rui Han, Pei Huang, Minghao Liu, Feifei Ma, and Jian Zhang. Improving Bit-Blasting for Nonlinear Integer Constraints. In ISSTA 2023 , pages 14-25. ACM, 2023.

Christopher W. Brown and James H. Davenport. The Complexity of Quantifier Elimination and Cylindrical Algebraic Decomposition. In ISSAC 2007 , pages 54-60. ACM, 2007.

Christopher W Brown. Companion to the Tutorial: Cylindrical Algebraic Decomposition. United States Naval Academy , 2004.

Andreas Dolzmann, Andreas Seidl, and Thomas Sturm. Efficient Projection Orders for CAD. In ISSAC 2004 , pages 111-118. ACM, 2004.

Changbo Chen, James H. Davenport, François Lemaire, Marc Moreno Maza, Bican Xia, Rong Xiao, and Yuzhen Xie. Computing the Real Solutions of Polynomial Systems with the RegularChains Library in Maple. ACM Commun. Comput. Algebra , 45(3/4):166-168, 2011.

Russell J. Bradford, James H. Davenport, Matthew England, and David J. Wilson. Optimising Problem Formulation for Cylindrical Algebraic Decomposition. In CICM 2013 , volume 7961 of Lecture Notes in Computer Science , pages 19-34. Springer, 2013.

Tereso del Río and Matthew England. New Heuristic to Choose a Cylindrical Algebraic Decomposition Variable Ordering Motivated by Complexity Analysis. In CASC 2022 , volume 13366 of Lecture Notes in Computer Science , pages 300-317. Springer, 2022.

Haokun Li, Bican Xia, Huiying Zhang, and Tao Zheng. Choosing Better Variable Orderings for Cylindrical Algebraic Decomposition via Exploiting Chordal Structure. J. Symb. Comput. , 116:324-344, 2023.

Matthew England and Dorian Florescu. Comparing Machine Learning Models to Choose the Variable Ordering for Cylindrical Algebraic Decomposition. In CICM 2019 , volume 11617 of Lecture Notes in Computer Science , pages 93-108. Springer, 2019.

Changbo Chen, Zhangpeng Zhu, and Haoyu Chi. Variable Ordering Selection for Cylindrical Algebraic Decomposition with Artificial Neural Networks. In ICMS 2020 , volume 12097 of Lecture Notes in Computer Science , pages 281-291. Springer, 2020.

Clark Barrett, Pascal Fontaine, and Cesare Tinelli. The Satisfiability Modulo Theories Library (SMT-LIB). www.SMT-LIB.org , 2016.

Dennis S. Arnon, George E. Collins, and Scott McCallum. Cylindrical Algebraic Decomposition I: The Basic Algorithm. SIAM J. Comput. , 13(4):865-877, 1984.

Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR 2017 . OpenReview.net, 2017.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In ICLR 2018 . OpenReview.net, 2018.

Dongkwan Kim and Alice Oh. How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision. In ICLR 2021 . OpenReview.net, 2021.

Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In NIPS 1999 , pages 1057-1063. The MIT Press, 1999.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with Deep Reinforcement Learning. CoRR , abs/1312.5602, 2013.

Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural Combinatorial Optimization with Reinforcement Learning. In ICLR 2017 . OpenReview.net, 2017.

Vitaly Kurin, Saad Godil, Shimon Whiteson, and Bryan Catanzaro. Can Q-Learning with Graph Networks Learn a Generalizable Branching Heuristic for a SAT Solver? In NIPS 2020 . OpenReview.net, 2020.

Yao Qin, Hua Wang, Shanwen Yi, Xiaole Li, and Linbo Zhai. A Multi-Objective Reinforcement Learning Algorithm for Deadline Constrained Scientific Workflow Scheduling in Clouds. Frontiers of Computer Science , 15:1-12, 2021.

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In ICML 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 1928-1937. JMLR.org, 2016.

William L Hamilton. Graph representation learning . Morgan & Claypool Publishers, 2020.

Russell J. Bradford, James H. Davenport, Matthew England, Scott McCallum, and David J. Wilson. Truth Table Invariant Cylindrical Algebraic Decomposition. J. Symb. Comput. , 76:1-35, 2016.

Chenqi Mou, Yang Bai, and Jiahua Lai. Chordal Graphs in Triangular Decomposition in Top-Down Style. J. Symb. Comput. , 102:108-131, 2021.

Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L. Dill. Learning a SAT Solver from Single-Bit Supervision. In ICLR 2019 . OpenReview.net, 2019.

Minghao Liu, Fan Zhang, Pei Huang, Shuzi Niu, Feifei Ma, and Jian Zhang. Learning the Satisfiability of Pseudo-Boolean Problem with Graph Neural Networks. In CP 2020 , volume 12333 of Lecture Notes in Computer Science , pages 885-898. Springer, 2020.

Quentin Cappart, Didier Chételat, Elias B. Khalil, Andrea Lodi, Christopher Morris, and Petar Velickovic. Combinatorial Optimization and Reasoning with Graph Neural Networks. In IJCAI 2021 , pages 4348-4355. ijcai.org, 2021.

Zongyan Huang, Matthew England, David J. Wilson, James H. Davenport, Lawrence C. Paulson, and James P. Bridge. Applying Machine Learning to the Problem of Choosing a Heuristic to Select the Variable Ordering for Cylindrical Algebraic Decomposition. In CICM 2014 , volume 8543 of Lecture Notes in Computer Science , pages 92-107. Springer, 2014.

Ronald J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Mach. Learn. , 8:229-256, 1992.

Matthias Fey and Jan E. Lenssen. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Matthew England, David J. Wilson, Russell J. Bradford, and James H. Davenport. Using the Regular Chains Library to Build Cylindrical Algebraic Decompositions by Projecting and Lifting. In ICMS 2014 , volume 8592 of Lecture Notes in Computer Science , pages 458-465. Springer, 2014.

François Lemaire, Marc Moreno Maza, and Yuzhen Xie. The RegularChains library in MAPLE. SIGSAM Bull. , 39(3):96-97, 2005.

Haniel Barbosa, Clark W. Barrett, Martin Brain, Gereon Kremer, Hanna Lachnitt, Makai Mann, Abdalrhman Mohamed, Mudathir Mohamed, Aina Niemetz, Andres Nötzli, Alex Ozdemir, Mathias Preiner, Andrew Reynolds, Ying Sheng, Cesare Tinelli, and Yoni Zohar. CVC5: A Versatile and Industrial-Strength SMT Solver. In TACAS 2022 , volume 13243 of Lecture Notes in Computer Science , pages 415-442. Springer, 2022.

Scott McCallum. An Improved Projection Operation for Cylindrical Algebraic Decomposition. In EUROCAL1985 , volume 204 of Lecture Notes in Computer Science , pages 277-278. Springer, 1985.

Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019 , pages 257-266. ACM, 2019.

Shyam A. Tailor, Felix L. Opolka, Pietro Liò, and Nicholas D. Lane. Adaptive filters and aggregator fusion for efficient graph convolutions. CoRR , abs/2104.01481, 2021.

Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Xavier Bresson and Thomas Laurent. Residual gated graph convnets. CoRR , abs/1711.07553, 2017.

William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs. In NIPS 2017 , pages 1024-1034, 2017.

Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 1548-1554. ijcai.org, 2021.

A Cylindrical Algebraic Decomposition

A.1 Detailed Description

A.1.1 Polynomial

N and R denote the set of natural numbers and real numbers, respectively. Let x = { x 1 , · · · , x n } be the variable set, where x 1 < x 2 < · · · < x n is the order of the variables.

A term t i is a finite production of powers of variables, i.e. t i = ∏ n j =1 x d i,j j , where d i,j ∈ N as the degree of the variable x j . We denote ∑ n j =1 d i,j as the degree of the term.

Apolynomial P ∈ R [ x ] of general form is a finite sum of terms, i.e., P = ∑ m i =1 c i t i , where c i ∈ R is coefficient of term t i . In addition, the equivalent nested form of polynomial Q ∈ R [ x 1 , · · · , x i -1 , x i ] ,

where 0 < d 1 < · · · < d m , and the coefficients a i are polynomials in R [ x 1 , · · · , x i -1 ] with a m = 0 . We denote x i as main variable , d m as degree , a m as leading coefficient of the polynomial Q . For example, given a variable order x 1 < x 2 < x 3 ,

The degree of x 1 x 2 x 2 3 , x 2 x 2 3 , x 2 3 , x 1 x 3 , x 2 x 3 are 4,3,2,2,2, while that of the polynomial p ( x 1 , x 2 , x 3 ) is 2, and leading coefficient is (( x 1 +1) x 2 ) .

A.1.2 Cylindrical Algebraic Decomposition

A Cylindrical Algebraic Decomposition (CAD) is a decomposition algorithm of a set of polynomials in ordered R n space resulting in finite sign-invariant regions, named cells . After CAD, it can query a limited set of sample points in corresponding cells. Due to the sign of each polynomial is either always positive, always negative, or always zero on any given cell, one can determine the sign of the polynomials at any point in R n using this set of sample points.

Note that computing the CAD of a set of univariate polynomials (one-dimensional CAD) is quite simple. We only need to calculate all the real roots of the polynomials, and the cells are precisely these real roots themselves along with the intervals they divide. This leads to a motivation for computing the CADof a set of polynomials with n variables, which involves recursively reducing the construction of a k -dimensional CAD to the construction of a ( k -1) -dimensional CAD, until reaching the recursive boundary of computing a one-dimensional CAD, and then constructing higher-dimensional CAD from lower-dimensional CAD continuously.

Formally, the algorithm consists of three components: projection, root isolation, and lift. A diagram is shown in Figure 5. The project phase eliminates the variables of a polynomial set P n with n variables by a strictly defined projection operator proj in a given order x 1 < x 2 < · · · < x n until only one variable x 1 left, resulting in polynomial sets P 1 , · · · , P n -1 , where P i = proj ( P i +1 ) contains only the variables x 1 , · · · , x i . The projection operator is carefully designed to ensure that the CAD of P i can be constructed from the CAD of P i -1 . Then the root isolate and lift phases are alternated successively. To make the following statement compatible, we replace the notation of P 1 with P 1 , 1 . Let N ( i ) denote the number of cells generated by P i (also equals to the number of sample points). We set N (0) to be 1. The root isolate phase will output all roots of a univariate polynomial set. When i = 1 , the roots of P 1 , 1 split R into N (1) cells. Let SP i,k , 1 ≤ k ≤ N ( i -1) denote the set of sample points of the cells generated by P i,k and SP i denote the union of SP i,k . Actually, SP i is precisely the sample points of the cells generated by P i and | SP i | = N ( i ) . The lift phase assigns each sample point s i,k of SP i to variables of P i +1 , i.e., ( x 1 , x 2 , · · · , x i ) ← s i,k , 1 ≤ k ≤ N ( i ) , to polynomials in P i +1 , resulting in univariate polynomial sets P i +1 ,k , 1 ≤ k ≤ N ( i ) . Invoking the root isolate phase will obtain each SP i +1 ,k and finally their union SP i +1 . After repeating root isolate and lift for n -1 times, we achieve the cells of P n characterized by the sample points SP n . The paper provides a concrete example of the CAD process in Example 2.1.

The projection operator proj plays a key role in the CAD process, which carries enough information to ensure that the CAD of any set of polynomials P can be constructed from the CAD of proj ( P ) .

̸

𝑃𝑃

𝑛𝑛

: {

𝑓𝑓 ∈ ℝ

[

𝑥𝑥1

,

⋯

,

𝑥𝑥

𝑛𝑛

]}

𝑃𝑃

𝑛𝑛-1

: {

𝑓𝑓 ∈ ℝ

[

𝑥𝑥1

,

⋯

,

𝑥𝑥

𝑛𝑛-1

]}

𝑃𝑃 1

: {

𝑓𝑓 ∈ ℝ

[

𝑥𝑥1

]}

⋯

𝑆𝑆𝑃𝑃

𝑛𝑛

: {

𝑝𝑝 ∈ ℝ 𝑛𝑛

}

𝑆𝑆𝑃𝑃

𝑛𝑛-1

: {

𝑝𝑝 ∈ ℝ 𝑛𝑛-1

}

𝑆𝑆𝑃𝑃 1

: {

𝑝𝑝 ∈ ℝ

}

⋯

Project

Project

Project

Lift

Lift

Lift

Root Isolation

The first projection operator that satisfies the above property is designed by George Collins[2], which is too complicated, however. Here, we introduce another classic simplified projection operator, the McCallum Projection Operator[40]. It is the default projection operator used in the ProjectionCAD package[37] and our architecture of GRL-SVO(UP).

In mathematics, the resultant is used to determine whether two polynomials have common zeros, while the discriminant is used to determine if one polynomial has repeated roots. Both these two tools are crucial in the construction of the projection operator. We first introduce the definitions of the resultant and the discriminant, and then the McCallum Projection Operator is detailed in Definition 5.

Definition 3 (Resultant) . Let f 1 , f 2 be two polynomials in R [ x 1 , . . . , x n ] . Assume that

The resultant of f 1 and f 2 with respect to x n , Res ( f 1 , f 2 , x n ) , is:

Definition 4 (Discriminant) . Let f be a polynomial in R [ x 1 , . . . , x n ] . Assume that

The discriminant of f with respect to x n , Dis ( f, x n ) , is:

Definition 5 (McCallum Projection Operator) . Let F = { f 1 , . . . , f k } be a set of polynomials in R [ x 1 , . . . , x n ] . The McCallum projection operator, proj m , is a mapping that maps F to proj m ( F ) , where proj m ( F ) is the set of polynomials in R [ x 1 , . . . , x n -1 ] defined by:

The coefficients of each polynomial in F ,

The discriminant of each polynomial in F with respect to x n ,

The resultant of any two different polynomials f i , f j in F with respect to x n .

Another important component in the CAD algorithm is the real root isolation algorithm, which accepts a set of univariate polynomials and results in all the roots (actually, the arbitrarily small intervals that contain each root) of the univariate polynomials. This can be accomplished by invoking

the subalgorithm RootIsolation multiple times and adjusting the upper and lower bounds of the initial interval for each univariate polynomial.

We provide the specifics of the algorithm RootIsolation in Algorithm 2. Before that, some necessary concepts like the sign variation, the 1-norm, and the Sturm sequence are introduced, which play a role in the algorithm RootIsolation .

Definition 6 (Sign Variation) . For a sequence of non-zero real numbers c : c 1 , c 2 , · · · , c k , the sign variation of c , V ( c ) , is:

For a sequence of univariate polynomials S : f 1 , f 2 , · · · , f k , the sign variation of S at real number a , V a ( S ) is:

where s is the real number sequence f 1 ( a ) , f 2 ( a ) , · · · , f k ( a ) .

Definition 7 (1-norm) . Given a univariate polynomial f = a m x d m + a m -1 x d m -1 + · · · + a 0 . The 1-norm of f , || f || 1 , is:

Algorithm 1 SturmSequence

Input

: f : a univariate polynomial

Output

:

sturm : the Sturm Sequence of f

1: sturm ← []

3: g ← f ′

̸

6: Append r to sturm

8: g ← r

10: end while

11: return sturm

Algorithm 2 RootIsolation

Input

:

f : a univariate polynomial

Output : ( a, b ) : f has a real root in ( a, b ) where a , b are rational numbers

10 2

10 3

10 4

Number of Cells

10 0

10 1

10 2

10 3

Time (s)

(a)

10 2

10 3

10 4

Number of Cells

10 1

10 2

10 3

10 4

10 5

Memory Usage (MB)

(b)

A.2 Case Study: Discriminant

Actually, whether x 2 + bx + c = 0 has a real root is equivalent to checking the satisfiability of ∃ x.x 2 + bx + c = 0 . Let's elaborate on why the quantified formula ∃ x.x 2 + bx + c = 0 can be transformed into an equivalent quantifier-free formula b 2 -4 c ≥ 0 via CAD techniques. By feeding the CAD algorithm with the set of polynomial { x 2 + bx + c } and an order b ≺ c ≺ x , CAD decomposes R 3 into 9 cells:

Note that the sign of the polynomial x 2 + bx + c is zero if and only if ( b, c, x ) belongs to cells:

So we know that there must exist x such that x 2 + bx + c = 0 when b 2 -4 c > 0 or b 2 -4 c = 0 , and it is impossible to find a x such that x 2 + bx + c = 0 when b 2 -4 c < 0 . So, we can conclude the quantified formula ∃ x.x 2 + bx + c = 0 is equivalent to the quantifier-free formula b 2 -4 c ≥ 0 . In fact, similar processes can be abstracted into a universal algorithm to solve the quantifier elimination problems in the real closed field. See [2] for more details.

A.3 Relation of #Cells And Efficiency

As in Figure 6, there is a strong correlation between the number of cells produced and the computation time, as well as the memory usage. When the number of cells increases, the computation time and the memory usage also increase.

B Experiment Setup

B.1 Datasets

B.1.1 Random Dataset

We generated the random polynomial set via the randpoly(vars, opts) function in MAPLE, where opts are specifying properties like coeffs, expons, terms, degree, dense, homogeneous . Table 2 lists the descriptions of parameters. We also show an example of random polynomial generation.

For example, randpoly([ x 1 , x 2 , x 3 ], terms = 4, expons = rand(0..2), coeffs = rand(-100..100)) will generate a polynomial,

where rand(a..b) is a random number generator in the range of [ a, b ]. Note that the random polynomial is sparse and non-homogeneous by default. We also ignore the parameter degree , because it is only valid in the case of dense random polynomial generation.

Table 3 lists the parameters for randpoly for generating the random dataset, where #vars = n corresponds to a list of variables [ x 1 , x 2 , · · · , x n ] . The random number for terms is generated with Python's random library outside the MAPLE script. If the polynomial produced by MAPLE lacks a constant term, one is added, ranging from -100 to 100, using the same Python library.

B.1.2 SMT-LIB Dataset

The SMT-LIB dataset is built by the instances that only involve real constraints in the QF_NRA category of the SMT-LIB [17]. Using the Python interface of Z3[5], we parse the instances to extract the polynomial sets, discarding any instances that cannot be correctly parsed. Then we categorize the processed polynomial sets based on the number of variables and finally build the SMT-LIB dataset, containing instances ranging from 3 to 9 variables with counts { 5908 , 1371 , 131 , 123 , 318 , 41 , 24 } . Furthermore, we observed that there are numerous polynomial sets in the dataset that yield the same result after factoring out the irreducible factors, which are indistinguishable from CAD algorithms. So we cluster them and only one instance of each clustering is included in the dataset, resulting in { 1777 , 387 , 17 } instances for 3-var, 4-var to 6-var and 7-var to 9-var, respectively.

B.2 Neural Networks

Table 4 lists the hyper-parameters of the GNN encoder, the networks used in GRL-SVO(NUP)'s architecture to transform the original embeddings linearly and simulate the project process, actor network, critic network, and RL architecture.

C Ablation Experiments

C.1 The Effect of Features

There are 14 different features to characterize a variable listed in Table 5. We conduct an experiment on the effect of features. We make masks for these features, where the mask will set the features that we do not care about as zero.

One-hot masks (test the effect of a single feature), for example, to test the effect of E 1 , the corresponding one-hot mask is (1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0). Multiplying with input feature will result in a feature vector with only E 1 while others are 0.

Operation masks (test the effect of different operations in features) will group features according to their operation type (maximum, sum, and proportion). Note that we treat E 1 , E 2 as sub-features utilizing sum operation.

Max: E 3 , E 5 , E 6 , E 7 , E 13 , E

14

Sum: E 1 , E 2 , E 4 , E 8 , E 9 , E 10

Prop: E 11 , E 12

Object masks (test the effect of different objects in features) will group features according to their target objects (variable, term, and polynomials).

Var: E 1 , E 3 , E 4 , E 13 , E 14

Term: E 5 , E 6 , E 7 , E 8 , E 9 , E 10 , E 12

Poly: E 2 , E 11

Because degree is a common feature that most heuristics are concerned with, testing the effect of degree is necessary. Degree masks will group features according to whether they utilize degree.

Degree: E 3 , E 4 , E 5 , E 7 , E 8 , E 9

NoDegree: E 1 , E 2 , E 6 , E 10 , E 11 , E 12 , E 13 , E 14

Table 6 and Table 7 show the results of different single, operation, object, and degree features. We can conclude that only one feature is not enough. The sum, term, and degree may be the most important factors, as using Sum/Term/Degree features will result in the largest difference in performance.

C.2 The Effect of Network Size

We conduct an experiment on the effect of network size, and the results are listed in Table 8. The number of GNN layers (for short, #G) ∈ {1, 2, 3, 4, 5} and the number of Intermediate layer features (for short, #I) ∈ {32, 64, 128, 256, 512} where Actor and Critic will keep the same proportion 2:4:1. Note that the input dimension of Actor and Critic is the same as #I. For example, if #I = 32, then the dimension of the Actor and Critic are both [32, 64, 16, 1]. Elements in the following tables are #SI in the validation set and #SI in the testing set, respectively.

C.3 The Effect of Network Structure

We conduct an experiment on the effect of network structure. We build two models: one without embedding (NO_EMB), and the other without edge (NO_EDGE). Note that the GNN updating operator we used is x ′ i = W 1 x i + W 2 ∑ j ∈N ( i ) e j,i · x j + b . As edges are not considered in NO_EDGE, actually, the operator will be x ′ i = W 1 x i + b . So NO_EDGE is equivalent to MLP. For NO_EMB, # SI = 1459 , AVG . T = 252 . 03 , AVG . N = 2874 . 91 . Table 9 shows the result of NO_EDGE on testing set. The performance of NO_EMB drops dramatically while that of NO_EDGE is good, and GRL-SVO can still outperform such models. We explore the performance of NO_EDGE(NUP) under different sizes of parameters with GRL-SVO(NUP). MLP_4_512 has twice as many parameters as ours. GRL-SVO(NUP) can outperform all MLPs as shown in Figure 7. It is the advantage of the graph structure where a variable can grasp neighbor information.

C.4 The Effect of Reward Normalization Factor ( M )

We make an ablation experiment on M that we train the models with M =10000, 20000, 50000(ours), 100000, and without M . Note that if there is no M , the reward (the number of cells) will be a relatively large integer. As shown in Table 10, M is necessary. The first number is the result of the validation set, while the second is the result of the testing set.

C.5 The Effect of GNN Architecture

We conduct experiments using different GNN architectures available in PyTorch Geometric that have similar formal parameters as the GNN architecture we used: ClusterGCNConv[41], EGConv[42],

1

10

20

30

40

50

60

70

80

90

100

Epochs

900

1000

1100

1200

1300

1400

1500

Number of Timeout Instances

GRL-SVO(NUP)

MLP_3_256

MLP_4_256

MLP_4_512

(a) On training set

1

10

20

30

40

50

60

70

80

90

100

Epochs

100

120

140

160

180

Number of Timeout Instances

GRL-SVO(NUP)

MLP_3_256

MLP_4_256

MLP_4_512

(b) On validation set

FiLmConv[42], LEConv[42], GATConv [20], GATv2Conv [43], GeneralConv [44], ResGatedGraphConv [45], SageConv [46], TransformerConv [47]. We could observe from Table 11 that each graph neural network (GNN) is capable of effectively learning this problem. Our approach is not reliant on the specific GNN structure.

C.6 The Effect of Coefficient

With regard to the choice of variable order, the current works do not consider coefficient (both Expert-Based or Learning-Based heuristics). Some experts in Symbolic Computation believe that the reason may be related to the calculation of CAD projection: CAD projection uses two polynomials to make a resultant (Definition 3) to eliminate a common variable, so it first depends on the common variable set of these two polynomials (which does not involve coefficient); secondly, the amount of calculation generally depends on the degree of public variables because the degree determines the size of the resultant matrix. For practical instances, the variation range of the coefficients is uncontrollable, which will increase the difficulty of Learning-Based design and training the neural network. The impact of the coefficient on the problem is complex. We provide some intuition through Figure 8.

(a) { x 3 y +4 x 2 + xy, -x 2 +2 xy -1 } , x ≺ y : 13 , y ≺ x : 89;

(b) { x 3 y +8 x 2 + xy, -x 2 +2 xy -1 } , x ≺ y : 13 , y ≺ x : 89;

(c) { x 3 y -4 x 2 + xy, -x 2 +2 xy -1 } , x ≺ y : 45 , y ≺ x : 125;

(d) { x 3 y +4 x 2 + xy, x 2 +2 xy -1 } , x ≺ y : 29 , y ≺ x : 101;

(e) {-x 3 y +4 x 2 + xy, -x 2 +2 xy -1 } , x ≺ y : 45 , y ≺ x : 97 .

(a)

(b)

(c)

(d)

(e)

The number of cells of (a) and (b) are the same, while (c), (d), and (e) are different. But the best variable order is the same ( x ≺ y ) in these cases. To a certain extent, in these cases, the coefficient mostly affects the number of cells. We conduct an experiment on coefficient that we have randomly modified the coefficients (in [-100, 100]) of 1000 instances randomly selected from a 3-var testing set. Since the coefficients were the only modification made, we used the original variable order generated from the unaltered instances. Our models continue to outperform other heuristics, as demonstrated by Table 12. We observe a slight decline in the performance of all heuristics, indicating that the coefficient also plays a significant role as a parameter (although it may not be the most crucial one).

D Additional Results

D.1 Results Under Other Criteria

Assume T and N denote the running time and number of cells. COMAVG . T and COMAVG . N denote the average of T and N of the instances that all heuristics solved within the time limit. Since the sotd and ndrr heuristics are not applicable to most instances with the number of variables more than 5, we remove the comparison with these two heuristics on the results of the new criteria.

There are 1325, 292, 142, 106, 84, 103, and 87 common instances that all heuristics except sotd and ndrr solved within the time limit for the random dataset from 3-var to 9-var; and for the SMT-LIB dataset, 1670, 354, 9 instances for 3-var, 4-var to 6-var and 7-var to 9-var, respectively. Table 13 shows the results and the best scores are bolded. We can observe that GRL-SVO still achieves a relatively good performance under the new criteria.

D.2 Performance of Fine-Tuning

As GRL-SVO(UP) has demonstrated strong generalization abilities and achieved the best performance on almost all datasets, there is still room for further improvement in the generalization capabilities of GRL-SVO(NUP). So, we further investigate the performance of GRL-SVO(NUP) after fine-

tuning. We conduct a case study on fine-tuning, utilizing 100 instances with 4 variables. All the hyperparameters employed during the training process were identical to those listed in Table 4.

As shown in Table 14, after fine-tuning, GRL-SVO(NUP) exhibits the best performance among all NUP methods across all categories of the random dataset. The # SI indicator for GRL-SVO(NUP) showed enhancements ranging from 1.00% to 14.38% across the 4-var to 9-var categories of the random dataset, with slight improvements also observed in the SMT-LIB dataset. Although fine-tuning can help improve the efficiency of GRL-SVO(NUP), the approach is not fit for the instances with a relatively larger number of variables, as we need to run the most variable orders of each instance to build a training dataset. So, for high-dimensional instances, how to fine-tune is a promising direction.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCylindrical algebraic decomposition (CAD) is a technique to decompose a multi-dimensional space into a finite number of cells. CAD cells are built to respect a set of polynomial constraints such that each constraint has constant truth value in each cell. As stated by the authors, the variable ordering in the CAD algorithm significantly influences the computational time, memory usage, and the number of cells. The authors used Reinforcement Learning based approaches to choose an optimal variable ordering. The authors represented a polynomial set in a graph, in addition to the embeddings formed with different indicators of each polynomial (e.g. degree statistics ), and used the number of obtained cells as a quality measure of the result. The authors used the Advantage Actor-Critic (A2C) framework to improve CAD by suggesting a better variable order. The actor neural network was represented by Graph Neural Network as they are permutation invariant.\n\nSTRENGTHS:\n- Reinforcement Learning methods are widely adopted to optimize non-differentiable objectives, and the authors demonstrate its potential in choosing the variable order for CAD algorithms .\n- The experimental results demonstrate that their method outperforms the baseline.\n\nWEAKNESSES:\nThe paper lacks some ablation studies.\n- The advantage of using a GNN and a graph representation of polynomial sets: The authors mentioned that GNNs are invariant to permutation, the results should indeed be independent from the order of the polynomial in the set, but there are simpler Neural Network that are invariant to permutation. Can we empirically check how much a GNN enhances the performance of GRL-SVO? To check this, the authors can for instance replace the GNN with an MLP performed on each polynomial embedding and then do a mean aggregation to have a learnable representation of the whole polynomial set.\n- I would also encourage authors to test different GNN architectures (e.g. GAT, GATv2, GraphSage ...) for the ablation study. \n- What is the effect of normalization factor “M” on the performance and speed of GRL-SVO?\n- More experiments should be carried out to assess robustness. \n- Code not shared to verify results.\n\nQUESTIONS:\n- What is the used architecture of the GNN? The authors mentioned “GraphConv” and cited the paper [26]. I didn’t see how the two are related to each other. If you are referring to the layer “GraphConv” in the DGL library, the adequate name in the paper is “Garph Convolution Network layer [Kipf and Welling 2017]”.\n- Also, the paper [26] is more a method to enhance the expressiveness of a GNN rather than a GNN architecture.\n\nLIMITATIONS:\n- The authors mentioned the limitations of their approaches in section 5.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
Given a curve y = x^2, it partitions the x-y plane into three sets where the sign y-x^2 is the same. On the curve y-x^2 it is zero, above the curve y = x^2, sign is positive and below the curve it is negative. Thus the polynomial has 3 cells - regions where the sign is invariant.
Given a polynomial with n-variables there is a mathematical procedure (project, root-isolate and lift) to identify cells.  The mathematical procedure may results in different cells depending upon on how the variables are ordered. The objective is to choose a variable ordering in order to minimize the number of cells generated by the mathematical procedure.

The contribution of the paper is to formulate a graph representation of the problem (the state), the action is the variable permutation and the reward is minimizing the number of cells. The  REINFORCE algorithm is used.

STRENGTHS:
1. It is nice to see that RL is being used for such problems. It is a nice and natural fit.
2. The model is trained on 3-variable set but generalizes to upto 9 variable set.

WEAKNESSES:
1. The novelty is on setting up the problem. There is not much RL novelty.
2. REINFORCE has high variance. There should be more analysis of the impact of the algorithm on the solution.

QUESTIONS:
1. The performance improves very quickly according to Figure 4. Is there an explanation. How stable are the runs of REINFORCE ?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a reinforcement learning based method for the selection of a more efficient variable order for the downstream CAD(cylindrical algebraic decomposition) task. The objective is to minimize the number of cells, a suitable metric that intuitively reflects CAD efficiency. The proposed GRL-SVP(UP/NUP) utilizes the inductive biases of GNNs to learn the relationships among the variables in the polynomial set and will output a variable order for CAD via the Actor-Critic algorithm.\n\nSummary:\nThe main novelty lies in the utilization of RL and GNN for the problem of suggesting variable order for CAD and brings the benefit of better empirical performance and generalization. It contributes to the field as the first work to try RL method for this task and it can be further improved by exploring how to encode polynomial coefficients as edge information, boost prediction time, etc. The inclusion of an interpretive analysis outlining the reasoning behind the variable order proposed by the RL approach would enrich the paper's contribution and reader's understanding.\n\nSTRENGTHS:\nPros:\n1. It effectively reframes the polynomial set as an associated graph, thereby capitalizing on the inherent advantages of GNNs such as permutation invariance and sparse input awareness. This approach may potentially unveil complex variable interrelationships that could otherwise be overlooked by traditional handcrafted heuristics.\n2. This study is the first to utilize reinforcement learning to treat this task as a reinforcement problem.\n3. Experimental results show compelling evidence for this method when compared to other heuristic methods in the past. And this approach exhibits good generalization when scaled to higher variable problems.\n4. The paper seems to be well-structured and provides sufficient detail about the experiments.\n\nWEAKNESSES:\nCons:\n1. The associated graph representation may not fully capture all information about the polynomials such as the coefficients. It only encodes relationships between variables. This could be a limitation as the coefficients in a polynomial do carry significant mathematical information. \n2. The proposed approach doesn't significantly enhance the inference time compared to existing heuristics as in Figure 4c.\n\nQUESTIONS:\n1. Could you clarify the role of polynomial coefficients within your proposed graph representation? Specifically, are these coefficients incorporated into the graph structure or the node embeddings? Furthermore, what is your perspective on the potential importance of these coefficients for determining optimal variable orderings?\n2. Could you elaborate on how GRL-SVO(UP) and GRL-SVO(NUP) are complementary to each other?\n\nLIMITATIONS:\nYes, the author has adequately addressed limitations in the designated section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new method for suggesting the variables order for Cylindrical Algebraic Decomposition (CAD) problem. Their method utilized Graph Neural Networks (GNN) and Reinforcement Learning(RL). They proposed two variants: one utilizing projection(UP) and one without projection (NUP). They test their methods on two datasets and showed that the UP method outperforms nearly all methods on nearly all tasks, while the NUP method outperforms almost all NUP method. In addition, the UP method they proposed is the first LB and UP method, according to their report.\n\nIn summary, I think this is a very creative application of RL and GNN on combinatorial optimization problems, though some improvement and ablation study can be made. The writing is very good, and the comparison to the current methods is complete and well categorized.\n\nSTRENGTHS:\n1. They proposed two algorithm: GRL-SVO(UP) and GRL-SVO(NUP), utilizing the techniques in RL and GNN to optimize the number of cells  of CAD problem for a certain variable order. I think this idea is creative and worth investigating.\n2. Their explanation to CAD and SVO(suggesting variable orders) is very clear (especially the detailed version in the appendix), and they gave many example to help people who are not familiar with CAD (like me) to quickly understand what they are doing. \n3. The experiments showed that their method outperform most of the current methods on most datasets, and they also compare the UP method and NUP method they developed.\n4. The writing is very good and the literature review is clear and complete.\n\nWEAKNESSES:\nIn general this is a good paper. Below are my personal suggestion for a better paper. I do not expect you to add more experiments during the short rebuttal period, so you should not worry about it.\n\n1. The interpretability is relatively weak. Compared to the Expert based methods, the learning-based method typically use some black-box model such as deep neural networks, GNN or RL to optimize the number of cells of CAD problem. However, more explanation and ablation study can be done, although in general, this GNN+RL method is a black box. For example, in EB methods, people often use some human-created features to suggest the variables order. Since you encode 14 human-crafted features in the graph embedding matrix, maybe you can show which features in the matrix are the most important? Or does this completely depend on the property of the set of polynomials? Also, since you use the adjacent matrix of association graph as well as the variable embeddings as inputs, what will happen if you simply input the adjacent matrix or variable embedding? Will the performance drop drastically? \n\n2. There might be overfitting when you apply a large neural networks on a small set of data. In your experiments, the NN is four-layers with hundreds of hidden size, but the data size is relatively small with only thousands and even hundreds of data. So I am not sure whether there will be overfitting in your training. I think one possible solution is to try to use a smaller neural networks to train the model and evaluate it (also you can try a larger one) to see whether the performance will drop when the size increases. Another question is, I do not see how you use the validation dataset (in Line 269, you said the training : validation : testing is 8:1:1). Typically the validation set is used to tune some hyperparameters, but I do not see in the appendix that you tuned any parameter using this set.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}