SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fuqi Jia $^{1,3*}$ , Yuhang Dong $^{2,3*}$ , Minghao Liu $^{1,3}$ , Pei Huang $^{4}$ , Feifei Ma $^{2,3\dagger}$ , and Jian Zhang $^{1,3\dagger}$
$^{1}$ State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, China
$^{2}$ Laboratory of Parallel Software and Computational Science, Institute of Software, Chinese Academy of Sciences, Beijing, China
<sup>3</sup>University of Chinese Academy of Sciences, Beijing, China
4Stanford University, Stanford, USA
{jiafq, liumh, maff, zj}@ios.ac.cn, dongyuhang22@mails.ucas.ac.cn, huangpei@stanford.edu
Cylindrical Algebraic Decomposition (CAD) is one of the pillar algorithms of symbolic computation, and its worst-case complexity is double exponential to the number of variables. Researchers found that variable order dramatically affects efficiency and proposed various heuristics. The existing learning-based methods are all supervised learning methods that cannot cope with diverse polynomial sets. This paper proposes two Reinforcement Learning (RL) approaches combined with Graph Neural Networks (GNN) for Suggesting Variable Order (SVO). One is GRL-SVO(UP), a branching heuristic integrated with CAD. The other is GRL-SVO(NUP), a fast heuristic providing a total order directly. We generate a random dataset and collect a real-world dataset from SMT-LIB. The experiments show that our approaches outperform state-of-the-art learning-based heuristics and are competitive with the best expert-based heuristics. Interestingly, our models show a strong generalization ability, working well on various datasets even if they are only trained on a 3-var random dataset. The source code and data are available at https://github.com/dongyuhang22/GRL-SVO.
As learned in school, we know how to answer the question of whether a quadratic equation for  $x$  has a real root. For example,
$$
x ^ {2} + b x + c = 0,
$$
where  $b, c$  are unknowns. We can answer it by checking whether the discriminant is non-negative, i.e.,  $b^2 - 4c \geq 0$ . What if the number and degree of variables increase, and the formula involves the combination of the universal quantifier  $(\forall)$ , existential quantifier  $(\exists)$ , and logical operators  $(\land, \lor, \neg)$ ? Checking whether polynomials satisfy some mathematical constraints is a difficult problem and has puzzled mathematicians since ancient times. Until the 1930s, Alfred Tarski [1] answered the question by proving that the theory of real closed fields admits the elimination of quantifiers and gives a quantifier elimination procedure. Unfortunately, the procedure was impractical due to its non-elementary complexity. In 1975, George Collins discovered the first relatively efficient algorithm, Cylindrical Algebraic Decomposition (CAD) [2]. Currently, CAD (and variants) has
(a)  $x\prec y$
(b)  $x\prec y$
Figure 1: Examples of CAD. Figure 1a shows cells of  $\{y - x^2\}$ . Figure 1b and 1c are CAD with different variable orders on  $\{x^3 y + 4x^2 + xy, -x^2 + 2xy - 1\}$ .
(c)  $y\prec x$
become one of the most fundamental algorithms in symbolic computation and is widely used in computational geometry [3], robot motion planning [4], constraint programming [5, 6, 7].
More precisely, CAD is an algorithm that eliminates (technically called project) variables one by one and finally results in a list of regions that are sign-invariant to the polynomials (technically called cells). CAD provides an efficient quantifier elimination in real space, thereby enabling the solution of various problems related to polynomials. For example, the space of the discriminant of the quadratic equation (i.e.,  $b^{2} - 4c \geq 0$ ) can be the combination of satisfied cells. We provide a detailed description in Appendix A. Due to its powerful analytical ability and great versatility, it is also accompanied by huge limitations. The theoretical worst complexity is double exponential to the number of variables [2].
Researchers have conducted in-depth studies on improving its efficiency. According to theoretical and practical research, there lives a very important conclusion that "variable order in CAD can be crucially important" [8, 9, 10, 11, 12, 13, 14, 15, 16]. The selection of variable orders has a great effect on the time, memory usage as well as the number of cells in CAD. As an example, [8] introduces a category of problems where one variable order leads to a result with double exponential complexity to the number of variables, while another order yields a constant-sized result.
In this paper, we present a Graph-based Reinforcement Learning for Suggesting Variable Order (GRL-SVO) approach for CAD. It has two variants: GRL-SVO(UP) (i.e., utilizing project) and GRL-SVO(NUP) (i.e., not utilizing project). GRL-SVO(UP) is integrated into the CAD and can select the "best" next projecting variable. Considering the high cost of interacting with the symbolic computation tool, we also propose a fast approach GRL-SVO(NUP), which will simulate the state transition (i.e., project) via two rules (update rule and delete rule). It can report a total variable order before the CAD process. To evaluate the effectiveness of the models, we conduct a dataset of random polynomial sets with 3 to 9 variables and collected instances from SMT-LIB [13, 17] to form a real-world dataset. Experimental results show that our approaches outperform state-of-the-art learning-based heuristics and are competitive with the best expert-based heuristics. GRL-SVO also exhibits a strong generalization capability. The models are only trained on a 3-var random dataset, but they still work well on other datasets.
In this section, we briefly introduce some basic definitions of CAD [2, 18]. We classify the previous works of SVO and give an overview of the techniques used.
2.1 Cylindrical Algebraic Decomposition (CAD)
A Cylindrical Algebraic Decomposition (CAD) is a decomposition algorithm of a set of polynomials in ordered  $\mathbb{R}^n$  space resulting in finite sign-invariant regions, named cells. As shown in Figure 1a, there are 3 cells with different colors (two infinite regions and the curve), and any points in the cell lead to the same sign of  $y - x^2$ . Let  $\mathbb{R}[x]$  be the ring of polynomials in the variable vector  $\boldsymbol{x} = [x_1, \dots, x_n]$  with coefficients in  $\mathbb{R}$  [9].
Definition 1 (Cell). For any finite set  $Q \subseteq \mathbb{R}[\pmb{x}]$ , a cell of  $Q$  is defined as a maximally connected set in  $\mathbb{R}^n$  where the sign of every polynomial in  $Q$  is constant.
CAD accepts a set of polynomials and a fixed variable order and mainly consists of three running phases: project, root isolate, and lift. The project phase eliminates a variable of a polynomial set once at a time. It will result in a new projected polynomial set that carries enough information to ensure possible decomposition. After repeating calls to project, CAD constitutes a step-like (from n-1 variables to 1 variable) set of projected polynomials. The root isolate procedure isolates all roots of the univariate polynomial set, and the roots split  $\mathbb{R}$  into some segmentations. The lift phase samples in the segmentations and assigns the sampled value to the former projected polynomial set so that the former polynomial set will become a univariate polynomial set. After repeating root isolate and lift  $n - 1$  times, CAD reconstructs the entire space via a set of cells characterized by the sample points. Since the CAD process and the project operators are not prerequisites to understand our approach, we arrange more details in Appendix A. Here, we exemplify the process and the effect of different variable orders.
Example 2.1. Consider a polynomial set  $\{x^3 y + 4x^2 + xy, -x^2 + 2xy - 1\}$  as in Figure 1b and 1c.
CAD process. Assume that the variable order is  $x \prec y$ . CAD eliminates  $y$  first and results in a polynomial set  $\{x, x^2 + 1, x^4 + 10x^2 + 1\}$  (i.e., project phase). The polynomial set has only one root  $x = 0$  (i.e., root isolation phase). Then  $\mathbb{R}$  will be split into three segmentations:  $\{x < 0, x = 0, x > 0\}$ . We sample  $\{x = -5, x = 0, x = 5\}$  and result in three different polynomial sets:  $\{-130y + 100, -10y - 26\}$ ,  $\{-1\}$ , and  $\{130y + 100, 10y - 26\}$  (i.e., lift phase). Let's take the first polynomial set as an example, and it has two roots, i.e.,  $\{\frac{10}{13}, -\frac{13}{5}\}$  (i.e., root isolation phase). Then  $\mathbb{R}$  will be split into five segmentations:  $\{y < -\frac{13}{5}, y = -\frac{13}{5}, -\frac{13}{5} < y < \frac{10}{13}, y = \frac{10}{13}, y > \frac{10}{13}\}$ . As shown in Figure 1b, the sample red point  $(-5, 4)$  can represent a sign-invariant region, the whole shaded area (i.e.,  $x^3y + 4x^2 + xy < 0 \wedge -x^2 + 2xy - 1 < 0 \wedge x < 0$ ).
Effect of different variable orders. Figure 1b first eliminates  $y$  then  $x$  and results in 13 cells, and Figure 1c first eliminates  $x$  then  $y$  and results in 89 cells, almost seven times that of the former.
2.2 Suggesting Variable Order for Cylindrical Algebraic Decomposition
An Expert-Based (EB) heuristic is a sequence of meticulous mechanized rules. It is mainly derived from theoretical analysis or a large number of observations on practical instances and summarized by experts. The heuristics can capture the human-readable characteristics of the problem. A Learning-Based (LB) heuristic will suggest an order through the scoring function or a variable selection distribution given by the learning model. It can exploit features deep inside the problem statement via high-dimensional abstraction.
Another important indicator is whether invoking project, as the project phases are time-consuming for SVO heuristics in practice. In the following, UP denotes heuristics utilizing project, and NUP denotes heuristics not utilizing project.
EB & UP. The heuristics sotd (sum of total degree) [10], and ndrr (number of distinct real roots) [12] will project utilizing all different variable orders until the polynomial sets with only one variable. Then sotd will select the order with the smallest sum of total degrees for each monomial, while ndrr will select the order with the smallest number of distinct real roots. Because of the combinatorial explosion of orders, the heuristics projecting all orders only work on the case with a small number of variables. Based on CAD complexity analysis, gmods [13] selects the variable with the lowest degree sum in the polynomial set after each project phase.
EB & NUP. The heuristics brown [9], and triangular [11] introduced a series of rules about statistical features like degree, total degree, and occurrence to distinguish the importance of variables. The heuristic chord [14] also provides an efficient algorithm based on the associated graph. It makes a variable order via perfect elimination ordering on the graph. Note that chord heuristic only works on the chordal graph. It is a special case that, after each project phase, the graph only removes the linked edges of the projected variable without changing the other components.
LB & UP. To the best of our knowledge, no heuristic should be classified into this category.
LB & NUP. The approach EMLP [15] utilizes an MLP neural network. The network takes the selected statistics of the polynomial set as input and outputs a label for variable order directly. If the
A polynomial set has 3 variables, then  $3! = 6$  output labels are necessary for the neural network. The approach PVO [16] combines neural network and EB & NUP heuristics like brown and triangular. The neural network is trained to predict the best first variable while the EB & NUP heuristics decide on other parts of orders. These kinds of heuristics work on 6 variables at most in their experiments.
According to the classification, our proposed approaches, GRL-SVO(UP) and GRL-SVO(NUP), can be categorized as LB & UP and LB & NUP, respectively.
2.3 Graph Neural Network and Reinforcement Learning
Graph Neural Networks (GNNs) are a class of deep learning models for graph-structured data. GNNs include many variants according to the characteristics of the problems, such as GCN [19], GAT [20], superGAT [21], and so on. Reinforcement Learning (RL) is an advanced machine learning paradigm where an agent learns to make decisions by interacting with its environment. It includes various frameworks, such as REINFORCE [22], DQN [23], and so on. By leveraging the expressive power of GNNs to learn complex graph structures and the adaptability of Reinforcement Learning (RL) to find optimal decision-making policies, researchers have remarkably succeeded in combinatorial algorithm design [24, 25, 26]. GRL-SVO is based on an Advantage Actor-Critic (A2C) framework [24, 27] with a Graph Network [28].
This section starts with the problem formulation for the framework, followed by an overview and description of the graph representation and architecture. Finally, we introduce the state transition without project, which is the key technique for GRL-SVO(NUP).
We now give the formulation of SVO for CAD. Our goal is to improve CAD efficiency by suggesting a better variable order. Computation time and memory usage are important indicators, but they will be affected by random factors, such as CPU clock, usage rate, etc. As the main output of CAD, cells can be the best candidate. In order to measure the quality of the result, the number of cells is an appropriate indicator that intuitively shows the effect of CAD [29]. In theory, a large number of cells means that the partitions are fragmented compared to a small number of cells. Usually, the polynomial set generated from project phase is complex and difficult for the next phases of CAD. In practice, the number of cells also strongly correlates with the computation time and memory usage. Figures of the relation are listed in Appendix B, and we found that the computation time and memory usage increase when the number of cells increases. The objective is to minimize the number of cells  $N(Q,\sigma)$ , where  $Q\subseteq \mathbb{R}[x]$  is a polynomial set with coefficients in  $\mathbb{R}$  and  $\sigma$  is the given variable order, i.e.,  $\min N(Q,\sigma)$ .
By analyzing the input polynomial set  $Q$ , we can derive the variable order so that we ought to minimize the objective:
$$
\min  N (Q, \sigma (Q)).
$$
The difficulties of this framework mainly come from two aspects:
- Huge input space. The expression of a polynomial is compressed, and any slight change in form (such as power) will change the geometric space drastically. The EB heuristics may become inefficient when encountering characteristics beyond the summarized patterns.
- Huge output space. The number of variable orders and the number of variables have a factorial relationship, i.e.,  $n$  variables resulting in  $n!$  different variable orders. For example, 10 variables lead to 3628800 candidate variable orders. sotd-like, ndrr-like, and EMLP-like heuristics become impractical due to the vast number of candidate variable orders.
3.2 GRL-SVO Overview
In this paper, considering the challenges mentioned above, we propose the GRL-SVO approach, and Figure 2 shows the overall architectures. For huge input space, we compress the polynomial
(a) GRL-SVO(UP)
(b) GRL-SVO(NUP)
Figure 2: The architecture of GRL-SVO(UP) and GRL-SVO(NUP) where  $\phi(\cdot, x) = MLP(CONCAT(\cdot, x))$  for updating the embedding for the neighbours of  $x$ . The dashed lines represent that it will be only utilized in training mode.
information into an associated graph [30] with embeddings, which is simple and can depict the relationship between variables. For huge output space, we utilize the neural network to predict the next best variable, and by repeating until no variables are left, the trajectory corresponds to a variable order. In detail, the actor neural network provides a distribution of actions, i.e., the choice of variables. The critic neural network scores the total order and stabilizes the training process as a state-dependent baseline. GNN encodes each variable of the current state as a high-dimensional embedding, and additional neural network components transform them into our policy. As for state transformation, GRL-SVO(UP) and GRL-SVO(NUP) are different in utilizing project. The environment of GRL-SVO(UP) projects the selected variable and reorganizes the total graph, while that of GRL-SVO(NUP) updates the graph via the update rule and delete rule.
3.3 Graph Representation
The graph of a polynomial set can be different. We introduce a graph structure that can reflect the coupling relationship between variables.
Definition 2 (Associated Graph [30]). Given a polynomial set  $F$ , and the variable set of  $F$ ,  $V = \text{var}(F)$ , an associated graph  $G_{F}(V, E)$  of  $F$  is an undirected graph, where  $E = \{(x_{i}, x_{j}) | \exists f \in F, x_{i}, x_{j} \in \text{var}(f)\}$ .
In other words, if two variables appear in the same polynomial, they will have an edge.
GNNs have invariance to permutations and awareness of input sparsity [31, 32, 33]. The strength similarly applies to our work. The associated graph is pure and simple, which only retains information related to variables and "neighbors" in the same polynomial. Note that such a graph can easily become a complete graph. For example,  $x_{1} + x_{2} + x_{3} + x_{4}$  corresponds to a complete associated graph. So, we need to distinguish nodes via rich embeddings, detailed in Appendix B. The embeddings are proposed based on former research [16, 34] and our observations. The embedding vectors of variables will be first normalized via the z-score method, i.e.,
$$
E _ {j} ^ {\prime} [ x ] = \frac {E _ {j} [ x ] - m e a n (\{E _ {j} [ v ] \})}{s t d (\{E _ {j} [ v ] \})}, v \in v a r (F ^ {0}),
$$
where  $F^0$  is the corresponding polynomial set and  $E_j[x]$  denotes the  $j$ -th scalar of the original embedding of variable  $x$ .
The graph representation is a tuple  $(F^0, A^0, X^0)$  where  $A^0 \in \{0, 1\}^{n \times n}$  is the adjacency matrix of the associated graph and  $X^0 \in \mathbb{R}^{n \times d}$  is a normalized node embedding matrix for variables.
To encode the representation, we utilize a stack of  $k$  GNN layers, Formula (5.7) in [28]. The process encodes the representations  $X^{i + 1} = [X^{i + 1}[0];\dots ;X^{i + 1}[n - 1]]$  via
$$
X ^ {i + 1} [ u ] = \sigma (X ^ {i} [ u ] \cdot \theta_ {g, s e l f} ^ {i} + \sum_ {v \in \mathcal {N} (u)} X ^ {i} [ v ] \cdot \theta_ {g, n e i g h} ^ {i} + \theta_ {g, b i a s} ^ {i}),
$$
where  $\theta_{g, self}^{i}$  and  $\theta_{g, neighbor}^{i}$  are trainable parameter matrices in  $\mathbb{R}^{H^i \times H^{i+1}}$ , and  $\sigma$  denotes the activation function.  $\theta_{g, bias}^{i}$  is the trainable bias term in  $\mathbb{R}^{H^{i+1}}$ , and  $\mathcal{N}(u)$  is the set of neighbours of variable
$u$ .  $H^{i}$  is the dimension of hidden channels, and  $H^0 = d$ . The layer will aggregate the local information of variables and update the embedding sequentially. Finally, we obtain the intermediate tuple  $(F^k,A^k,X^k)$ .
3.4 Architecture
3.4.1 Markov Decision Process (MDP)
State Space and Action Space. GRL-SVO(UP/NUP) will suggest a variable order for any polynomial set. The state space includes the graph representation of any polynomial set, i.e.,  $S = \{G = (F^0, A^0, X^0)\}$ . Although it is a very large space, the state  $s$  provides sufficient statistics to evaluate actions. Action corresponds to a candidate variable to project. For a given state  $s$ , the action space is  $\mathcal{A} = \text{var}(F_s)$ , where  $F_s$  denotes the polynomial set of current state  $s$ .
Environment. At the time  $t$ , the environment of GRL-SVO(UP) takes current state  $s^t$  and selected variable (action  $a^t$ ) as input and outputs a new state  $s^{t+1}$  via processing the projected polynomial set of CAD. That of GRL-SVO(NUP) only removes the selected variable and linked edges from the current state  $s^t$  and updates embeddings via neural networks, which is detailed in Section 3.5.
Reward. The number of cells is sufficient to reflect the impact of variable order on efficiency.  $R(\sigma | s^0) = -N(F_{s^0}, \sigma) / M$  denotes the reward for a given variable order  $\sigma$  under the initial state  $s^0$ , i.e., the negative number of cells divided by a static normalization factor  $M$ . If the order leads to running timeout and cannot obtain the number of cells,  $R(\sigma | s^0) = -1$  directly. The reward of agent policy will increase as the training progresses.
3.4.2 Neural Network Architecture
Neural Network. The actor network  $\theta_{a}:\mathbb{R}^{n\times d}\to \mathbb{R}^{n}$  combines an MLP and softmax layer that transforms the  $X_{t}^{k}$  of  $s^t$  to the action distribution at time  $t$ . The action obeys the distribution, i.e.,
$$
a ^ {t} \sim \theta_ {a} \left(X _ {t} ^ {k}\right) = \text {s o f t m a x} \left(M L P \left(X _ {t} ^ {k}\right)\right).
$$
The critic network  $\theta_c: \mathbb{R}^{n \times d} \to \mathbb{R}$  is a combination of MeanPool and MLP layers, where MeanPool:  $\mathbb{R}^{n \times d} \to \mathbb{R}^d$ . The critic value is defined as
$$
\theta_ {c} \left(X _ {0} ^ {k}\right) = M L P \left(M e a n P o o l \left(X _ {0} ^ {k}\right)\right).
$$
**Training.** The parameters of GRL-SVO  $\theta = \{\theta_g, \theta_a, \theta_c\}$  will go through an end-to-end training process via stochastic gradient descent method. Given initial state  $s$ , we aim to learn the parameters of a stochastic policy  $p_{\theta}(\sigma | s)$ , which assigns high probabilities to order with a small number of cells and low probabilities to order with a large number of cells. Our neural network architecture uses the chain rule to factorize the probability of a variable order  $\sigma$  as  $p_{\theta}(\sigma | s) = \prod_{t=1}^{n} p_{\theta}(\sigma^t | s, \sigma^{< t})$ , where  $p_{\theta}(\sigma^t | s, \sigma^{< t}) = \theta_a(X_t^k)$  is current action distribution.  $\sigma^t$  is the  $t$ -th element in the variable order  $\sigma$  and  $\sigma^{< t}$  is the partial order from  $\sigma^1$  to  $\sigma^{t-1}$ . The training objective is the expected reward, which is defined as  $J(\theta | s) = \mathbb{E}_{\sigma \sim p_{\theta}(\cdot | s)} R(\sigma | s)$ .
Through the well-known REINFORCE algorithm [35], the gradient of the training objective is
$$
\nabla_ {\theta} J (\theta | s) = \mathbb {E} _ {\sigma \sim p _ {\theta} (\cdot | s)} [ (R (\sigma | s) - c (s)) \nabla_ {\theta} \log p _ {\theta} (\sigma | s) ],
$$
where  $c(s) = \theta_c(X_0^k)$  is the predicting critic value.
Through Monte Carlo sampling, we obtain  $N$  i.i.d polynomial sets corresponding to states  $s_1,s_2,\dots ,s_N\sim \mathcal{S}$ , and  $N$  variable orders  $\sigma_{i}\sim p_{\theta}(\cdot |s_{i})$ . So we update the parameters of neural networks via
$$
\theta_ {a} \gets \theta_ {a} + \frac {1}{N} \sum_ {i = 1} ^ {N} (R (\sigma_ {i} | s _ {i}) - c (s _ {i})) \nabla_ {\theta} l o g p _ {\theta} (\sigma_ {i} | s _ {i}),
$$
$$
\theta_ {c} \gets \theta_ {c} + \frac {1}{N} \sum_ {i = 1} ^ {N} (R (\sigma_ {i} | s _ {i}) - c (s _ {i})) \nabla_ {\theta} c (s _ {i}).
$$
Inference. At inference time, we generate an order via a greedy selection. For the  $i$ -th element of the order  $\sigma$ , we select the variable  $x$  with the maximal probability, i.e.,  $x = \arg \max_{x \in V_{t-1}} (p(x|s, \sigma^{<t}))$ ,
where  $V_{t-1} = \text{var}(F_{s^{t-1}})$  is the set of variables that have not been projected at time  $t$ . After selecting variables  $n-1$  times, we obtain the total order  $\sigma$ .
3.5 State Transition without Project
We provide an LB & NUP heuristic to free from interaction with the symbolic computation tools. It simulates the project via a neural network for embedding transformation and a delete rule.
As an example, Figure 3 shows a specific case of state transition. The polynomial set changes after project phase the variable  $x_{4}$  where  $3x_{1}^{2}x_{4} - 4x_{4}^{3} - 1$  is reduced to a set  $\{x_{1} - 1, x_{1} + 1, x_{1}^{2} - x_{1} + 1, x_{1}^{2} + x_{1} + 1\}$  without  $x_{4}$  while the polynomial  $x_{1}^{3} - 4x_{2}^{2}x_{3} + 12x_{2} + 3$  remains unchanged. So, the embedding of neighbors of  $x_{4}$ , i.e.,  $x_{1}$ , will change greatly while