SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tom Coates
180 Queen's Gate
London, SW7 2AZ
t.coates@imperial.ac.uk
Alexander M. Kasprzyk
Nottingham, NG7 2RD
a.m.kasprzyk@nottingham.ac.uk
Sara Veneziale*
s.veneziale21@imperial.ac.uk
Algebraic varieties are the geometric shapes defined by systems of polynomial equations; they are ubiquitous across mathematics and science. Amongst these algebraic varieties are  $\mathbb{Q}$ -Fano varieties: positively curved shapes which have  $\mathbb{Q}$ -factorial terminal singularities.  $\mathbb{Q}$ -Fano varieties are of fundamental importance in geometry as they are 'atomic pieces' of more complex shapes – the process of breaking a shape into simpler pieces in this sense is called the Minimal Model Programme.
Despite their importance, the classification of  $\mathbb{Q}$ -Fano varieties remains unknown. In this paper we demonstrate that machine learning can be used to understand this classification. We focus on eight-dimensional positively-curved algebraic varieties that have toric symmetry and Picard rank two, and develop a neural network classifier that predicts with  $95\%$  accuracy whether or not such an algebraic variety is  $\mathbb{Q}$ -Fano. We use this to give a first sketch of the landscape of  $\mathbb{Q}$ -Fano varieties in dimension eight.
How the neural network is able to detect  $\mathbb{Q}$ -Fano varieties with such accuracy remains mysterious, and hints at some deep mathematical theory waiting to be uncovered. Furthermore, when visualised using the quantum period, an invariant that has played an important role in recent theoretical developments, we observe that the classification as revealed by ML appears to fall within a bounded region, and is stratified by the Fano index. This suggests that it may be possible to state and prove conjectures on completeness in the future.
Inspired by the ML analysis, we formulate and prove a new global combinatorial criterion for a positively curved toric variety of Picard rank two to have terminal singularities. Together with the first sketch of the landscape of  $\mathbb{Q}$ -Fano varieties in higher dimensions, this gives strong new evidence that machine learning can be an essential tool in developing mathematical conjectures and accelerating theoretical discovery.
Systems of polynomial equations occur throughout mathematics and science; see e.g. [3, 17, 18, 35]. Solutions of these systems define shapes called algebraic varieties. Depending on the equations involved, algebraic varieties can be smooth (as in Figure 1a) or have singularities (as in Figures 1b and 1c). In this paper we show that machine learning methods can detect a class of singularities called terminal singularities.
(a)  $x^{2} + y^{2} = z^{2} + 1$
(b)  $x^{2} + y^{2} = z^{2}$
(c)  $x^{2} + y^{2} = z^{3}$
Figure 1: Algebraic varieties in  $\mathbb{R}^3$  with different defining equations.
A key class of algebraic varieties are Fano varieties: positively curved shapes that are basic building blocks in algebraic geometry. Fano varieties are 'atomic pieces' of more complex shapes, in the sense of the Minimal Model Programme [7, 25, 26]. Running the Minimal Model Programme – that is, breaking an algebraic variety  $X$  into atomic pieces – involves making birational transformations of  $X$ . These are modifications on subsets with zero volume (and codimension at least one), and can either introduce or remove singularities. The building blocks that emerge from this process are not necessarily smooth: they satisfy a weaker condition called  $\mathbb{Q}$ -factoriality, $^2$  and can have mild singularities called terminal singularities [38]. Fano varieties that are  $\mathbb{Q}$ -factorial and have terminal singularities are called  $\mathbb{Q}$ -Fano varieties.
The classification of  $\mathbb{Q}$ -Fano varieties is therefore a long-standing problem of great importance [4, 15, 27, 33, 34] – one can think of this as building a Periodic Table for geometry. But, despite more than a century of study, very little is known. In what follows we exploit the fact that machine learning can detect terminal singularities to give the first sketch of part of the classification of higher-dimensional  $\mathbb{Q}$ -Fano varieties.
We probe the classification of  $\mathbb{Q}$ -Fano varieties using a class of highly-symmetrical shapes called toric varieties. (For example, the algebraic varieties pictured in Figure 1 are toric varieties.) Toric varieties are particularly suitable for computation and machine learning, because their geometric properties are encoded by simple combinatorial objects. We consider Fano toric varieties of Picard rank two. These can be encoded using a  $2 \times N$  matrix of non-negative integers called the weight matrix; here the dimension of the toric variety is  $N - 2$ .
To determine whether such a toric variety  $X$  is a  $\mathbb{Q}$ -Fano variety we need to check whether  $X$  is  $\mathbb{Q}$ -factorial, and whether the singularities of  $X$  are terminal. Checking  $\mathbb{Q}$ -factoriality from the weight matrix of  $X$  turns out to be straightforward (see §3) but checking terminality is extremely challenging. This is because there is no satisfactory theoretical understanding of the problem. We lack a global criterion for detecting terminality in terms of weight data (such as [24] in a simpler setting) and so have to fall back on first enumerating all the singularities to analyse, and then checking terminality for each singularity. Each step is a challenging problem in discrete geometry: the first step involves building a different combinatorial object associated to the  $n$ -dimensional toric variety  $X$ , which is a collection of cones in  $\mathbb{R}^n$  called the fan  $\Sigma(X)$ ; the second step involves checking for various cones in the fan whether or not they contain lattice points on or below a certain hyperplane. To give a sense of the difficulty of the computations involved, generating and post-processing our dataset of 10 million toric varieties in dimension eight took around 30 CPU years.
An algebraic variety  $X$  is  $\mathbb{Q}$ -factorial if it is normal and, in addition, for each rank-one reflexive sheaf  $E$  on  $X$ , some tensor power of  $E$  is a line bundle. This implies that the dimension of the singular locus in  $X$  is at most  $\dim X - 2$ , and that some tensor power of the canonical sheaf (of top-degree differential forms) is a line bundle.
To overcome this difficulty, and hence to begin to investigate the classification of  $\mathbb{Q}$ -Fano varieties in dimension eight, we used supervised machine learning. We trained a feed-forward neural network classifier on a balanced dataset of 5 million examples; these are eight-dimensional  $\mathbb{Q}$ -factorial Fano toric varieties of Picard rank two, of which 2.5 million are terminal and 2.5 million non-terminal. Testing on a further balanced dataset of 5 million examples showed that the neural network classifies such toric varieties as terminal or non-terminal with an accuracy of  $95\%$ . This high accuracy allowed us to rapidly generate many additional examples that are with high probability  $\mathbb{Q}$ -Fano varieties – that is, examples that the classifier predicts have terminal singularities. This ML-assisted generation step is much more efficient: generating 100 million examples in dimension eight took less than 120 CPU hours.
The fact that the ML classifier can detect terminal singularities with such high accuracy suggests that there is new mathematics waiting to be discovered here – there should be a simple criterion in terms of the weight matrix to determine whether or not a toric variety  $X$  has terminal singularities. In §5 we take the first steps in this direction, giving in Algorithm 1 a new method to check terminality directly from the weight matrix, for toric varieties of Picard rank two. A proof of correctness is given in the Supplementary Material. This new algorithm is fifteen times faster than the naive approach that we used to generate our labelled dataset, but still several orders of magnitude slower than the neural network classifier. We believe that this is not the end of the story, and that the ML results suggest that a simpler criterion exists. Note that the neural network classifier cannot be doing anything analogous to Algorithm 1: the algorithm relies on divisibility relations between entries of the weight matrix (GCDs etc.) that are not visible to the neural network, as they are destroyed by the rescaling and standardisation that is applied to the weights before they are fed to the classifier.
In §6 we use the ML-assisted dataset of 100 million examples to begin to explore the classification of  $\mathbb{Q}$ -Fano varieties in dimension eight. We visualise the dataset using the regularized quantum period, an invariant that has played an important role in recent theoretical work on  $\mathbb{Q}$ -Fano classification, discovering that an appropriate projection of the data appears to fill out a wedge-shaped region bounded by two straight lines. This visualisation suggests some simple patterns in the classification: for example, the distance from one edge of the wedge appears to be determined by the Fano index of the variety.
Our work is further evidence that machine learning can be an indispensable tool for generating and guiding mathematical understanding. The neural network classifier led directly to Algorithm 1, a new theoretical result, by revealing that the classification problem was tractable and thus there was probably new mathematics waiting to be found. This is part of a new wave of application of artificial intelligence to pure mathematics [10, 14, 16, 20, 39-41], where machine learning methods drive theorem discovery.
A genuinely novel contribution here, though, is the use of machine learning for data generation and data exploration in pure mathematics. Sketching the landscape of higher-dimensional  $\mathbb{Q}$ -Fano varieties using traditional methods would be impossible with the current theoretical understanding, and prohibitively expensive using the current exact algorithms. Training a neural network classifier however, allows us to explore this landscape easily - a landscape that is unreachable with current mathematical tools.
Why dimension eight? We chose to work with eight-dimensional varieties for several reasons. It is important to distance ourselves from the surface case (dimension two), where terminality is a trivial condition. A two-dimensional algebraic variety has terminal singularities if and only if it is smooth. On the other hand, we should consider a dimension where we can generate a sufficient amount of data for machine learning (the analogue of our dataset in dimension three, for example, contains only 34 examples [23]) and where we can generate enough data to meaningfully probe the classification. Moreover, we work in Picard rank two because there already exists a fast combinatorial formula to check terminality in rank one [24]; Picard rank two is the next natural case to consider.
Data and code availability The datasets underlying this work and the code used to generate them are available from Zenodo under a CC0 license [11]. Data generation and post-processing was carried out using the computational algebra system Magma V2.27-3 [5]. The machine learning model was built using PyTorch v1.13.1 [36] and scikit-learn v1.1.3 [37]. All code used and trained models are available from BitBucket under an MIT licence [12].
2 Mathematical background
The prototypical example of a Fano variety is projective space  $\mathbb{P}^{N - 1}$ , which can be thought of as the quotient of  $\mathbb{C}^N\setminus \{\mathbf{0}\}$  by  $\mathbb{C}^{\times}$  acting as follows:
$$
\lambda \cdot (z _ {1}, \dots , z _ {N}) = (\lambda z _ {1}, \dots , \lambda z _ {N})
$$
Fano toric varieties of Picard rank two arise similarly. They can be constructed as the quotient of  $\mathbb{C}^N\setminus S$ , where  $S$  is a union of subspaces, by an action of  $(\mathbb{C}^{\times})^{2}$ . This action, and the union of subspaces  $S$ , is encoded by a weight matrix:
$$
\left[ \begin{array}{l l l} a _ {1} & \dots & a _ {N} \\ b _ {1} & \dots & b _ {N} \end{array} \right] \tag {2.1}
$$
Here we assume that all  $(a_i, b_i) \in \mathbb{Z}^2 \setminus \{\mathbf{0}\}$  lie in a strictly convex cone  $C \subset \mathbb{R}^2$ . The action is
$$
(\lambda , \mu) \cdot (z _ {1}, \dots , z _ {N}) = (\lambda^ {a _ {1}} \mu^ {b _ {1}} z _ {1}, \dots , \lambda^ {a _ {N}} \mu^ {b _ {N}} z _ {N})
$$
and  $S = S_{+} \cup S_{-}$  is the union of subspaces  $S_{+}$  and  $S_{-}$ , where
$$
S _ {+} = \left\{\left(z _ {1}, \dots , z _ {N}\right) \mid z _ {i} = 0 \text {i f} b _ {i} / a _ {i} > b / a \right\} \tag {2.2}
$$
$$
S _ {-} = \left\{\left(z _ {1}, \dots , z _ {N}\right) \mid z _ {i} = 0 \text {i f} b _ {i} / a _ {i} <   b / a \right\}
$$
and  $a = \sum_{i=1}^{N} a_i$ ,  $b = \sum_{i=1}^{N} b_i$ : see [6]. The quotient  $X = (\mathbb{C}^N \setminus S) / (\mathbb{C}^\times)^2$  is an algebraic variety of dimension  $N - 2$ . We assume in addition that both  $S_+$  and  $S_-$  have dimension at least two; this implies that the second Betti number of  $X$  is two, that is,  $X$  has Picard rank two.
Since we have insisted that all columns  $(a_i, b_i)$  lie in a strictly convex cone  $C$ , we can always permute columns and apply an  $\mathrm{SL}_2(\mathbb{Z})$  transformation to the weight matrix to obtain a matrix in standard form:
$$
\left[ \begin{array}{c c c c} a _ {1} & a _ {2} & \dots & a _ {N} \\ 0 & b _ {2} & \dots & b _ {N} \end{array} \right] \tag {2.3}
$$
where all entries are non-negative, the columns are cyclically ordered anticlockwise, and  $a_{N} < b_{N}$ . This transformation corresponds to renumbering the co-ordinates of  $\mathbb{C}^N$  and reparametrising the torus  $(\mathbb{C}^{\times})^{2}$  that acts, and consequently leaves the quotient variety  $X$  that we construct unchanged.
We will consider weight matrices (2.1) that satisfy an additional condition called being well-formed. An  $r \times N$  weight matrix is called standard if the greatest common divisor of its  $r \times r$  minors is one, and is well-formed if every submatrix formed by deleting a column is standard [2]. Considering only well-formed weight matrices guarantees that a toric variety determines and is determined by its weight matrix, uniquely up to  $\mathrm{SL}_r(\mathbb{Z})$ -transformation.
Testing terminality As mentioned in the introduction, an  $n$ -dimensional toric variety  $X$  determines a collection  $\Sigma(X)$  of cones in  $\mathbb{R}^n$  called the fan of  $X$ . A toric variety is completely determined by its fan. The process of determining the fan  $\Sigma(X)$  from the weight matrix (2.1) is explained in the Supplementary Material; this is a challenging combinatorial calculation. In the fan  $\Sigma(X)$ , the one-dimensional cones are called rays. For a Fano toric variety  $X$ , taking the convex hull of the first lattice point on each ray defines a convex polytope  $P$ , and  $X$  has terminal singularities if and only if the only lattice points in  $P$  are the origin and the vertices. Verifying this is a conceptually straightforward but computationally challenging calculation in integer linear programming.
3 Data generation
We generated a balanced, labelled dataset of ten million  $\mathbb{Q}$ -factorial Fano toric varieties of Picard rank two and dimension eight. These varieties are encoded, as described above, by weight matrices. We generated  $2 \times 10$  integer-valued matrices in standard form, as in (2.3), with entries chosen uniformly at random from the set  $\{0, \ldots, 7\}$ . Minor exceptions to this were the values for  $a_1$  and  $b_N$ , which were both chosen uniformly at random from the set  $\{1, \ldots, 7\}$ , and the value for  $a_N$ , which was chosen uniformly at random from the set  $\{0, \ldots, b_N - 1\}$ . Once a random weight matrix was generated, we retained it only if it satisfied:
Table 1: Final network architecture and configuration.
Hyperparameter	Value	Hyperparameter	Value
Layers	(512,768,512)	Momentum	0.99
Batch size	128	LeakyReLU slope	0.01
Initial learning rate	0.01
1. None of the columns are the zero vector.
2. The sum of the columns is not a multiple of any of them.
3. The subspaces  $S_{+}$  and  $S_{-}$  in (2.2) are both of dimension at least two.
4. The matrix is well-formed.
The first condition here was part of our definition of weight matrix; the second condition is equivalent to  $X$  being  $\mathbb{Q}$ -factorial; the third condition guarantees that  $X$  has Picard rank two; and the fourth condition was discussed above.
We used rejection sampling to ensure that the dataset contains an equal number of terminal and non-terminal examples. Before generating any weight matrix, a boolean value was set to True (terminal) or False (non-terminal). Once a random weight matrix that satisfied conditions (1)-(4) above was generated, we checked if the corresponding toric variety was terminal using the method discussed in §2. If the terminality check agreed with the chosen boolean, the weight matrix was added to our dataset; otherwise the generation step was repeated until a match was found.
As discussed, different weight matrices can give rise to the same toric variety. Up to isomorphism, however, a toric variety  $X$  is determined by the isomorphism class of its fan. We deduplicated our dataset by placing the corresponding fan  $\Sigma(X)$ , which we had already computed in order to test for terminality, in normal form [19, 29]. In practice, very few duplicates occurred.
4 Building the machine learning model
We built a neural network classifier to determine whether a  $\mathbb{Q}$ -factorial Fano variety of Picard rank two and dimension eight is terminal. The network was trained on the features given by concatenating the two rows of a weight matrix,  $[a_1, \ldots, a_{10}, b_1, \ldots, b_{10}]$ . The features were standardised by translating their mean to zero and scaling to variance one. The network, a multilayer perceptron, is a fully connected feedforward neural network with three hidden layers and leaky ReLu activation function. It was trained on the dataset described in §3 using binary cross-entropy as loss function, stochastic mini-batch gradient descent optimiser and using early-stopping, for a maximum of 150 epochs and with learning rate reduction on plateaux. We tested the model on a balanced subset of  $50\%$  of the data (5M); the remainder was used for training ( $40\%$ ; 4M balanced) and validation ( $10\%$ ; 1M).
Hyperparameter tuning was partly carried out using RayTune [31] on a small portion of the training data, via random grid search with Async Successive Halving Algorithm (ASHA) scheduler [30], for 100 experiments. Given the best configuration resulting from the random grid search, we then manually explored nearby configurations and took the best performing one. The final best network configuration is summarised in Table 1.
By trying different train-test splits, and using  $20\%$  of the training data for validation throughout, we obtained the learning curve in Figure 2a. This shows that a train-validate-test split of 4M-1M-5M produced an accurate model that did not overfit. Training this model gave the loss learning curve in Figure 2b, and a final accuracy (on the test split of size 5M) of  $95\%$ .
5 Theoretical result
The high accuracy of the model in §4 was very surprising. As explained in the introduction,  $\mathbb{Q}$ -Fano varieties are of fundamental importance in algebraic geometry. However, asking whether a Fano variety has terminal singularities is, in general, an extremely challenging geometric question. In the case of a Fano toric variety one would typically proceed by constructing the fan, and then performing
(b)
Figure 2: (a) Accuracy for different train-test splits; (b) epochs against loss for the network trained on 5M samples.
a cone-by-cone analysis of the combinatorics. This is computationally expensive and unsatisfying from a theoretical viewpoint. The success of the model suggested that a more direct characterisation is possible from the weight matrix alone. An analogous characterisation exists in the simpler case of weighted projective spaces [24], which have Picard rank one, however no such result in higher Picard rank was known prior to training this model.
Inspired by this we prove a theoretical result, Proposition 5.3, which leads to a new algorithm for checking terminality directly from the weight matrix, for  $\mathbb{Q}$ -factorial Fano toric varieties of Picard rank two. Consider a weight matrix as in (2.1) that satisfies conditions (1)-(4) from §3, and the toric variety  $X$  that it determines. As discussed in §2, and explained in detail in the Supplementary Material,  $X$  determines a convex polytope  $P$  in  $\mathbb{R}^{N-2}$ , with  $N$  vertices given by the first lattice points on the  $N$  rays of the fan. Each of the vertices of  $P$  is a lattice point (i.e., lies in  $\mathbb{Z}^{N-2} \subset \mathbb{R}^{N-2}$ ), and  $X$  has terminal singularities if and only if the only lattice points in  $P$  are the vertices  $e_1, \ldots, e_N$  and the origin.
Definition 5.1. Let  $\Delta_i$  denote the simplex in  $\mathbb{R}^{N-2}$  with vertices  $e_1, \ldots, e_i, \ldots, e_N$  where  $e_i$  is omitted. We say that  $\Delta_i$  is mostly empty if each lattice point in  $\Delta_i$  is either a vertex or the origin.
Notation 5.2. Let  $\{x\}$  denote the fractional part  $x - \lfloor x\rfloor$  of a rational number  $x$ .
Proposition 5.3. Consider a weight matrix
$$
\left[ \begin{array}{c c c} a _ {1} & \dots & a _ {N} \\ b _ {1} & \dots & b _ {N} \end{array} \right]
$$
that satisfies conditions 1-4 from §3. Let  $g_i = \gcd \{a_i, b_i\}$ , and let  $A_i, B_i$  be integers such that  $A_i a_i + B_i b_i = g_i$ . Set
$$
\alpha_ {i} ^ {j} = \frac {a _ {j} b _ {i} - b _ {j} a _ {i}}{g _ {i}}
$$
$$
\alpha_ {i} = \sum_ {j = 1} ^ {N} \alpha_ {i} ^ {j}
$$
$$
\beta_ {i} ^ {j} = - A _ {i} a _ {j} - B _ {i} b _ {j}
$$
$$
\beta_ {i} = \sum_ {j = 1} ^ {N} \beta_ {i} ^ {j}
$$
$$
f _ {i} = \frac {\alpha_ {i} g _ {i}}{\operatorname * {g c d} \{g _ {i} , \beta_ {i} \}}
$$
noting that all these quantities are integers. Then  $\Delta_{i}$  is mostly empty if and only if for all  $k\in$ $\{0,\ldots ,f_i - 1\}$  and  $l\in \{0,\dots ,g_i - 1\}$  such that
$$
\sum_ {j = 1} ^ {N} \left\{k \frac {\alpha_ {i} ^ {j}}{f _ {i}} + l \frac {\beta_ {i} ^ {j}}{g _ {i}} \right\} = 1
$$
$$
\left\{k \frac {\alpha_ {i} ^ {j}}{f _ {i}} + l \frac {\beta_ {i} ^ {j}}{g _ {i}} \right\} = \left\{\frac {\alpha_ {i} ^ {j}}{\alpha_ {i}} \right\}
$$
for all  $j$
Let  $s_{+} = \{i \mid a_{i}b - b_{i}a > 0\}$ ,  $s_{-} = \{i \mid a_{i}b - b_{i}a < 0\}$ , and let  $I$  be either  $s_{+}$  or  $s_{-}$ . Then  $\Delta_{i}$ ,  $i \in I$ , forms a triangulation of  $P$ . Thus  $X$  has terminal singularities if and only if  $\Delta_{i}$  is mostly empty for each  $i \in I$ . This leads to Algorithm 1.
Algorithm 1 Test terminality for weight matrix  $W = [[a_{1},\dots ,a_{N}],[b_{1},\dots ,b_{N}]]$ .
1: Set  $a = \sum_{i=1}^{N} a_i$ ,  $b = \sum_{i=1}^{N} b_i$ .  
2: Set  $s_+ = \{i \mid a_i b - b_i a > 0\}$  and  $s_- = \{i \mid a_i b - b_i a < 0\}$ .  
3: Set  $I$  to be the smaller of  $s_+$  and  $s_-$ .  
4: for  $i \in I$  do  
5: Test if  $\Delta_i$  is mostly empty, using Proposition 5.3.  
6: if  $\Delta_i$  is not mostly empty then  
7: return False.  
8: end if  
9: end for  
10: return True.
Comparisons Testing on 100000 randomly-chosen examples indicates that Algorithm 1 is approximately 15 times faster than the fan-based approach to checking terminality that we used when labelling our dataset (0.020s per weight matrix for Algorithm 1 versus 0.305s for the standard approach implemented in Magma). On single examples, the neural network classifier is approximately 30 times faster than Algorithm 1. The neural network also benefits greatly from batching, whereas the other two algorithms do not: for batches of size 10000, the neural network is roughly 2000 times faster than Algorithm 1.
6 The terminal toric Fano landscape