Can Transformers Do Enumerative Geometry?
Authors: Baran Hashemi
Roderic Guigo Corominas
Alessandro Giacchetto
Venue: ICLR 2025 Poster (2025)
Keywords: AI for Mathematics
Algebraic Geometry
Theorem Discovery
Transformers
Recursive functions
Interpretability Analysis and world model.
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: We introduce a Transformer-based approach to computational enumerative geometry, specifically targeting the computation of $\psi$-class intersection numbers on the moduli space of curves. Traditional methods for calculating these numbers suffer from factorial computational complexity, making them impractical to use. By reformulating the problem as a continuous optimization task, we compute intersection numbers across a wide value range from $10^{-45}$ to $10^{45}$. To capture the recursive nature inherent in these intersection numbers, we propose the Dynamic Range Activator (DRA), a new activation function that enhances the Transformer's ability to model recursive patterns and handle severe heteroscedasticity. Given precision requirements for computing the intersections, we quantify the uncertainty of the predictions using Conformal Prediction with a dynamic sliding window adaptive to the partitions of equivalent number of marked points. To the best of our knowledge, there has been no prior work on modeling recursive functions with such a high-variance and factorial growth. Beyond simply computing intersection numbers, we explore the enumerative "world-model" of Transformers. Our interpretability analysis reveals that the network is implicitly modeling the Virasoro constraints in a purely data-driven manner. Moreover, through abductive hypothesis testing, probing, and causal inference, we uncover evidence of an emergent internal representation of the the large-genus asymptotic of $\psi$-class intersection numbers. These findings suggest that the network internalizes the parameters of the asymptotic closed-form and the polynomiality phenomenon of $\psi$-class intersection numbers in a non-linear manner.

Published as a conference paper at ICLR 2025
CAN TRANSFORMERS Do ENUMERATIVE GEOMETRY?
Baran Hashemi
ORIGINS Data Science Lab Technical University Munich baran. hashemi (t um. de
Roderie G. Corominas
Department of Mathematics
Harvard University
mruiig@@mtth.harvaaee
Alessandor Giacchetto
Departement Mathematik
ETH Zürich www.ndo..o.ccchhoooooooo e www..cccetotmme...eeeee..
a clessandro.
ABSTRACT
We introduce Transformer-based approach to computational enumerative ge- ometry, specifically targeting the computation of vi-class intersection numbers the moduli space of curves. Traditional methods for calculating these num- bers suffer from factorial computational complexity, making them impractical to continuous optimization task, we compute
By reformulating the problem intersection numbers across a wide value range from 10-45 to 1045, To cap- ture the recursive nature inherent in these intersection numbers, propose the Dynamic Range Activator (DRA) a new activation function that enhances the Transformer's ability model recursive patterns and handle severe heteroscedas- ticity. Given the precision required to compute these invariants, we quantify the uncertainty the predictions using Conformal Prediction with dynamic sliding window, adaptive to partitions of equivalent numbers of marked points. To the best of our knowledge, there has been no prior work on modeling recursive fune- tions with such high-variance and factorial growth. Beyond simply computing intersection numbers, we explore the enumerative "world-model" of Transform- ers. Our interpretability analysis reveals that the network is implicitly modeling the Virasoro constraints purely data-driven manner. Moreover, through ab- ductive hypothesis testing, probing, and causal inference, an emergent internal representation of the the large-genus asymptotic of dl-class intersection numbers. These findings suggest that the network internalizes the parameters the asymptotic closed-form and the polynomiality phenomenon di-class intersection numbers in a non-linear manner.
uncover evidence of
INTRODUCTION
Enumerative geometry is branch of mathematics concerned with counting geometric objects finding invariants that satisfy certain geometric conditions (Mumford 1983 Katz 2006). A clas- sical question in enumerative geometry is: How many conics pass through five given points in the plane? A more contemporary one is: How many rational curves of degree one are there on a quintic threefold? Addressing such problems often boils down computing intersection numbers, which are fundamental in computing Weil-Petersson volumes, Hurwitz numbers, Gromov-Witte invari- ants, and quantum gravity amplitudes. An example of such objects v/-class intersection numbers. Traditional recursive methods like the KdV hierarchy (Witten 1991 Kontsevich 1992) have com-
GitHub Code: httpps://giiiii http:///w...bb.ccoooaaaaaaaa

---

Published as a conference paper at ICLR 2025
putational complexities that grow like O 2*), where g is the genus and n is the number of marked points, making calculations infeasible even for low values of g and n. To overcome this computational bottleneck, propose employing Machine Learning (ML), particularly Transform- ers (Vaswani et al. 2017), to approximate vi-class intersection numbers.
It is knoll that Transformer- baased models have difficulty with even periodic patterns, let alone recursive problems Ziyin et al. (020). Therefore, we first examine the architectural design choices and training pipeline variants necessary for learning the recursive reasoning processes involved in quantum Airy structures (Kontsevich & Soibelman 2018 Andersen et al. 2024). Hence, we in- troduce a non-linear activation function, the Dynamic Range Activator (DRA), which is particu- larly suited for learning recursive functions. Consequently, we develop DynamicFormer, modified Transformer- -eased model designed to predict si-class intersection numbers given the quantum Airy structure initial data. Additionally, we believe that predictive model without uncertainty estimation inherently unreliable. Therefore, we incorporate uncertainty quantification into our model's pre- dictions using Conformal Prediction (Shafer & Vovk 2008). Going further, we investigate whether the network is capable of abductive reasoning for know ledge discovery. Specifically, we ask whether can conduct parametric inference and extract mathematical insights from its internal understand- ing beyond the raw data provided to offer intuition and potentially generate new conjectures. We provide causal and correlational evidence that the network is actually understanding the underlying mathematics. As a result, we aim to introduce a new paradigm for using Transformers in algebraic geometry as both powerful and explainable amortization method and source of intuition for the mathematician.
While the studies discussed in Appendix B primarily focus on symbolic logic or compact numer- ical sequences, they fall short in handling the highly recursive and complex structures found in enumerative geometry. Existing models are limited to in-distribution settings and struggle with the high-variance and recursive nature of problems in computational algebraic geometry. For example, Belcak et al. (2022) discuss and benchmark various MI. models only up to unique-type functions and do not even enter the realm of recursive functions.
Our work bridges machine- collaborative guidance and predictive problems while introducing new methods. Most studies in prediction tasks remain within in-distribution and compact settings, whereas we extend our approach to out-of-distribttion recursive predictions, incorporating Confor- mal Prediction uncertainty estimation essential in mathematics, especially when a mathematician needs to assess whether an informed conjecture is correct based on available samples. In such decision-making processes, having access to the uncertainty of Al-predicted samples/values is crucial, as it provides insight into the reliability of the predictions and guides the validation of mathematical hypotheses. We also enhance the pre- dictive task by using multi-modal data, involving both discrete sets and continuous graph (sequence) representations. This approach moves beyond the focus on mere compact symbolic/numerical se- quence modalities (Meidani et al. 2024). Furthermore, in the field of machine-humma collaborative guidance, we conduct causal inference and correlational analyses understand the mathematical "world-model"" of Transformers (Li et al. 2023 Micheli et al. 2023), offering deep insights into the underlying algebraic geometry problem hand.
often overlooked aspect in the literature. Precision is
To the best of our knowledge, this is the first time Transformers and explainable MI have been applicd to enumerative algebraic geometry to extract knowledge and illuminate the model's internal understanding of deep research-level mathematical concepts. In the end, we aim to target both machine learning scientists and mathematicians, as our work is relevant to both communities.
2 QUANTUM AIRY STRUCTURES
The conceptually simplest approach to computing tl-class intersection numbers involves using Vira- soro constraints recast in the language of quantum Airy structures (Kontsevich & Soibelman [0018] Andersen et al.] 2024). The latter is an algebraic reformulation of topological recursion (Eynard
& Orantin 2007). ). quantum Airy structure on a complex vector space V. dually generated by
(xr) her. is a family of differential operators (L4)irr in the Weyl algebra over V of the form
L =h:h0; - 112 Σ (fArasa x² b +BB, 20
1 co 8.0h)) - 122 Dr.
(2.1)
2
a.bel
2

---

Published as a conference paper at ICLR 2025
and forming Lie sub-algebra of the Weyl algebra. Here 0; 0₂ and Ai,a,b Ai.b,a B C Che and D., are scalars indexed by elements in 1, and fi is formal parameter that keeps track of the genus. Given a quantum Airy structure, one can uniquely find formal function Z on V, called partition function, which is annihilated by the differential operators (Li)ser. More precisely, the following theorem holds.
Theorem 2.1. If (Li)ez is quantum Airy structure V, there exists a unique formal function Z of the form
2-2-2+t
Z( fi) exp
Σ
Egdeenndd
(2.2)
n!
$20
EI
230
with Food, Foodi d2 O, F gids gidd..ddd da symmetric in d1 di....,d
dn E I, and such that
=,Z=0 =0
viel.
(2.3)
The elements gg = Edr..., d., F Fgidi....dy 2h d are symmetric tensors of rank a over V*. The collection of coefficients (A, B, C, D) can also be regarded tensors, and will often refer to them as the quantum Airy structure initial data. For a given quantum Airy structure, the coefficients .g.......d.. called quantum Airy structure amplitudes,
(minus) the Euler characteristic 2g- 2 + n. For the lowest values of g and n. the base cases are given by Fos.j,k Ai.j. and FI, D,, where Ai,, and D, are part of the data of the quantum Airy structure. For higher values of and n satisfying 2g + n >> 0, the recursion formula, known as topological recursion, is given by
determined recursively
F gi da.....dh
Σ m= AET
Σ
Bai,dmm drn.da....am....
Σ Ca.b Fg
Σ
Foriatt F Forib.t
(2.4)
a.bEl
GL (d.....d.
In general, the computation of the amplitudes has a computational complexity O 2"), which makes the calculation of the amplitudes at high genera problematic. For nigh-dimensional vector spaces finding a closed-form solution becomes increasingly impractical, not impossible. The central example is the one associated to the té-class intersection numbers. In this case, the underlying vector space generated by vectors indexed by I = N, the non-negative integers. The differential operators (L)) are determined by the tensors
(2k: 1)!!
Ar,,A 8t.j.k
Bt, = k++ (2i + 1)!! - 1)!!
(2.5)
(2j + (2k + 1)!! (2i 1)!!
8,1
ct == d
D, 24
The resulting operators, after shifting of the indices and rescaling L, (2i-1)!!
L; form
representation of the Virasoro algebra: [L,, Ly] 1² ونززاز for all i > -1. Remarkably, the associated amplitudes coincide with ci-class intersection numbers (see Appendix C
(Td.
Tan) g.n if d1 .... d, dg. == 3g - n, otherwise.
F Fg:d
(2.6)
In this paper, we focus on the specific choice of quantum Airy structure data given by Equation (25)) thereby computing the tl-class intersection numbers. For ease of notation, will denote generic partition of dg. of length n as be denoted by (d),,., g. = (Ta, Tam)a.n
.a....,,,,.. The associated ti-class intersection number will
The goal the next sections to provide the possibility of regressing these numbers via a Transformer-base based model, given the quantum Airy structure initial data. Our model is trained

---

Published as a conference paper at ICLR 2025
known data computed by brute force algorithm up to genus 13 and is tested up to genus 17. Dur- ing experimentation, we observed that when comparing the contributions of B and C, the tensor had greater impact el-class intersection numbers. Consequently, we incorporated only this ini- tial datum (excluding C),, a decision further supported by the inherent properties of the intersection numbers (see Appendix D). This initial observation motivated the subsequent series of explainabil- ity analysis experiments and findings.
METHODS
The recursive nature of functions in enumerative geometry, combined with the sparse and high- variance target distributions of vi-class intersection numbers, introduces a high complexity in mod- eling and accurately approximating these functions. A main property of recursive maps (e-g facto- rial function), is their dramatic growth and drop. Learning this recursive behavior requires not only fitting high-frequency patterns within a bounded region but also successfully extrapolating those patterns beyond that region. In time series prediction tasks, capturing periodic even behavior is challenge. Various methods (Miller et al. 2024) have been employed to model periodic patterns effectively. However, these approaches typically deal with uni-modal data that also exhibit rela- tively low variance in both In-Distribution (ID) and Out-Of-Distribution (OOD) regions and do not generalize well recursive problems with the high-variance observed in our context. Therefore, traditional methods for modeling high-variance recursive data, including those used in time series analysis and standard Transformer architectures, are insufficient for our needs (Lakretz et al. [021] Belcak et al., 2022; Zhang et al. 2024). Thus, to capture such behavior and perform proper inference for modti-modal recursive problems, we enhance Transformers by introducing the Dynamic Range Activator (DRA), and introduce DynamicFormer, depicted in Figure [5] The DRA is designed to handle the recursive and factorial growth properties inherent in enumerative problems with mini- mal computational overhead and can be integrated into existing neural networks without requiring significant architectural changes.
Dynamic Range Activator. It has been shown (Parascandolo et al. 2017; Dauphin et al. 2017; Ziyin et al. 2020; So et al. 202T) that the choice of activation functions plays an important role in the interpolation and extrapolation properties of Transformers. To enable Transformer models to precisely capture the recursive behavior of the data, we introduce a simple activation function that we call the Dynamic Range Activator (DRA). DRA integrates both harmonic and hyperbolic components as follows,
DRA(#)= == sin" ( + ccos(bz) + d tanh(bx),
(3.1)
where a, c, d are learnable parameters. allows the function to simultaneously model periodic data (through sine and cosine) and rapid growth attenuation (through the hyperbolic tangent) response. DRA is inspired by the Snake non-linear function (Ziyin et al. 2020). However, Snake only offers sinusoidal modulation added to a linear term, which, while providing a basic non- linear transformation, lacks the additional flexibility for rapid damping or amplification effects that hyperbolic tangents can provide.
To demonstrate the advantage of DRA in capturing recursive behavior, we set up a small experiment. We generate small dataset based the recursive function r(n) + (n, AND, r(n 1)). where AND is the bitwise logical AND operator (Sloane, 2007). We train a fully connected neural network with two hidden layers consisting of 64 and 32 neurons over the interval n E [0, 120), then test the model over n. E [12,, 200). we also provide a comparison between Multi-Layer Perceptron (MLP) and the vanilla Kolmogorov-Arnole Networks (KAN) (Liu al. 2024). The results are shown in Figure [ This experiment demonstrates that ReLU, Tanh, Gated Linear Unit (GLU) (So et al., 2021), and KAN are unable to capture the recursive nature of the underlying function within finite training time. Snake appears to learn some periodicity in both the training and testing regions. In contrast, DRA shows a better ability to capture the correct fluctuations, as well as the rapid growth and drops of the underlying recursive function, in both the interpolation and extrapolation regimes. This provides an evidence that DRA has the desired flexibility for modeling recursive behavior and has the potential to effectively model such problems. We further conduct experiments, in Section 4 to demonstrate the advantages of DRA over typical non-linear activation functions in predicting gi-class intersection numbers.
4

---

Published as a conference paper at ICLR 2025
400
True rin)
400
Truestn)
200
200
100
100
50
175
100
200
400
True rin)
True sin)
Tanh
ReLU
200
200
25
50
75
100
125
150
175
200
90
75
100
125
150
175
200
400
Truerin) rin)
Swish
GW
300
200
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
((n)
PolyNorm
KAN
300
300
200
25
50
75
100
125
150
175
2000
50
75
100
125
150
175
200
Figure/ Comparison of DRA MLP with Snake, ReLU, Tanh, GLU, Swish, and PolyNorm activation functions, and KAN in training region [0, 120 and test (extrapolative) region n E [121, 200). We ensured that all models have a comparable number of parameters. The MLP with DRA non-linearity demonstrates superior performance in capturing the function's behavior.
4 RESULTS
To evaluate the performance, we consider two setups: In-Distribution (ID) and Out-Of- Distribution (OOD) settings. We use R² compare the results and assess the goodness-offfe fit in the intersection number regression task. We then compare DRA with ReLU, Snake, and GLU acti- vation functions baselines.

---

Published as a conference paper at ICLR 2025
4.1 IN-DISTRIBUTION RESULTS
In the ID setting, we examine data with the same genera as the training data, that Pin [1, 13). but with different, unseen numbers of marked points vin [35, 11]. In this setting, the R², the empirical coverage, and the conformal width (CW) (see Appendix are shown in Table Ta
(g, n))
R² 1
Coverage
CW
(1, 35)
99.8
90.35
1.03
(2, 33)
99.6
83.60
0.84
(3, 31)
99.9
74.79
0.76
(4, 29)
98.7
95.66
1.11
(g. n)
R² f
Coverage
CW
(5, 27)
99.1
92.66
1.03
(14, [1,9])
99.6
93.82
0.93
(6, 25)
99.3
91.18
0,80
(15, [1.
95.9
84.27
0,91
(7, 23)
99.1
93.05
0.68
(16, [1,5]
94.1
89.60
3.55
(8, 21)
99.8
90.88
0.76
(17. 11.3)
93.8
95.27
8.30
(9, 19)
99.9
96.71
0.91
(10,17)
99.9
90.01
1.04
(b) R and conformal uncertainty estimation results with 0.1 (90% target coverage) in the OOD set-
15)
99.8
91.97
0.87
12, 13)
99.6
89.08
1.30
ting.
13, 11)
99.9
95.90
0.97
(a) R² and conformal uncertainty estimation results with = 0.1 (90% target coverage) in the ID setting.
Table 1: R² and conformal uncertainty estimation results.
4.2 OUT-OI-DISTRIBUTTI RESULTS
Now, we study the more challenging task of OOD prediction. In the OOD setting, we examine data with a higher genera than the training data, specifically good [14, 15, 16, 17]. and a number of marked points nOOD E [1.9]. In this setting. the R², the empirical coverage, and the conformal width shown in Table 1b As evidenced by Table 2 the DRA outperforms the ReLU, GLU, and Snake activation functions in predicting the recursive intersection numbers the OOD setting.
ReLU R²
GLU
Snake
DRA R²
CW 4
R² t
CW
R² f
CW 1
CW
1
71.5
9.73
74.7
8.34
82.9
6.55
95.8
3.42
Table 2: Comparison of R² and Conformal Width between models with ReLU, GLU, Snake, and DRA their activation functions the OOD regime.
How DOES THE NETWORK Do ENUMERATIVE GEOMETRY?
Up until now, have tried to showcase that DynamicFormer is capable of predicting ti-class in- tersection numbers. However, several key questions remain. Do Transformers actually understand the underlying enumerative geomelry? Is il possible extract usefuu knowledge or hints toward possible (re Jriscoovyy of a theorem? To address these questions, we have made several interesting observations throughout our work that offer deeper insights into the network's "world model" and learning process, hinting at potential mathematical knowledge discovery.
5.1 THE DILATON EQUATION
The topological formula equation 2.4 for the specific case of vi-class intersection numbers and the choice of di takes a particularly simple form, known as the Dilaton equation,
(1, d .d..... g. (2g (d1, -.. d g.
(5.1)

---

Let (d) be the intersection numbers indexed by genus g and number of marked points Given Published as a conference paper at ICLR 2025
The equation states that the intersection number involving 71-term can be reduced to simpler intersection number with one fewer marked point.
the the set of values of the tensor B and the set of partitions d N", the neural network em- bedding P = { [Rd.. Xddg.n xdg. No } I2 maps the input data & dimensional vector space R$ denoted (g)n((c))g. g.n B, d)). This embedding describes the hidden understanding of the Transformer just before the prediction head, capturing the learned representation the intersection numbers. By equipping IR* with the standard inner product, St. R R IR, obtain the sine of the angle between the normalized embeddings of db-class intersection numbers as a measure of geometric similarity:
(ar x g.n.
S
(5.2)
| |
Observing an interesting recursive pattern in Figure 2 we hypothesize that they stem from the Vira- soro constraints between intersection numbers. To verify this, we visualized the Dilaton equations and confirmed that the model has rediscovered it as relationships between intersection numbers. We translated the Dilaton equation into Dilaton relation matrix by flagging any two intersection then observed that the intersection numbers with high cosine similarity follow at least the Dilaton equation. We anticipate that other observed patterns may also be manifestations of the remaining Virasoro constraints. This phenomenon persists even in the OOD setting. This explains how the model has succeeded in generalizing well by learning the underly- ing symmetries and relations. provides an evidence that the network has learned the Virasoro constraints in data-driven way without prior exposure these governing rules.
numbers that satisfy this equation.
0.0
088
0.0
Intersections
Intersections
Intersections
Interneciions
InlerannE SITH
Intersections
Figure 2: The Dilaton relations matrix (left) for ri-class intersection numbers at 13(top) and g = 14(bottom). Cosine similarity between embeddings of predicted intersection numbers (right) and with cut S.g 0.98 (middle).
5.1.1 LARGE GENUS ASYMPTOTIC AND ABDUCTIVE REASONING
Delecroix et al. 202T) conjectured that, in the large genus limit, vi-class intersection numbers sim- plify to specific asymptotic form.
Theorem 5.1. For o())), uniformly in di,..., . dn as g too:
I(2g - n) (1 o1)) 4 29 2tn
(d) g.m II (2d, + 1)!!
(5.3)

---

Published as a conference paper at ICLR 2025
Here [(x) is the Euler Gamma function.
This theorem, proven by Aggarwal (202T) illustrates that, as the genus grows large, the inter- section numbers grow factorially as (2g) with a specific exponential behavior modulated by the constant A = 2/3. Recently, another approach for computing the large genus asymptotics of in- tersection numbers was proposed by Eynard et al. (2023). J. The strategy of this proof based on a resurgent analysis of the n-point functions of t/-class intersection numbers, which are computed via determinantal formulae. The determinantal formula a consequence of the integrability property of the intersection numbers, namely KdV. With this approach, the authors extended Aggarwal's I's results by computing all subleading corrections:
2n [[2g 2 + n²)
2 3 on
+ +
(d) gm II
(2d, + 1)!!
1
(5.4)
4x (2)20-22
2g n
Identifying components and parameters such as A 2/3 or the sublead- ing corrections in Equation (5.4) can also be approached through abductive reasoning, framing the task as hypothesis testing. Abduction involves proposing plausible explanations for observed data, to determine which best aligns with the network's 's understanding of the data. In this context, we investigate how the network can infer the parameters like the constant A 2/3 in the asymptotic formula, treating it as an inverse problem. Thus, an important question we want to answer is: In scenarios where parameters in asymptotic formulas are unknown, can we use abduction to infer and provide evidence for their potential values? To address this, we first examine the reasoning of DynamieFormer by deciphering its understanding of the asymptotic parameters of the intersection numbers, which are not explicitly provided to the model. The parameters we aim to infer and redis- cover are the exponential growth and the first few subleading asymptotic polynomials 01, ..,.... (uoo& & Yang 2022
Abductive Reasoning:
Eynard et al. (023))
One can claim that there is an evidence that the model's 's embedding vector space (()) g.n B. d) encodes the parameter A = 2/3 and the subleading polynomials if there is correspondence be- tween the embeddings and these parameters. this correspondence is probing (Conneau et al., 2018), also known as diagnostic prediction (Hupkes) (ct al., 2020). Under this methodology, one trains a linear or non-linear model as a probe to predict any desired information from the latent representations of the network. Formally, how well the network's ' hidden representations encode the fundamental parameter A by predicting the target function I, associated with each input j. Let zij- E R be the hidden state for input j, and In be defined based on approximate conjectural estimation of Equation (.4). To map the hidden representations o'ynn to the target values Ij. we employ both linear and non-linear (MLP) probes f. The probe f is trained, using a conformal prediction procedure (Shafer & Vovk 2008) quantify the statistical uncertainty, by minimizing the squared error between the predicted and actual values of 1,. arg ming (f
prominent method that attempts to shed light on
aim to evaluate
-,)² High prediction performance is interpreted as an evidence that the information is encoded in the Transformer's 's enumerative world model. The efficacy of the encoding is gauged by the coefficient of determination (R²) of the probe.
For A 2/3, we set up the experiment with grid of alternative hypotheses. Specifically, we probe a discrete value space for the numerator of A, mum = 1,..., 10, and the denominator, denom 1, .,, 10, which covers 90 candidates (with A 1 excluded as trivial hypothesis). It is expected from the Wentzel-Kramers-Brillll quantum curve that A would be period on the associated Riemann surface. For this specific enu- merative problem, periods are in Q. We use both linear and non- non-linear probes to see if the network's 's internal representation for = 2/3 has a linear form or not. The higher and more precise perfor- mance of the non-linear probe to the linear one suggests that A could be encoded non-linearly, as shown in Figure
(WKB) method applied to the underlying Airy
We perform the same analysis for the first few terms of the subleading series ag with & 0, I, 3. We found that with 90% coverage, the Transformer's 's vector space repre- sentation can predict the coefficients with an R² = 0.98. Notably, the linear probe yields a lower value R2 = 0.63. This suggests that the network's 's internal representation of the polynomiality phenomenon (Guo & Yang, 2022; Eynard et al. 2023) does not have a simple linear form, but non-linear representation.
Causal Tracing:
Originally, causal tracing (Vig et al.
2020 was introduced to quantify information
storage and transfer within Transformer components Meng et al. 2022 Feng & Steinhardi 2024
8

---

Published as a conference paper at ICLR 2025
AIMI 11.1
10.04
WOM
+330
+033
+531
0.0
0.0
2218
1227
2014
4127
.322
10.0
1022
Mamerator
NAY
1:34
-131
PLACE
329
4037
0.2
0.2
3214
5990
5364
22.00
1/1
81.99
2101
8000
34
36.1
V.
2011
+1.4
61.30
+2.37
+0.35
+2.01
.031
10.15
2
20
1
2
10
Denominator
Denaminator
Figure 3: The network's ' internal representation's 's linear (left) and non non-linenn linear (right) predictive power of the exponential growth constant of s--class intersection numbers.
Wang et al. 2024). which is not our focus here. Instead, we use causal tracing to analyze the model's 's decision decision-making specifically identifying which input modalities influence the prediction of vi-class intersections. In other words, we aim to uncover how the network causally comprehends the underlying mathematics. We define an input modality as m|i] [ii), B d[i] for each instance There are then two steps:
1. Clean run. This run records the model's prediction on a regular input. With this run, we record auto-correlation of the predicted intersection numbers across different number of marked points.
Counter-factual intervention. We perform counterfactual interventions (Baron 2023) by modify- ing instances in one modality while keeping others unchanged to observe how these changes affect the model's predictions. This involves incorrectly assigning features to instances for example, as- sociating the partition d (29, 10, 5, 1) with n 3 instead of 6. Previous research factual associations in Transformers has explored methods such as adding noise to inputs (Meng al. 2022), which creates redundant distribution shifts (Zhang & Nanda 2024), and exchanging tokens (Feng & Steinhardi, (204)). Our approach is similar to the latter method, focusing on how changes in input tokens affect model predictions. In our case, for the modality of interest (while keeping the other modalities intact), we replace the input instances with random alternatives of the same genus. This replacement results in a miss-assignment of features the target prediction, i.e., m|i] + m[j] where * j. to obtain gg.n((d g.n (do(m(i))). This allows us to study how the model relates features to instances and to attribute cause and effect between interventions and the intersections. The difference observed between the clean and intervened runs is measured by R², the autocorrelation of the intersection numbers, and the Rymobe of the linear probe for the exponential growth constant A 2/3.
sample, which alters
No Intervention
B
d
R2 R2
0.96
-2.6
0.54
-2.2
probe
0.83
0.52
-2.77
0.43
Table 3: Causal Strength for Different Modalities
What causal reasoning is the model learning? R² of the intersection of d-cllsses heavily affected when using counter-factual instances for the number of marked points n and partitions d. This indicates a strong causal connection between the partitions and the intersection numbers. This is also evident from the autocorrelation plot Figure where the correlation between the intersection numbers is completely lost after disrupting the binding between instances and their partitions. The presence of such strong causal relationship for and d is not surprising, but their mild impact on
9

---

Published as a conference paper at ICLR 2025
3,02
2.05
6.75
6.75
0:2
0.50
0.25
6.00
0.00
0.00
-1.25
6.25
-0.35
-0.25
4.52
-8.22
-0.50
0.53
0.75
-0.75
Figure 4: Autocorrelation plot across different number of marked points n for clean (leftmost) and intervened runs (from left to right) for B, and d respectively.
Reooe is an interesting discovery. Despite a weak causal connection between B and the predicted intersection numbers, there is an evidence that the exponential growth constant, A = 2/3, is causally connected to the quantum Airy structure initial datum B. This suggests that the network's 's hidden states of the B initial datum encodes this parameter. Meanwhile, the weak causal impact of B on the intersection numbers results in only mild effect on their autocorrelation. This is expected since B contributes linearly, shown in Equation (2.4). while the dependence in n and d are factorial.
6 CONCLUSION
In this work, we tried to address fundamental question: Can Trunsformers perform and learn enumerative geomelry and topological recursion? To answer, we investigated a deep enumerative problem in algebraic geometry, specifically the computation of vi-class intersection numbers. As a result, we introduced DynamicFormer, a multi-modal Transformer-base cl-class intersection numbers. We analyzed the ability of the network to tackle such task in zero- shot setting. Capturing the recursive behavior of these invariants poses a non-trivial challenge for experiments,
model, to predict the
Transformers. Throughout functions and introduced a new one, Dynamic Range Activator (DRA), which improves prediction precision of recursive maps. One important message of this work is that merely predicting or classifying mathematical object with ML is insufficient. A prediction without proper uncer- tainty estimation unreliable, especially mathematics. We enhance reliability by incorporat- ing conformal uncertainty estimation and explainability for knowledge discovery.
identified the crucial role of non-linear activation
In the second part of the results, we asked simple question: How exactly does the network perform enumerative geometry? To answer this, we conducted series of correlational, conformal, and causal interoperability analyzes. Firstly, by examining the internal vector space of the model, we discovered that the network is learning Virasoro constraints without them being imposed on the model. This finding intriguing because suggests that cases where such relations and equations unknown but are suspected to exist, pre- trained Transformer could potentially aid in human- machine collaboration by providing hints and evidence for these relations in a purely data-driven manner. We further explored whether it is possible to perform some form of abductive reasoning and hypothesis testing to estimate the parameters of asymptotic form for intersection numbers. Often, there is an expectation for such asymptotic growths that leads to a process of conjecture building and deductive reasoning to derive the final formula. In this work, we advocate that by using the network's internal understanding of the data, one can provide informed guesses and even reject alternative hypotheses about candidate parameters. To achieve this, we conducted grid search on the exponential growth constant of the asymptotic limit for vl-class intersection numbers, based on the network's hidden representation. Using probing techniques, we showed that this information is encoded non-linearly in the model's internal representation. We also performed the same analysis on the first few terms of the subleading polynomial terms. We found that these sublcading terms also encoded in a non-linee non-linear manner in the model's hidden space.
In the end, we analyzed the internal decision-making procedure of DynamicFormer using causal inference. By applying the causal tracing method with counterfactual interventions, we aimed to shed light on how various input modalities noaali, the partitions, quantum Airy structure data, and number of marked points-aoe are causally responsible for the model's understanding vl-class intersection numbers and their large genus parameters.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported by the Excellence Cluster ORIGINS, funded by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under Germany's Excellence Strat- egy -0y-EXC-2000 200-20099-97731111 A.G. acknowledges support from an ETH Fellowship (22-2 FEL-003) and a Hermann- Weyl Instructorship from the Forschungsinstitut für Mathematik at ETH Zürich. B.H. expresses his gratitude to Gaëtan Borot, Elba Garcia-Failde, Lukas Heinrich, and François Charton for their invaluable discussions throughout this work. We extend our thanks to the orga- nizers of the TR Salento 2021 School and Workshop on Topological Recursion, where this project and the present collaboration were initiated. Finally, B.H wishes to thank Setarch Fadavi for her unconditional support.
REFERENCES
Hervé Abdi, Vincent Guillemot, Aida Eslami, and Derek Beaton. Canonical Correlation Analysis. Springer New York, 2018, doi: 10. 10.10007978-114999--1111-22
Amol Aggarwal. Large genus asymptotics for intersection numbers and principal strata vol- umes of quadratic differentials. Invent. Math., 226(3):897-1010, 2021. doi: 10.1007/
-00222-021-01059...
Alexander Alexandrov. Cut-and-join operator representation for Kontsevich- tau-function. Mod. Phys. Lett. A, 26(29):1933-2199 2011. doi: 10. 10.1144//0217733311153
Abdulhakim Alnugaydan, Sergei Gleyzer, and Harrison Prosper. SYMBA: symbolic computation of squared amplitudes in high energy physics with machine learning. Mach. Learn: Sci. Technol., (1):015007, 2023.
Chenyang An, Zhibo Chen, Qihao Ye, Emily First, Letian Peng, Jiayun Zhang, Zihan Wang, Sorin Lerner, and Jingbo Shang. Learn from failure: Fine-tuning LLMs with trial-and-crror data for intuitionistic propositional logic proving. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 776-790. Association for Computational Linguisties, 2024.
Jørgen E. Andersen, Gaëtan Borot, Séverin Charbonnier, Vincent Delecroix, Alessandro Giacchetto, Danilo Lewański, and Campbell Wheeler. Topological recursion for Masur-Veech volumes. J. Lond. Math. Soc., 107(1):254-332, 2023. doi: 10.11 12666
Jørgen E. Andersen, Gaëtan Borot, Leonid O. Chekhov, and Nicolas Orantin. The ABCD of topolog- ical recursion. Ady. Math., 139:109473, 2024. ISSN 0001-8708. doi: 10.1016/j.aim. 2...:1/6..ii.222...1... 1023.10947..
Lara B. Anderson, Mathis Gerdes, James Gray, Sven Krippendorf, Nikhil Raghuram, and Fabian Ruchle. Moduli-dependent Calabi-Yau and SU(3) 3U33)-strctture metrics from Machine Learning. J.
High Energy Phys., 2021(5): 2021. doi: 10. 001107//HEP05((022000.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Flo- rian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gor- don Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A cookbook of self-supervised learning, 2023.
Jiakang Bao, Sebastián Franco, Yang-Hui He, Edward Hirst, Gregg Musiker, and Yan Xiao. Quiver mutations, Seiberg duality, and machine learning. Phys. Rev. D, 102(8):086013, 2020. doi: 10. 10.100//hyyRe==.1122..852
Sam Baron. Explainable AI and causal understanding: Counterfactual approaches considered. Minds Mach,, 33(2):3477777 2023. doi: 10000///11122-22--9969999
Kai Behrend. Gromov-Witten invariants in algebraic geometry. Invent. Math., 127(3):601-617, 1997. doi: 10.10077/4002220011332
Peter Belcak, Ard Kastrati, Flavio Schenker, and Roger Wattenhofer. Fact: Learning governing abstractions behind integer sequences. Ady. Neural Inf. Process. Syst., 35:17968-11799 17980, 2022.
11

---

Published as a conference paper at ICLR 2025
Michel Bergère and Bertrand Eynard. Determinantal formulae and loop equations, 2009.
Per Berglund, Giorgi Butbaia, Yang-Hui He, Elli Heyes, Edward Hirst, and Vishnu Jejjala. Gener- ating triangulations and fibrations with reinforcement learning, 2024.
David S. Berman, Yang-Hui He, and Edward Hirst. Machine learning Calabi Yau hypersurfaces. Phys. Rev. D. 105(6):06600,, 2022. doi: 10. 1033/PyyseeD...... 1w..DD.....00000 066002.
Tianji Cai, Garrett W. Merz, François Charton, Niklas Nolte, Matthias Wilhelm, Kyle Cranmer, and Lance J. Dixon. Transforming the bootstrap: Using transformers compute scattering amplitudes in planar N 4 super Yang-Mills theory, 2024.
Tom Coates, Alexander M. Kasprzyk, and Sara Veneziale. Machine learning the dimension of a Fano variety. Nat. Commun., 14(1):5526, 2023. doi: 10.1038/441467-033 -211-------------1
Tom Coates, Alexander M. Kasprzyk, and Sara Veneziale. Machine learning detects terminal singu- larities, Adv. Neural Inf. Process. Syst., 36, 2024.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single S&!# vector: Probing sentence embeddings for linguistic properties. In Iryna Gurevych and Yusuke Miyao (eds.), 56th Anmual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 2126-12662 2136. Association for Computational Linguistics, 2018. doi: 1..18653/11/P18-11999
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets, 2020.
Jessica Craven, Mark Hughes, Vishnu Jejjala, and Arjun Kar. Learning knot invariants across di- mensions. SciPost Phys., 14(2):021, 2023. doi: 10.21 (0.2146//SiPPsstPyyy. 14..0...... 14.2.021.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In 12th International Conference on Learning Representations, 2024.
Stéphane d' Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample, and François Charton. Deep symbolic regression for recurrent sequences, 2022.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In 34th International Conference Machine Learning, pp. 933 933-941, 2017.
Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al. Advancing mathe- matics by guiding human intuition with AI. Nature, 600(7887):70-74, 2021. doi: 10.1038/
Vincent Delecroix, Élise Goujard, Peter Zograf, and Anton Zorich. Masur-Veede volumes, frequen- cies of simple closed geodesics, and intersection numbers of moduli spaces of curves. Duke Math. doi: 10.1215500127994-222--
J. (000112):663322111 2021.
Aurélien Dersy, Matthew D. Schwartz, and Xiaoyuan Zhang. Simplifying polylogarithms with machine learning. Int. J. Data Sci. Math. Sci., 0109)))135-17,, 2023. doi: 10.1142/
$2810999223500028..
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 418614416,110 2019.
Robbert Dijkgraaf, Herman Verlinde, and Erik Verlinde. Topological strings in d Nucl. Phys. B, 352:59-86, 86, 1991. doi:
Bin Dong, Xuhua He, Pengfei Jin, Felix Schremmer, and Qingchao Yu. Machine learning assisted exploration for affine Deligne-Lusztig varieties. Peking Math. J., pp. 1-50, 2024. doi: 10.1007/
12

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, 2021.
Bertrand Eynard and Dimitrios Mitsios. A new formula for intersection numbers, 2023.
Bertrand Eynard and Nicolas Orantin. Invariants of algebraic curves and topological expansion. Commun. Number Theory Phys., 1(2):347-452, 2007. doi: 10.4310/CNTP.000. .n.2..4.
Bertrand Eynard, Elba Garcia-Failde, Alessandro Giacchetto, Paolo Gregori, and Danilo Lewański. Resurgent large genus asymptotics of intersection numbers, 2023.
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? In 12th International Conference on Learning Representations, 2024.
Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends, 2024.
Sergei Gukov, James Halverson, Fabian Ruehle, and Piotr Sulkowski. Learning to unknot. Mach. Learn.: Sci. Technol., 2(2):025035, 2021. doi: 10. 10.1088/2622221 2-15322222233/111.11
Sergei Gukov, James Halverson, Ciprian Manolescu, and Fabian Ruehle. Searching for ribbons with machine learning, 2023.
Jindong Guo and Di Yang. On the large genus asymptotics of psi-class intersection numbers. Math. Ann., pp. 1-37, 2022. doi: 10. 10.1000700000--02-0555555
James Halverson, Brent Nelson, and Fabian Ruchle. Branes with brains: exploring string vacua with deep reinforcement learning. J. High Energy Phys., 2019(6):1660, 2019. doi: 10.1007/ JHEP06(20190003.
Yang-Hui Hc. Elli Heyes, and Edward Hirst. Machine learning in physics and geometry. Artif. Intell., 49:47, 2023. doi: htt1//bshost.202...o
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? J. Artif. Intell., 67:757-795, 2020. doi: 10.iar3/jair... 1.1674.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In 32nd International Conference Machine Learning, pp. 448-456, 2015.
Albert Q Jiang, Wenda Li, and Mateja Jamnik. Multilingual mathematical autoformalization, 2023.
Sheldon Katz. Enumerative Geometry and String Theory, volume 32. American Mathematical Soc., 2006.
Maxim Kontsevich. Intersection theory on the moduli space of curves and the matrix Airy function. Commun. Math. Phys., 44::1-23, 1992. doi: 10. 10.1007/BF02099522.
Maxim Kontsevich and Yan Soibelman. Airy structures and symplectic geometry of topological recursion. In 2016 AMS von Neumann Symposium, Topological Recursion and Influence in Analysis, Geometry and Topology, volume 100, pp. 433 489. Amer. Math. Soc., Providence, RI, 2018. doi: 10. 00.1090/pspum/100.
Yair Lakretz, Théo Desbordes, Dieuwke Hupkes, and Stanislas Dehaene. Causal transformers per- form below chance recursive nested constructions, unlike humans, 2021.
Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. 11th International Conference on Learning Representation, 2023.
Xiaohan Lin, Qingxing Cao, Yinya Huang, Zhicheng Yang. Zhengying Liu, Zhenguo Li. and Xiao- dan Liang. ATG: Benchmarking automated theorem generation for generative language models, 2024.
13

---

Published as a conference paper at ICLR 2025
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov-Ama Ammold networks, 2024.
Sarah Loos, Geoffrey Irving. Christian Szegedy, and Cezary Kaliszyk. Deep network guided proof search. EPiC Ser. Comput., 46:85-105, 2017. doi: 10.29007/8mwc.
Kazem Meidani, Parshin Shojace, Chandan K. Reddy, and Amir Barati Farimani. SNIP: Bridg- ing mathematical symbolic and numeric realms with unified pre-training. In 12th International Conference on Learning Representations, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Adv. Neural Inf. Process. Syst., 15:17359-17372 2022.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficie efficietf world mod- els. In / Ith International Conference on Learning Representations, 2023.
Maciej Mikula, Szymon Tworkowski, Szymon Antoniak, Bartosz Piotrowski, Albert Q. Jiang, Jin Peng Zhou, Christian Szegedy, Łukasz Kuciński, Piotr Mitoš, and Yuhuai Wu. Magnusham- mer: A measformer-bbsee approach to premise selection. In 12th International Conference on Learning Representations, 2024.
John A. Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, Budak Arpinar, and Ninghao Liu. A survey of deep learning and foundation models for time series forecasting, 2024.
Maryam Mirzakhani.
Weil-Petersson volumes and intersection theory on the moduli space of curves. J. Am. Math. Soc., 20(1):1-23, 2007. doi: 10. 0000500894-037-00-0000000
Maryam Mirzakhani. Growth of the number of simple closed geodesies hyperbolic surfaces. Ann. Math., pp. 97-125, 2008, doi: 10.4007/amnals.2088 168.97.
David Munford. Towards an enumerative geometry of the moduli space of curves. Arithmetic and Geometry: Papers Dedicated to I. R. Shafarevich on the Occasion of His Sixtitth Birthday. Volume II: Geometry, pp. 271-328, 1983.
Andrei Okounkov. Generating functions for intersection numbers on moduli spaces of curves, Inf. Math. Res. Not., 2002(18):933 957, 2002. doi: 10.1155/S107379280211 10099.
Aditya Paliwal, Sarah Loos, Markus Rabe, Kshitij Bansal, and Christian Szegedy. Graph represen- tations for higher-order logic and theorem proving. In AAAI Conference on Artificial Intelligence, volume 34, pp. 2967-2974, 2020.
Harris Papadopoulos, Volodya Vovk, and Alex Gammerman. Conformal prediction with neural networks. In 19th IEEE International Conference on Tools with Artificial Intelligence, volume 2, pp. 318-388,.... 2007. doi: 10.1109//CTT 10.1109/CTA122007.47
Giambattisa Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: sine as activation function in deep neural networks, 2017.
Marco Ragni and Andreas Klein. Predicting numbers: An ai approach to solving number series. In Joscha Bach and Stefan Edelkamp (eds.), KI 20/1: Advances in Artificial Intelligence, pp- 255-259. Springer Berlin Heidelberg, 2011.
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. Ruiz, Jordan Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from pro- gram search with large language models. Nature, 625(7995):468 475, 2024. doi: 10.1038/
9411886-023-06224-6
Phil Saad, Stephen H. Shenker, and Douglas Stanford. JT gravity matrix integral, 2019.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilities neural models. 7th International Conference on Learning Representations, 2019.
14

---

Published as a conference paper at ICLR 2025
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. J. Mach. Learn. Res., 9(3), 2008. doi: 10.5555/1390681. 1339069..
Neil J. A. Sloane. The on-line encyclopedia of integer sequences. In Manuel Kauers, Manfred Kerber, Robert Miner, and Wolfgang Windsteiger (eds.), Towards Mechanized Mathematical As- sistants, pp. 130-130. Springer Berlin Heidelberg, 2007.
David So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Searching for efficient transformers for language modeling. Adv. Neural Inf. Process. Syst., 04-6010-6022, 2021.
Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. Solving olympiad ge- ometry without human demonstrations. Nature, 625(7995):476-482, 2024. doi: 10.1038 11188-023--6747--
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 2017.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural NLP: the case of gender bias, 2020.
Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of al- gorithmie randomness. In 16th International Conference on Machine Learning, pp. 444-453,
1999.
Adam Zsolt Wagner. Constructions in combinatories via neural networks, 2021.
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: a mechanistic journey the edge of generalization. In ICML 2024 Workshop on Mechanistic Interpretability, 2024.
Mingzhe Wang and Jia Deng. Learning prove theorems by learning generate theorems. Adv.
Neural Inf. Process. Syst., 33: 13:1814665 13:11876- 2020.
Qingxiang Wang, Chad Brown, Cezary Kaliszyk, and Josef Urban. Exploration of neural machine translation in autoformalization of mathematies in mizar. In 9th ACM SIGPLAN International Conference on Certified Programs and Proofs, pp. 85-98, 2020.
Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. CrossFormer: versatile vision transformer hinging on cross-scale attention. In 10th International Conference on Learning Representations, 2022.
Edward Witten. Two-dimensional gravity and intersection theory on moduli space. In Surveys in Differential Geometry, volume 1, pp. 243-310, 1991. doi: 10.4310/SDG.. 19...................
Yuhuai Wu, Albert Q. Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. Autoformalization with large language models. Adv. Neurul Inf. Prucess. Syst., 35: 2022.
Chen Xu and Yao Xie. Conformal prediction for time series, 2023.
Huaiyuan Ying, Zijian Wu, Yihan Geng, Jiayu Wang, Dahua Lin, and Kai Chen. Lean workbook: A large-scale Lean problem set formalized from natural language math problems, 2024.
Jure Zbontar, Li Jing. Ishan Misra. Yann LeCun, and Stéphane Deny. Barlow Twins: self-supervised learning via redundancy reduction. In 38th International Conference on Machine Learning, 2021.
Dylan Zhang, Curt Tigges, Zory Zhang, Stella Biderman, Maxim Raginsky, and Talia Ringer. Transformer-based models are not yet perfect at learning to emulate structural recursion, 2024.
Fred Zhang and Neel Nanda. Towards best practices activation patching in language models: metrics and methods. In 12th International Conference on Learning Representations, 2024.
Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how fix it. Adv. Neural Inf. Process. Syst., 33:1583-1594, 2020.
15

---

Published as a conference paper at ICLR 2025
A FUTURE DIRECTIONS
An interesting open question to explore with interpretability methods developed here, is the regime the same time (so that g/n n is kept bounded above and below). At the mo- ment, there is not even a conjecture as to what the asymptotic should behave like, and how it should depend on the partitions. Another fascinating avenue for future research is to explore recent devel- opments in intersection numbers (Eynard & Mitsios 2023). This new approach proposes formula that involves sums over partitions combinatorial factors, revealing how certain partitions with specific conditions have vanishing coefficients in their decomposition into elementary symmetric polynomials. Remarkably, empirical observations suggest that even more partitions have vanish- ing coefficients than the trivial ones. This unexpected result points to deeper, hidden structure in intersection numbers, where the internal representation of the network could be very beneficial in examining these new results and uncovering the underlying patterns.
when (g. n) +
B RELATED WORKS
In recent years, new frontier of ML applications, known Artificial Intelligence for Mathemat-
ies (He et al. (202)), has emerged. This branch includes Automated Theorem Proving (Loos et al.
2017 Pa Paliwal al., 2020 An al.] 2002)) Generation (Trinh al. 2024 Wang & Deng 2020
Lin et al. 2024), Autoformalization (Wang et al.) 2020: Wu et al. 2022 Jiang et al. [023]] Mikula
al.
2024
Ying et al. 2024). machine-human collaborative guidance (Bao et al.) |020; Anderson
et al.
2021 Davies al., 2021: Berman al.) 2022] Craven et al. [023] Coates et al. 2023 Dong
ctt al.
2024),
examples/ccuntteexxmpple search problems (Halverson et al. 2011 Gukov et al.
2021
Wagner [202]] Gukov et al. 2023; Berglund et al. 22024] Romera-Paredee et al. 2024), and predic-
tive problems (such as amplitude and mathematical expression prediction) RRggi & Klein
2011
Saxton al.] 20199] Dersy et al.] 2023 Alnuqaydan et al. [023 Belcak al. 2022 d d''Acooi al.
2022 Cai et al. 2024: Coates et al. (024). These areas extensively use Transformers, particularly
problems arising from scattering processes.
C MATHEMATICAL BACKGROUND
In this section, we review some background on the original mathematical problem. Let Mg.n n. be the moduli space of genus 2 0 stable algebraic curves with n > 0 distinct marked points. This is complex, compact, smooth orbifold of dimension 3g is natural cohomology class vb; ei (C:), defined as the first Chern class of the cotangent line bundle L, at the i-th marked point.
n. Associated with each marked point
In the early 90s, Witten (Witten 1991) analyzed a theory of two-dimensional topological quantum gravity, where v)-classes play the role of observables, and the intersection numbers (sometimes called amplitudes)
(Td, Ta.))
do
di + ........ dn dg. = 3g 3+ n
(C.1)
Mg.
represent correlators of the theory. Based on low genus profile and considerations from physics, Witten conjectured that the exponential generating function
22g-24h 2
Z(x; h) = exp
Σ
(Td, ... Tdnn) -TTT) g.
Ih dd.
(C.2)
n!
920 n50
di 20
dit
satisfics an infinite tower of quadratic partial differential equations with respect to the formal vari- ables (x');> known the Korteweg de Vries (KdV) hierarchy. This was later proven by Kontse- vich (Kontsevich) 1992).
In recent years, it has been discovered that many fundamental invariants in physics and geometry can be described via intersection numbers. In physics, interest vi-class intersection numbers has been revitalized due to their connection with Jackiy-Teitellan gravity (Saad al., 2019). From
16

---

Published as a conference paper at ICLR 2025
an algebraic geometry perspective, e/-class intersection numbers represent the most basic form of Gromov-Wint Witten invariants (Behrend 1997). These intersection numbers are central to enumera- tive problems addressed by (Kontsevich 1992] and (Mirzakhani, (007) regarding the symplectic volumes of the moduli space of metric ribbon graphs and hyperbolic Riemann surfaces. Comput- ing intersection numbers is pivotal in the asymptotic counting of geodesic curves in both flat and hyperbolic random geometries (Mirzakhani 2008 Delecroix et al. 2021; Andersen et al. 2023).
Given the wide and fascinating applications of intersection numbers, computing them for arbitrarily large genera remains an open problem. Their association with the KdV hierarchy has been illu- minating for calculating these invariants (Witten 1991] Kontsevich, 1992). Another approach to characterizing these numbers involves Virasoro constraints (Dijkgraaf et al. 1991), where the parti- tion function is annihilated by a collection of differential operators that form representation of the Virasoro algebra. Alternative recursive approaches include topological recursion (Eynard & Orantin, 2007]. the cut-and-join equation (Alexandrov) (01T), and exact formulae for the generating series, such stror-funcioon-type integrals (Okounkov 2002) and the determinantal formula (Bergère & Eynard 2009). However, the general computation of intersection numbers remains an open prob- lem, as these methods can only be used to compute t--class intersection numbers for cases with small g or n. Meanwhile, universality phenomena in flat and hyperbolic geometry and in the dynamics of surfaces manifest themselves in large genera. Only recently, Aggarwal (Aggarwal) [021] has found a closed-form formula for the large genus asymptotic of si-class intersection numbers.
D DATA REPRESENTATION AND SETUP
The input data during training consisted of the sparse tensors B and C' from Equation (2.5 the genus g, the number of marked points n, and the partitions d = (d1,-.-- dz) of dy.n. We did not include A and D as input, as they are fixed initial conditions for all ai-class intersection numbers.
As we will consider values of g and n such that the dimension dg,n will never exceed dmax 100, the initial data will consist of only finitely many data. More precisely, the initial data during training were represented follows.
B Initial Data: A rank 3 tensor B E Rd Such tensor links intersection numbers of the same genus (cf. Equation (2.4)). Since sparse tensor with dy,n S dmax = 100, we chose to represent its non-zero components in the COO (Coordinate List) format. In the COO format, B is represented as a set of 4-tuples, each containing the indices and the corresponding non-zero value:
B (i. B #0}
(D.1)
where i,j, k E [0, dy.n), and B are the non-zero components of the tensor. With this choice of dmax. the maximum length of B is approximately 1500.
Initial Data: A rank tensor C. E Xdg.n Such a tensor links intersection
numbers of different genera (cf. Equation (2.4). Again, since it is sparse tensor with dg.n I dmax = 100, we chose to represent its non-zero components in the COO format. In the COO format, C' is represented as set of 4-tuples, each containing the indices and the corresponding non-zero value:
C - c Ciitt ca + #0}
(1.5,k,
(D.2)
where i,j.k k [0, d0, and C.. are the non-zero components of the tensor. With this choice of dmux. the maximum length of C is approximately 1500. In this setup, the C initial datum are excluded. This is because the C'terms contributes quadratically, while the B-term contributes linearly. Hence, B has a stronger effect on computing th-class intersections confirmed in (Aggarwal) 2021).
Partitions: The intersection numbers (d) are labeled by partitions d = (di...., dn) No of dy.n of length n. An important feature of intersection numbers (and more generally, amplitudes computed from quantum Airy structures) is that they are invariant under per- mutation of the indices, that is, elements of the partition d. For example given a partition d (3, 0),, we have (3, 0, 0),,3 30,3, 0)1,3 3 (0, 0,
17

---

Published as a conference paper at ICLR 2025
We trained the model using data up to genus g 13 and then tasked it with predicting the intersec- tion numbers for genera g = 14, 15, 16 and 17.
E
MODEL DETAILS
DynamicFormer
B
[YN]
DRA MLP
DRA MHA
Talking Modalities
Self- sulf-supervieed
d
[DYN]
DRA MLP
DRA MHA
Intersection numbers
prediction head
DRA
(g)
PNA Pooling
Figure 5: Illustration DynamicFormer. It processes two main input modalities: the quantum Airy structure datum B as ordered sequence and partitions d as and number of marked points n are incorporated as input properties, modulated with the main modalities at various stages. All layers, including the Multi-Head Attention (MHA) blocks, the Dynamic Range Activator (DRA) non-linear activation function. The MHA block for B-modality integrates Dynamic Positional bias with linear positioning as well. The Modalltiss block consists of Batch Normalization (Toffe & Szegedy| 2013) and the self-supervised loss. The DRA prediction head is 2-layer MLP that predicts u--llass intersection numbers logarithmic scale.
innittatioonnvviiannt set. The genus
To handle the different input modalities, i.c. the continuous tensor in COO sequence format and the discrete permutation invariant set d No, want develop multi-modal Transformer- based model. Transformers can effectively handle such structures due to their masked attention mechanisms and relative positional embeddings, which are adept at capturing the sparse structure of B. Moreover, the input d is permutation invariant, and Transformers naturally accommodate this property through their self-attention mechanism that treats input elements symmetrically without im- posing any ordering biases. Furthermore, Transformers are state-of the-artt art models known for their flexibility and effectiveness in handling multi-modal data, which is crucial for our problem involving complex mathematical structures. In contrast, while Multi-Layer Perceptrons (MLPs) are universal approximators, they lack the inductive biases necessary to effectively handle the structured data and invariances inherent in multi-modal input. They treat all input dimensions independently, making it challenging to leverage the sparse graph structure of or the permutation symmetry of d. This results in comparatively weaker scalability, especially when faced with the limited, com- plex, high-variance target distributions present in the vi-class intersection numbers. Table shows a comparison between MLPs and DynamicFormer.
oftte-of-the-sss
The trunk of DynamicFormer consists of two modified Transformers: one discrete, acting on the embedding of the partitions d and maintaining permutation equivariance, and one continuous, act-
18

---

Published as a conference paper at ICLR 2025
Snake MLP
DRA
MLP
DynamicFormer
R² CW 4
R² f
CW
R t
CW 4
69.3
8.44
76.5
7.73
95.8
3.42
Table 4: Comparison of R2 and Conformal Width between models DynamicFormer and MLP with DRA and Snake non-linearity in the OOD regime.
ing on the B tensor. The positions of elements in B, represented in COO format, correspond to particular combinations of indices that compute the intersection numbers. To enhance generaliza- tion for longer imput sequences,
integrated modified version of Dynamic Positional Bias (Wang ict al. 2022 Zhang et al. 2024) with linear positioning. Dynamic Positional Bias computes a relative positional bias map for each attention head, which is learnable and adapts based sequence length. Since does not inherently account for the distance from the start or end of a COO sequence, we apply masking based on input sequence length.
After the Transformer trunk of the network, all the embedded information is aggregated by the Principal Neighbourhood Aggregation (PNA) layer (Corso et al., 2020), pooling the information from the modalities (B, d). Once the information from these two branches, for each sample, is modulated with their corresponding genus and number of marked points positional attributes, (g. n). it is mapped to the outputs, i.c., el-class intersection numbers, via an MLP head. DRA is the non- linear activation function used throughout these blocks. As the loss function, we use the Mean Absolute Error (MAE) loss. The total loss is, Lioua [MAE + LIM, where LIm introduces Supervieee Learning objective between the [DYN] registry tokens (Darcet et al. 2024) each modality, inspired by Barlow Twins (Zbontar et al., 202
[DYN] registry tokens and Talking Modalities. In multi-modal learning, effectively sharing and integrating information across different modalities is crucial for building robust models. Inspired by the success of Self-Supervised Learning (SSL) (Gui et al.) 2024), we introduce simple modi- fication to enhance correspondence between modalities. To achieve this, we concatenate dynamic register tokens (Darcet et al. 2024), [DYN], to each modality. These tokens are similar to [CLS] tokens in BERT (Devlin et al.
2019 and Vision Transformers (Dosovitskiy et al. (021), in that they are passed through the attention layers alongside the input data. However, unlike [CLS] tokens, they not directly used for the downstream supervised task of amplitude prediction. Instead, they used indirectly, as register tokens (Darcet et al. 2024). which store and process global information, introduced for Vision Transformers. By attending to all tokens in the input, [DYN] tokens, similar to register tokens, do soft contrastive learning update, enabling them to learn global context for each sample. Unlike register tokens, [DYN] tokens are also optimized to ensure that the outputs from different modalities of data data- sprcifficlll from the B tensor, which provides global informa- tion per sample, and the partitions d, which serve as local information carriers-are -re as similar as possible while simultaneously reducing the redundancy of the information they carry. Their opti- mization is guided by the Canonical Correlation Analysis (Abdi et al. 2018) family of SSL, where the aim is to infer the relationship between two modalities by analyzing their covariace ma- trices (Balestriero et af. 2023). Inspired by Barlow Twins (Zbontar et al. (021), the two modalities start to have "conversation" promoting interaction in which the two modalities collaboratively refine their representations. The idea behind this is that such a consensus mechanism not only en- hances the model's ability to generalize better to Out-Or-Diste --iitrrbiiitiitiinnn (OOD) data but also improves its robustness. The Talking Modalities loss is then defined as follows:
LIM
Σ E=1
(B.d) cl
+1
(cIB.d)) Σ (+)
(E.1)
where C is the cross-correlation matrix of the representations from the two modalities, (B, k is the dimensionality of the model's embeddings (vector representations), and À is the weighting factor that balances the contributions of talking modalities. The first term penalizes the diagonal elements of the cross cross-correlation matrix C for deviating from the identity, thus encouraging invariance. The second term penalizes the off off-diagonal elements, encouraging the components of the representation
19

---

Published as a conference paper at ICLR 2025
vectors to be decorrelated. Table shows that this modification slightly enhances the performance of the model in the OOD prediction regime.
None
[DYN]
[DYN+TM
R² 1 CW 4
R² f
CW 4
R²
CW +
91.1
4.91
94.6
4.05
95.8
3.42
Table 5: Comparison of R and CW between models with and without [DYN] register token and Talking Modalities. A small performance improvement can be observed as result of using register tokens Talking Modalities in the OOD regime.
F COMPARISON BETWEEN TRUE AND PREDICTED INTERSECTION NUMBERS
Figure shows the numerical comparison between the true sl-class intersection numbers and Dy- namicFormer's predictions, illustrating its performance in capturing the recursive behavior.
1el0
Trush
Fred
Pred cied 1..1
Predicted Intersections
Parfect Rt
Intersections
162
30-1
30-1
20-19
10-1
10-11
10-15
True Intersections
(a) 13
Predicted Intersections
Intersections
30-1
10-3
20-4
2.0
1.5
10mu
10-41
10-11
10-29
True Intersections
(b) g 14
20*
13-1
12-4
20.
Intersections
Predinted
16.9
10-1
10-
30
70
Thue Intersections
(c) g = 15
Figure 6: Comparison of true, predicted, and R values for = 13, 14, 15.
20

---

Published as a conference paper at ICLR 2025
G UNCERTAINTY QUANTIFICATION: CONFORMAL PREDICTION
To estimate the uncertainty of the prediction, we incorporate Conformal Prediction (CP) (Vovk et al. 1999). CP a probabilistic uncertainty quantification technique that provides prediction intervals in finite samples without making any distributional assumptions. We integrate the Inductive Conformal Prediction (Papadopoulos al. 2007) with dynamic sliding window (Xu Xie 2023) that, given the high heteroscedasticity in the data, computes rolling residuals for predicted intersections for each partition of equivalent marked points n. In our modification, we fit quantile regression model to the residuals within rolling window of equivalent marked points. This ensures that the residuals are more scale-dependent and appropriately adjusted for the inherent variability in the data, potentially capturing trends and heteroscedasticity better. As a result, we also report the empirical coverage and Conformal Width (CW) in the log scale.
H PRINCIPAL COMPONENT ANALYSIS OF MODEL'S INTERNAL REPRESENTATION
If perform Principal Component Analysis (PCA) the hidden embedding of each sample, g. we observe in Figure 7hat DynamicFormer shows hierarchical representations of intersection numbers across both the genus g and the number of marked points It seems that the model has indeed learned a recursive structure within the data.
27
Inteppal Campaves
23
kiedpal Companeet
aindaal Campanent
Figure 7: The first two principle components of model's hidden states across different genera (left) and number of marked points (right).
21


Reviews:
[{"review_id": "exo9C2nJkK", "reviewer": "Reviewer_ncfx", "summary": "The paper proposes and tests the usage of transformers in the field of enumerative geometry, specifically regarding topological recursions and $\\psi$-class intersection numbers. To accomplish this, the paper proposes a new class of activation functions called Dynamic Range Activators (DRAs), and presents evidence of their performance in predicting a simple recursive function as part of a fully connected neural network, and then their ability to predict $\\psi$-class intersection numbers as part of their DynamicFormer architecture. The paper then attempts to investigate the trained DynamicFormer to see if it can predict other concepts in enumerative geometry, including the Dilation equation that stems from Virasoro constraints, as well as the asymptotic behavior of $\\psi$-class intersection numbers using abductive reasoning, verified using counter-factual intervention.", "questions": "**Notice: These questions have been answered during the discussion phase, and remain unedited for posterity.**\n\n* What is the significance of $\\hbar$ in the quantum Airy structure? How is it relevant specifically to training the DynamicFormer?\n* Figure 3 may be a discrete sampling of an underlying (continuous?) map that gives an $R^2$ for each $A$, with a maximum at $A=2/3$. Can the authors characterize this map?\n* Figure 4 shows a significantly weaker causal impact of $B$ on the number of intersection points, compared to $n$ and $d$. Though the authors call this unexpected in section 5's last paragraph, is there any explanation regarding the weak causal impact of $B$?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The new DRA functions, motivated by the evidence presented in the paper, are a significant contribution that may interest machine learning scientists.\n* Training a DynamicFormer to predict $\\psi$-class intersection numbers, which then allows one to investigate a system's deeper geometry, is a significant, novel contribution that will interest mathematicians investigating enumerative geometry.\n* The use of Conformal Prediction to estimate uncertainty provides a concrete measure of confidence in the experimental results, contributing to the paper's soundness.\n* The figures are clear and high-quality, with informative captions.\n* The writing is clear and mostly organized, including the mathematical background, methodology, and results.", "weaknesses": "**Notice: These weaknesses have been adressed during the discussion phase, and apply only to the initial version of the manuscript. However, these will remain unedited for posterity.**\n\n### Section 2\n* The equations in this section use $\\hbar$ without defining it in the text. It may be worth explicitly calling it the reduced Planck constant in the text.\n* The last paragraph mentions excluding the tensor $C$ due to a decreased impact on the computed $\\psi$-class intersection numbers, observed during experimentation. Appendix C justifies this exclusion, yet is not referenced in the text, making for seemingly unsound reasoning for excluding $C$. The authors should consider referencing appendix C here to further justify the exclusion of $C$, and expanding on this point within appendix C proper.\n\n### Section 3\n* In the first two paragraphs, the paper presents the DynamicFormer for the first time and references a figure placed within an unrelated appendix, resulting in a disjointed reading experience. The authors may consider moving some parts of section 3 (such as its first two paragraphs) into a new appendix showcasing the DynamicFormer in detail and including the figure close by.\n* In the same paragraphs, the authors use the initials COO without previously defining them. These initials seem to appear nowhere else in the main text, and only in appendix B are they defined as Coordinate List. Besides hurting the paper's readability, this seems to be an implementation detail that does not need to appear in the main text.\n* The last paragraph mentions the [DYN] registry tokens, but fails to reference appendix B1. It may be appropriate to reference it here.\n\n### Section 5\n* Equation 5.4 is presented without proof, with the authors claiming they used an approach described in Eynard et al. (2023). A sketch of the proof (perhaps in an appendix) will contribute to the work's soundness.\n* Figure 3, and the relevant experiment, are based on the assumption that $A$ is rational. The authors should consider justifying the choice of testing only rational values of $A$, perhaps by connecting it back to equation 5.3, as proven by Aggarwal (2021).\n* Figure 3 presents a significantly higher value of $R^2$ for $A=2/3$ compared to the values for $A=4/6$ and $A=6/9$, despite being identical numbers. This issue does not appear for other such sets of identical rational numbers, such as $A=3/4$ and $A=6/8$. Since the rest of the subsection on Abductive Reasoning relies on $A=2/3$ being the correct answer, **this error calls the entire subsection into question and significantly hurts the paper's soundness and overall rating**. The authors must justify how the $R^2$ of $A=2/3$ is different from the other two values, or replace the figure (and perhaps rewrite some of the supporting text). Based on the other values of the figure, it should be expected to see a maximal $R^2$ around $A=2/3$, but without such a significant jump.\n\n### Typos\n* Section 5.1 line 319: \"The topological recursion formula equation 2.4 [...]\". Consider removing either \"formula\" or \"equation\", or placing all of \"equation 2.4\" in parentheses.\n* Section 5.1.1. has multiple citations included in sentences with their parentheses. The ICLR 2025 formatting instructions (section 4.1) require such references to not have parentheses except around the year. The references in question appear in lines 371, 378, and 381.\n* Section 5.1.1 line 417: \"As a result, We find an evidence [...]\". \"We\" does not need to be capitalized, and \"an\" should be removed.\n* Appendix C line 950: \"Figure 6 shows (s) numerical [...]\".\n* The title of appendix D and the caption of figure 7 both mistakenly write Princip**le** Component Analysis instead of Princip**al** Component Analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "otpaJRGeGN", "reviewer": "Reviewer_daJv", "summary": "This paper introduced DynamicFormer to learn and predict the $\\psi$-class intersection numbers. Experiments include both in-distribution results and out-of-distribution results. The author also presented some experiments to illustrate how transformers perform enumerative geometry. Meanwhile, the authors also investigated whether the proposed method could perform abductive reasoning and hypothesis testing to estimate the parameters of asymptotic form for intersection numbers.", "questions": "+ I wonder why the authors choose transformers as the regression function? \n+ In Figure 1, have you tried to apply DRA to MLP or other potential neural networks?\n+ Will the code and the datasets be available?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The idea of using transformers to do enumerative geometry is new. \n     + Meanwhile, the authors proposed a new activation function, DRA, which found to be useful to improve the prediction performance.\n     + The authors compared DRA with other popular activations functions in Figure 1 and Table 2.\n\n+ Experiments show some evidence of transformers can learn to predict the $\\psi$-class intersection numbers.\n     + Meanwhile, the authors also presented a discussion on how transformers being able to achieve that by inspecting internal vector space of the model.\n\n+ The author also investigated how inputs affect the model’s understanding of $\\psi$-class intersection numbers and the parameters for large genus.", "weaknesses": "I am not an expert in \"enumerative geometry\". However, I think the paper lacks many important clarifications and discussions.\n \n+  The paper lacked discussion of the reasons/motivations of using transformers.  At the moment, the paper seemed only a combination of  a popular neural network architecture and a new mathematical problem.\n+ The \"Related Work\" section is quite weak at the moment: the authors spent only one paragraph to discuss related works and then summarized their contributions.\n+  From my perspective, the proposed DRA is not the only way to capture the periodic behavior in data. This lacks sufficient discussion in the paper.\n+ From experiments in Figure 1, the authors did not apply DRA to other neural network architectures (eg MLP), and provided readers with more discussions on that.\n+ Lack of theoretical discussion on the proposed method.\n+ Code is not available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mnXE8hS5UI", "reviewer": "Reviewer_Yn4i", "summary": "The authors investigate the ability of transformers to compute the psi-intersection numbers in geometry and found that they perform (unsurprisingly) very well in distribution and quite well outside of distribution, and performed a series of analyses to understand which structures are being learned this way. They find in particular that the model learns the Dilaton equation and some information about the exponential growth of these psi-intersection numbers.\n\nOverall, this paper is written very carefully, with excellent explanations of what is being done, although the importance of some details is unclear (like: what do we need to know about psi-intersection numbers? most of the sophisticated formulae are not really used in a meaningful way). However I am not sure the work is very interesting from a geometric point of view (the interesting thing is to gain theoretical insight into what psi-intersection numbers, not get somehow numerically accurate estimates of them) or from a machine learning point of view (it is not clear what is more interesting about these numbers than about any sequence in the OEIS, say). The experimental results are not particularly surprising given what is known about transformers, or at least I don't see it. \n\nWhile this work can be viewed as a first step towards making progress in applying machine learning to enumerative geometry, and the carefulness of the writing and experiments should be commended, I don't think it brings a lot of interesting new informations about machine learning or enumerative geometry.", "questions": "It would be good if the authors could at least mention questions that would bring something interesting to the enumerative geometry (something feels interesting when they perform an analysis of the internal representation of the network, but it stops just before it gets interesting...).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The care and clarity of the writing, the fact that some extensive research has been done, the general trust in the results that this paper inspires.", "weaknesses": "What do we learn about machine learning or enumerative geometry? We seem to learn something that could be expected, a particular case of a general phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MIgYxiRXlN", "reviewer": "Reviewer_qDDj", "summary": "Unfortunately, I have no expertise at all in computational enumerative geometry. My review will thus be quite superficial.\n\n*Summary*\n\nThis paper proposes to use transfomer models to tackle what I understood is a central problem in enumerative geometry: computing the phi-class intersection numbers on the moduli space of curves. From my pretty crude rudimentary and pragmatical ML perspective, the authors reduce this problem to learning a multi modal function mapping input tuples of the form (quantum Airy structure datum [a tensor / sequence of tensors], genus [integer], number of marked points [integer], partitions [permutation-invariant set]) to output intersection numbers [sequence of integers (?)]. The model is trained on solutions computed using brute-force methods up to some genus, and evaluated on its ability to extrapolate to find solutions for higher genus (geni?). \n\nThe main technical contribution of the papers are methodological and consist in \n\n(i) designing a specific multi-modal transformer architecture suited to the problem at hand (combining mostly existing models / techniques)\n(ii) introducing a novel activation function specifically suited to model recursive functions, which are crucial to solve the problem. \n\nExperiments on synthetic data are provided demonstrating that the model seems to be able to extrapolate to higher geni than the ones seen in the training data. The authors also provide some more qualitative analysis to investigate to which extent the internal representations learned by the model encode mathematical structures that are known to be relevant to solve the problem.", "questions": "*Recommendation*\n\nI recommend to reject the paper mainly because I believe ICLR is not a suited venue, both for referring this paper (this paper needs to be reviewed by at least one expert in enumerative geometry, I don't know if there are such reviewers at ICLR) and for disseminating it (a journal in the field of computational enumerative geometry may be more suited). Furthermore, in my opinion the presentation of the material can be improved in several aspects before publication.\n\n \n*Comments and questions*\n\n- To which extent incorporating the conformal prediction framework in your analysis necessary? I am afraid this adds an additional layer of complexity that further hinders the communication of your findings. Maybe this discussion should be deferred to the appendix, keeping only what is strictly necessary to understand the main conclusion of your experiments in the main paper. \n\n- I don't understand the paragraph on top of p. 7, and I don't think this is due to my lack of expertise in enumerative geometry. In particular, what does \"the neural network embedding p_g,n ... is a vector space\"  means ? How can a function be a vector space? What does \"go to the inner product space\" means? These are (to me) very loose nonsensical mathematical statements.\n\n*Minor comments & typos*\n\n- p.3 the acronym COO has not been introduced\n\n- Figure 5 should be included in the main part of the paper. In general, avoid forward references to far away, especially in the appendix without mentioning that it is in the appendix. \n\n- Use capitalization when reference tables, figures, sections, equations, etc. in the text (no capitalization needed when referring to figures or tables in general). E.g. Figure lines 168, 197, 334, Section lines 204, Table lines 274,298, Equation lines 319, 389 ... ... \n\n- line 196 we -> We\n\n- line 421: the sentence \"The interesting thing is that this is the performance of the non-linear probe.\" could be rephrased to better suit a formal publication.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "(S1) Investigating to which extent the recent successes of transformer models can transfer to other tasks, such as the one of solving fundamental problems in mathematics, is worthwhile and relevant.", "weaknesses": "(W1) The relevance and technical aspects cannot be well understood / evaluated unless the reader has some non-trivial background knowledge of enumerative geometry.\n\n(W2) The writing and exposition of the material can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)