KinFormer: Generalizable Dynamical Symbolic Regression for Catalytic Organic Reaction Kinetics
Authors: Jindou Chen
Jidong Tian
Liang Wu
ChenXinWei
Xiaokang Yang
Yaohui Jin
Yanyan Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Kinetic Equation Prediction
Dynamical Symbolic Regression
Transform; Conditional Strategy
MCTS
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a generalizable Transformer-MCTS framework for chemical reaction kinetic equation prediction

Abstract: Modeling kinetic equations is essential for understanding the mechanisms of chemical reactions, yet a complex and time-consuming task.  Kinetic equation prediction is formulated as a problem of dynamical symbolic regression (DSR) subject to physical chemistry constraints. Deep learning (DL) holds the potential to capture reaction patterns and predict kinetic equations from data of chemical species, effectively avoiding empirical bias and improving efficiency compared with traditional analytical methods. Despite numerous studies focusing on DSR and the introduction of Transformers to predict ordinary differential equations, the corresponding models lack generalization abilities across diverse categories of reactions. In this study, we propose KinFormer, a generalizable kinetic equation prediction model. KinFormer utilizes a conditional Transformer to model DSR under physical constraints and employs Monte Carlo Tree Search to apply the model to new types of reactions. Experimental results on 20 types of organic reactions demonstrate that KinFormer not only outperforms classical baselines, but also  exceeds Transformer baselines in out-of-domain evaluations, thereby proving its generalization ability.

Published as a conference paper at ICLR 2025
KINFORMER: GENERALIZABLE DYNAMICAL SYM- BOLIC REGRESSION FOR CATALYTIC ORGANIC REAC- TION KINETICS
Jindou Chen S! Jidong Tian 1.1.2 Liang Wu' Xinwei Chen' Xiaokang Yang' Yaohui Jin Yanyan Xu MMoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University School of Chemistry and Chemical Engineering, Shanghai Jiao Tong University
ABSTRACT
Modeling kinetic equations is essential for understanding the mechanisms of organic chemical reactions, yet a complex and time-consuming task. Kinetic equation prediction is formulated as a problem of dynamical symbolic regression (DSR) subject to physical chemistry constraints. Deep learning (DL.) holds the potential to capture reaction patterns and predict kinetic equations from data of chemical species, effectively avoiding empirical bias and improving efficiency compared with traditional analytical methods. Despite numerous studies focusing on DSR and the introduction of Transformers to predict ordinary differential equations, the corresponding models lack generalization abilities across diverse categories of reactions. In this study, we propose KinFormer, a generalizable kinetic equation prediction model. KinFormer utilizes conditional Transformer to model DSR under physical constraints and employs Monte Carlo Tree Search apply the model to new types of reactions. Experimental results on 20 types of organic reactions demonstrate that KinFormer not only outperforms classical SR algorithm baselines, but also exceeds Transformer baselines in out-of-domain evaluations, thereby proving its generalization ability.
INTRODUCTION
A mechanistic comprehension of chemical reactions is imperative for devising novel catalysts, exploring various modes of reactivity, and developing environmentally friendly and sustainable chemical processes (van Dijk al. 2021 Salazar et al. 2020 Butcher et al. 2020 Hutchinson et al., 2021]. Reaction mechanism and its kinetics are two sides of a chemical reaction, both of which are of fundamentally importance. Reaction mechanism helps at molecule level and identify the reaction pathway, and the reaction kinetics on the other hand is the quantitative understanding of the factors influencing the reaction rate. Conventional pipeline for kinetics analysis involves three stages: (1) assuming the possible reaction mechanism based on existing knowledge of chemistry; (2) deriving kinetic equations differential form based on the possible mechanism and integrating the kinetic equations; (3) fitting the kinetic equations with experimental data (Bédard al. 2018 Shi et al., (02T). However, traditional kinetics analysis is limited by human knowledge of chemical reaction and more probably introduces empirical errors, and requires case-by-case analysis, resulting in low efficiency and poor generalization (Burés & Larrosa 2023). To avoid these challenges, recent studies of deep learning provide insights for reaction kinetics analysis, such as automatic reaction mechanism exploration (Yang et al. 2019)Jormnr et al.) 202T Feng & Wang 2023), kinetic property regression (Farrar & Grayson 2022 Garciiaa Gnddia-Andrade et al. 2023) and rate constant estimation (Gao et al. 2016 Maeda et al.) 2022). These methods typically target only one specific reaction may require expert- guided training, and cannot be generalized.
gain insights into chemical reaction
Actually, the essence of reaction kinetics analysis lies in kinetic equation prediction (KEP), which combines mechanism generation and constant estimation. Specifically, a catalytic organic reaction of ext P is taken as the example: S (substrate) combines with cot (catalyst) to form cot.S
Equal contribution
Corresponding authors: /frank92, jinyh, yanyanxu] @ @sttuedu..o

---

Published as a conference paper at ICLR 2025
cat
d[S]
k-1 (ca[c) [][S][cat]
de
S
Substrate [5]
AsfeatS] K_. [cat([P
Coneastrant
Product [P]
Core
d[cat]
(k-1 ky)[eas]] (Ac 2 (PD)c
Catalyst
[ea]]
dt d[catS] (k,[5] + [[](]]) (K., kc][aass]
Intemnaliaii [carS]
catS
(a) Mcchanism diagram
(b) Kinetics equation
(c) Temporal concentration profiles
Figure 1: Illustrations of catalytic organic reaction. (a) shows the core mechanism (MI) with transformation relations among species. (b) provides the governing equations (ODEs) under the constraints of mass-action law. (c) presents one of the corresponding temporal concentration profiles.
(intermediate), subsequently it decomposes to P (product), and cat is recovered, as shown in Figure D.. According to the mass-action law, the chemical mechanism is mathematically controlled by corresponding ordinary differential equations (ODEs), shown in Figure Ib ([-) represents the concentration and the reaction rate constant). The form of the ODEs is determined by the type of reaction mechanism. In this contribution, due to the dependence of kinetic data with the underlying reaction mechanism, the objective KEP is to predict the whole set of ODEs from a group of kinetic data (temporal concentration profiles, shown in Figure Ibb.
In this work, we innovatively formulate KEP as a dynamical symbolic regression (DSR) prob- lem (Brunton et al. 2016; Weilbach et al. 2021) with physical constraints derived from the mass- action law. Our goal is to instruct model to successfully regress ODEs from temporal concentration profiles, and possess generalization capabiliiy. Classical SR methods (Brunton et al.) 2016 Cranmer 2023) are usually individual methods without any generalization ability, and performs poorly in specific domain due lack of pre-training stage. With advent of Transformer (Vaswani al. 2017). researchers have developed pre-trained prrueeccett-sqquene tasks. ODEFormer (d Ascoli al.) 2023) first introduces Transformer into DSR, showing surprising performance. However, directly applying ODEFormer to KEP faces challenges in generalization, as the common training strategies cannot make the model effectively learn physical constraints controlled by the mass-action law. On one hand, the universal training strategy (d' Ascoli et al. (023) ODEFormer adopts, generates the entire ODEs in an end-to-end manner, leading to overfitting on training patterns, good in-domain performance, but poor generalization. On the other hand, inspired by P-tuming (Liu et al., 2022). the independent strategy generates each equation with the pre-defined prompt, completely ignoring the physical constraints and correlations among equations. To address these issues, we first design a conditional strategy to train Transformer on simulated kinetic data, which randomly selects several equations conditions, and chooses additional one to predict. The strategy adopted in the equation level more balanced to enables the model to capture such physical correlations. However, conditional strategy requires the reasonable and explainable generation order to further improve the generalization ability of the model. To tackle this, we introduces a generation order search module with Monte Carlo Tree Search (MCTS), which combines with the conditional strategy and forms a new generalizable KEP framework, KinFormer, We experiment on simulated dataset with 20 types of catalytic organic reactions (Burés Larrosa 2023). Results demonstrate the powerful generalization of KinFormer compared with other baselines. In addition, Monte Carlo trees provide an explanatory approach for how KinFormer predicts reaction kinetic equations.
Transformer models for SR
In general, our contributions include: (1) We propose generalizable training strategy, the conditional strategy, to solve improve the generalization ability KEP. (2) We design an MCTS post-processing module to search the optimal generation order and providing explanations corresponding to the conditional strategy, which forms the KinFormer framework. (3) Experiments on a catalytic organic reaction simulated dataset demonstrate KinFormer's generalization ability.
2 RELATED WORKS
2.1 SYMBOLIC REGRESSION
Symbolic regression (SR) aims to discover the most optimal mathematical function that can accurately fit dataset. One of the dominant methods for SR is Genetic Programming (GP) algorithm that
2

---

Published as a conference paper at ICLR 2025
simulates the evolution of human history (Schmidt & Lipson 2009 Btadek & Krawiec 2022 Trujillo
et al., 2016: Tohme et al.] 2022 Virgolin et al.) 2021). Afterwards, SR has garnered increasing
attention from the deep learning (DL) community, driven by the observation that neural networks
(NNs) excel in discerning qualitative patterns (Petersen et al. 2019 Udrescu & Teggarrk 2020
Udrescu al. 2020 Holt et al. 2023). The advent of transformer (Vaswani et al.] (0177) have
shown remarkable performance in natural language processing, which has inspired researchers
to develop pre-trained transformer models for SR (Vastl et al., 2024; Lample & Charton 2019)
Charton 202T). After large-scale pre-trate training on synthetic data, inference often achieves significant acceleration, since it necessitates no training for unseen dataset (Biggio al. (02I). For example, the E2E framework (Kamienny et al., (022) is end-to-end and designed to forecast the entire equation, including constants. Most of these works concentrate on functional SR, instead of ODEs.
2.2 DYNAMIC SYMBOLICAL REGRESSION
Dynamical SR, as a special case of SR, refers to the inference of ODEs based on temporal data (1, (,,) A significant obstacle lies in the lack of regression targets d(I) since temporal derivatives
are usually not observable directly. A typical remedy involves utilizing numerical approximations
of the absent derivatives as substitute objectives (Gaucel et al., 2014 La Cava et al. 2016 Brunton iet al. 2016). NNs have also been integrated with GP for dynamical SR (Udrescu et al. 2020 Omeje
ctt al. 2023 Weilbach et al. 2021), but there still exists generalization problems due to the absence of prior knowledge. NSODE (Becker et al.) 2023 is transformer-based method for dynamical SR but can only work on univariate ODEs. ODEFormer (d' Ascoli et al.] (023) is proposed to
infer multidimensional ODEs in symbolic form from the observation of single solution trajectory. However, the training strategy of ODEFormer leads to poor generalization, especially in KEP.
3 PRELIMINARIES
3.1 KINETIC EQUATION PREDICTION
Kinetic equation prediction (KEP) is a kinetics analysis task to discover the ODEs (denoted by f' of
an catalytic organic reaction (S cut, P) according to temporal concentration profiles. Therefore, the input of KEP is the representation of profiles, time series of X = {(t, x(t))]t E {1, (1. ?)). where T' is the maximum timestep and x is a D-dimensional vector corresponding to the concentra- tions of reaction species, including substrates, products, catalysts and intermediates (x, (t)(+))s E
{S, P. cat, catS. ... }). The profiles are governed by kinetic equations (ODEs), shown in Equation The objective of KEP is to formulate f from observed temporal concentration profiles.
d x(t) = f(x(t)), f: R:R R dt
Hence, KEP can be formulated as a DSR task (Brunton et al. 2016 d' Ascooli et al. 2023 where f can be inferred in symbolic form constrained by the mass-action law R. In practice, f can be converted into prefix notations y for deep learning (Lample & Charton 2019). b. It means that y g(f) and f f=g- (y) hold, where g and g -1 are conversion functions. Therefore, given the training dataset D D= ((X,)) = {(t, y)) (X {(t, E 21,2, T),, y [y1, ".. ,yrl, yo is the symbolic token in the ODE, such as "add", "mul" and discrete numeric symbols), deep learning-basee KEP is train a DSR NN Fa with parameters 0, making v(X, y) E D satisfy Equation 2 fy
y Fo (()),
d xx(1) = f((()) RFF
R F f means the form of f should satisfy physical constraints R. Actually, it is challenging to explicitly model R into Fo to guarantee compliance and generalization. In this work, we instead model R implicitly through novel training strategy and MCTS, which is consistent with Equation 2
3.2 CATALYTIC ORGANIC REACTION
Although Equation provides a general objective of KEP, we focus on catalytic organic reactions, following the previous Nature work (Burés& & Larrosa 2023). We have considered 20 commonly
3

---

Published as a conference paper at ICLR 2025
0.49
0.49 0.18
1 0.49
0
0.18
1
0.22x, -0.43x
Embedder
dxD
Embedder
Embedder
Prefix-intix conversion
Encoder
Encoder
Encoder
enosz, drti, add,
mud, +, N2280,
F4,,g,
ceos
Embedder
<00S>
Embedder
Embedder
N1000, mul, +, N2043, E-2, mul, Xp,
Decoder
Decoder
Decoder
xx, dx1
V
add, mul, +, N2280, E-4, 2y, mul,
mul, +, N2280, F-4, A,,
add, mul, + N2931, F F,,, mul, -, N10000 -3, mul, N4890, E-4, mul, Xx.
Input
N1000, & 5-3, mud, N2043, E-2, mul, add, N2931, 84,,8
-N10000E- -3, mud, ma,,
N2044,
8-2, mul. xa. <e0s>
Model
O Profix-iniix conversion
Prefix-infix conversion
Prefix-infix
Output
0.22x; 20.43x,
0.22xg- 20.43x,
4 0.29x, 0.49xxux
0.29kr
0,49x,X1
Equation
(a) Universal strategy
(b) Independent strategy
(c) Conditional strategy
Figure 2: Comparison of three types of training strategies. (a) the framework of universal Transformer that generates the whole ODEs according a prior order. (b) the independent Transformer that independently generate each ODE through an additional prompt, such as dx0. (c) our proposed conditional strategy that provides contexts for decoder to capture correlations among different ODEs.
encountered types of catalytic organic reactions, whose species can be simplified into unified ex- pressions: substrates (S), products (P), catalysts (cal) and intermediates (cal.8, cal2S..... These 20 types can further be categorized into four groups of distinct chemical mechanisms: (1) core mechanism (MI), shown in Figure T (2) bicatalytic mechanisms (M2-M5); (3) catalyst activation mechanisms (M6-M8); (4) catalyst deactivation mechanisms (M9-M20) (See Appendix for more details). Like other chemical reactions, each type of reaction is mathematically described by set of rigorously expert-verified ODEs function of kinetic constants k and concentration variable of chemical species. Therefore, solving ODEs allow the generation of an infinite number of temporal concentration profiles of reactions, constituting a kinetic space. In this work, we choose to solve the set of ODEs that describe the kinetic behavior of each mechanism function of kinetic constants create catalytic organic kinetic datasets.
METHODOLOGY
4.1 CONDITIONAL STRATEGY
Inspired by ODEFormer, we adopt the same Transformer backbone that encodes numerical inputs by the discrete three-token method Ascoli al.) 2023). The model architecture, optimization and other encoding/decoding techniques are all aligned with ODEFormer. On the backbone, we explore three types of training strategies: universal, independent and conditional strategies.
Universal strategy provides an end-to-end encoder-decoder model that generates the whole ODEs for reaction, which is also the strategy of ODEFormer. Figure 2 shows the whole framework of the universal one. The encoder is used to encode the discrete time series X, while the decoder generates each token autoregressively starting from the "<eas>" token. Therefore, the objective of the universal strategy is shown in Equation 3, where yo is the "<eas>" token, y<i [yo. 1), Po(.) is the probability calculated by the netwrok, and represents trainable parameters.
max log X) = log II
X.(e)) = Σ log X.( i=1
(3)
4

---

Published as a conference paper at ICLR 2025
Iterate
Selection
Expansion
Simulation
Backpropagation
<stort>
<stort>
<start>
<start>
equation
equation 3
equation 0
equattio
equation 0
equazion
equction
equationn 3
equation
equation 2
equction
equation 2
equation
equation
autoregressive generation
reward computation
Reward R2m
Condition
Conditional strategy
Complete ODEs
>>>>>
equation
R2
Figure 3: MCTS generation order search with based on conditional model.
Independent strategy re-splits the whole ODEs into each independent ODE. Specifically, given kinetic equations y, includes several separate ODEs e. Therefore, y = [e1, ... epl holds, where is continuous token segment of and represents the prefix
D) is the dimension of species. notation of the êth ODE. To indicate which ODE to be predicted, the independent model introduce a prompt token p; with the form of "dxi". Figure 2p shows the framework. Following P-tuning (Liu et al. 2022), the prompt is encoded in the encoder instead of the decoder. In general, the objective of the independent strategy is shown in Equation 4 From the equation, this strategy assumes the independence among ODEs, so each ODE is predicted separately through prompts.
max log Xo(y/X)= log II Pole)(Xpp))
Σ log Po(e:(X,p))
Conditional strategy is balanced strategy that have proposed. It also re-splits into ODE segment e; Different from the independent one, conditional strategy models the correlations among e, by introducing the condition (Figure P demonstrate how use the condition prompt the model). To calculate unordered we shuffle [e1, epl and acquire a shuffled list [e'1. (-' e'pp D]. where y shuffle([e).
ep]) = [e' 1,,... e' D]. Therefore, c; comprised of the first i-1 ODEs from the shuffled list (c, = jc'1, .(;-1]) and the objective of the conditional strategy is shown in Equation [ where с is empty set. From the equation, this strategy treats each ODE as whole, introducing correlations between among ODEs through conditional probability modeling. During the training stage, the number and elements of the condition are randomly sampled to ensure the model's generalization. During the inference phase, conditions added one by one.
max log Poly X) log II Po(c
Σ log Po(c's(X,c,)
(5)
Comparing three strategies, the universal strategy, without incorporating any prior knowledge, allows the model to autonomously learn and explore the physical constraints R within KEP, resulting in pattern rigidity and lack of generalization ability. The other two focus on modeling each individual ODE in KEP. Independent strategy ignores the mutual constraints among ODEs, with each equation adopting an optimal solution. However, the predicted ODEs do not satisfy the constraint (R f). Conditional strategy effectively captures physical constraints R by directly modeling the correlations among ODEs. Compared with the former two, it provides a generalizable framework for KEP. However the challenge of the generation order still remains.
4.2 GENERATION ORDER SEARCH
To solve the order challenge, introduce novel Monte Carlo Tree Search (MCTS) search module into KinFormer based on conditional strategy. as shown in Figure 3 Different from previous
works (Holt et al. 2023 Shojaee et al. 2023 Li al. 2024). the proposed search module does not
5

---

Published as a conference paper at ICLR 2025
focus on the generation order of each token, but aims to coordinate with conditional strategy to search for the generation order of individual ODEs. Therefore, nodes of Monte Carlo the represent independent equations (e'), while edges reflect correlations between two equations. To complete an iteration, the MCTS search module includes four major stages: Selection, Expansion, Simulation, and Backpropagation (Silver al. 2017).
Selection: When given the current activated node c',, the next activated node should be selected from e','s child nodes. In this work, the probabilistic upper confidence bound heuristic (P-UCB) (Silver et al. 2018) is adopted to execute the selection operation. In details, the node with the maximum P-UCB score (S) is activated. The calculation S in the Equation where e' s+1 E child(e', :) is the next activated node, one of the child nodes of e's, alpha is constant, Pg is the probability calculated by the neural model, C++1 [e' e'.] ]] is the condition. MCTS, V(.) and N()) are two important indicators to record the cumulative score and the number of traversals, respectively.
S(e' V(e(++1) a- Po(e' (6(ee++1(X,c--1) N(e'1+1)
In N(e..) V 1+ N(e' e'it1 child(e',)
(6)
When the activated node c'; is leaf node, to facilitate tree growth, the expansion operation generates all child nodes (the next possible equations) of the current one. this work, all untraversed ODEs can be predicted by Equation which is the expansion of e';.
Expansion:
arg max, Po(n|X, (e'))) n child(e',) y/(c U (e'i))
Simulation: Given an activated node e';, the proposed module employs action random select sequence of the rest ODEs and executes the action through the neural model step by step, resulting in simulation result y TTeen, the evaluation score score can be computed by Equation where r2.. and a2 are two R2 metrics that will be defined in the next section, + B = 1.a 0, 2,3200.
score(e',) r2m (y' X) + 3 (22
(8)
Propagation: After simulation, the activated node's score is recursively backpropagated to its parent node until reaching the root node (N = N + 1 and V = V + score).
We set iterations the maximum N (Q is used for equal N values) is selected as the final ODEs. The detailed MCTS algorithms and the specific example shown in the Appendix.
100 for our MCTS framework. After removing error nodes, the leaf node with
EXPERIMENTS
5.1
EXPERIMENTAL SETTINGS
For three training strategies, construct different formats of training data, as shown in Figure 2 Because twenty types of reactions belong to four chemical mechanism categories mentioned in 3.2 and reactions under the different categories possess the distinct pattems, have defined two distinct modes: Intra-class and inter-class generalization. Intra-class: randomly hold out one reaction type from bicatalytic mechanism, catalyst activation and catalyst deactivation (e.g. M5.M6 and M20). inter-class: hold out all three reaction types within catalyst activation (e.g. M6-M8). The detail of data sample construction presented in Appendix C three training strategies and two classical SR methods, namely SINDy (Brunton et al.) 2016) and PySR (Cranmer 2023). Although SINDy and PySR are usually plug- aun-and--lay, and do not possess pre-training and the concept of "ggenraliizaton". we still conduct comparison experiments to prove the superiority of KinFormer. Following the previous work (d' Ascoli et al. 2023), we compute di xt) by the central finite difference algorithm as regression targets. Each test sample corresponds to one ODE system, so we apply SINDy or PySR to each sample and test each individual model. In SINDy, only the polynomial dimension (d) of hyper-parameter
We mainly compare KinFormer with
significantly influences the results, setting.
exhibit the performance of n E {2, 3, 4, 5). In PySR, we keep the default To further illustrate effectiveness MCTS, also perform the equation-level beam search to replace

---

Published as a conference paper at ICLR 2025
MCTS. However, as beam search is time-consuming set the iteration n = 20, sample 100 cases for each type of mechanism, and keep KinFormer under the same setting for a fair comparison. is worth noting that the comparison between the conditional strategy and KinFormer can be regarded as the ablation study of the MCTS module.
5.2 EVALUATION METRICS
In this work, we adopt R2- 12-based measurement and RMSEE as our evaluation metrics. R2-score (avv et al., [202]) is classical metric for regression tasks. R22-ccre is unbounded from below and single outlier prediction causes severely biased score. To circumvent this, we set 0 for any negative R22-ccore (Kamienny el al. 2022 d' Ascoli et al. 2023). Based on the definition of R2-sccore, we define four corresponding metries to evaluate models. The first two are micro-R2 (m2m) and macro-R2 (r2M). r2m is the average R2-score on the test set. Due to the smaller scales of certain species (e.g. intermediate) compared with other species, r2.m might overlook the significance of these species. Instead, r2M calculates the R2-score for each generated species and then take the average. The latter two are accuracy scores. We consider a sample with RR--ccore greater than 0.9 to be a correct one. Values calculated by It2m and M22M are defined as micro-accuracy Accm and macro-accuracy AceM, respectively. Furthermore, we also compare the accuracy of the equation form (Aceform) Specifically, fcom is calculated by exact match in form neglecting kinetic constants. RMSE is calculated to measure the difference between the predicted and actual values. We report the median of RMSE to avoid the influence of outliers. is the average inference time.
5.3
MAIN RESULTS
presents the main results of KinFormer and other baselines. SINDY and PySR does not perform well as KinFormer. We speculate that one possible reason that classical SR methods lack domain-specific knowledge (e.g. mass-action law) due to the absence of pre-training, Another reason probably is the imprecision of the approximated temporal derivatives. Comparing three training strategies, firstly, the universal strategy has performed optimally across all metrics in in-domain settings, demonstrates the power of Transformer on DSR. In both out-of-domain scenarios, however, it performs poorly, failing to predict the ODEs form correctly. This indicates that the universal strategy lacks of generalization capability, merely memorizing formulas seen in training. Secondly, the independent strategy underperforms in all three scenarios, which is within expectations. When training a model to learn a specific equation of ODEs based on the entire concentration profiles, several irrelevant information is introduced and physical constraints from other equations are absent, which results in inaccurate predictions. Thirdly, the conditional strategy has made significant progress compared with the other strategies in two oft-of-domain scenarios, even fcm exceeding 70%. The improvement demonstrates its generalization capability and proving our assumption that the extra conditional information enables the model successfully learn the physical constraints among equations. Last but not least, the model trained under the universal strategy have the fastest inference speed, but the difference among the three is not significant.
Table
KinFormer incorporates the conditional strategy and the MCTS module, which has achieved the superior results. Specifically, compared with the universal strategy, in the intra-class scenario, Accm, Accas, R2m. and R2M increase by 47.17%, 21.31 %, 0.563 and 0.445, respectively; in the inter-class scenario, Accm, ACEM. A22m and R22 increase by 30.42%, 18.52%, 0.399 and 0.410, respectively. Overall, the experimental results demonstrate that KinFormer (n 100) significantly enhances generalization capability by implicitly modeling physical constraints through the conditional strategy and optimizing the sequential order of formulas generation via MCTS post-processing. Consequently, KinFormer is capable of predicting successfully for reaction types it has not previously encountered. To further illustrate the selection of MCTS instead of other search metheds, we exhibit the performance of equation-level beam search. Under the same setting (n = 20), the inference time for Beam Searct is almost three times that of MCTS. Considering performance, MCTS algorithm consistently outperforms Beam Search in terms of both accuracy and efficiency. These results indicate that MCTS leverages the physical correlations learnt by conditional strategy search the better generation sequence. In addition, disregarding the differences in data volumn, MCTS with 100
In2 20 for Beam Search means we retain at most the top 20 equations after each equation generating step (e.g. generate [dr0, dr3] from [dz0] or generate [dx0, dn3, dzd] from [dz0, dz3] ) and end up with 20 ODEs.
7

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation Results of KinFormer and other baselines. ID, (Itra)) and (OD(In)ra) represent in-domain, intra-class, and inter-class evaluations, respectively. The bolding methods are proposed this work. means summary result of individual models for each test sample is reported. Underline "n=20" means that test set contains 100 samples for each type of mechanism.
Methods
Acem
ACcM
fcom
R2m
R2M
RMSE
t(s)
SINDy*
d=2
1.59
0.72
0
0.020
0.042
174.9
1
d=3
0.09
0.04
0
0.002
0.032
5.252
1
d=4
0
0
0.001
0.028
1.175
d=5
0
0
0
0.003
0.042
1.278
1
PySR
24.32
7.85
0.315
0.434
0.063
36
ID
Univeral
71.49
38.02
67.80
0.754
0.689
0.008
6
Independent
4.14
1.86
0
0.059
0.133
0.248
10
Conditional
36.08
14.78
57.15
0.471
0.447
0.018
15
BeamSearch
==20
35.48
17.70
63.60
0.463
0.455
0.010
669
KinFormer
==20
63.45
28.23
64.99
0.726
0.638
0.007
222
n=100
59.76
27.56
64.00
0.698
0.629
0.010
371
Universal
10.40
1.87
0
0.158
0.183
0.095
6
Independent
0.47
0.07
0
0.009
0.077
0.304
10
Conditional
31.73
9.67
70.28
0.455
0.425
0.023
15
OOD
(Intra)
BeamSearch ==20
31.67
12.67
73.19
0.498
0.469
0.017
669
KinFormer
#=20
50.83
18.75
79.42
0.662
0.583
0.014
222
n=100
57.57
23.18
80.47
0.721
0.628
0.013
371
Universal 10.73
0.60
0
0.182
0.205
0.067
6
Independent
0.47
0
0
0.012
0.075
0.322
10
Conditional
23.40
4.40
74.30
0.325
0.397
0.044
15
OOD
(Inter)
BeamSearch ==20 23.00
6.67
76.05
0.325
0.404
0.021
669
KinFormer
==20
40.47
13.71
80.61
0.515
0.534
0.015
222
==100
41.15
19.12
81.41
0.581
0.615
0.012
371
iterations shows only marginal improvement over MCTS with 20 iterations, which supports that KinFormer can coverage within a small number of iterations, demonstrating high search efficiency.
5.4 GENERALIZATION ANALYSIS
We plot distribution diagrams of the R2-score under both intra-class and inter-class conditions, shown in Figure The distribution that more skewed towards the right indicates a superior fitting performance, implying better generalization. Since r2m is to evaluate the overall performance and r2M to assess the individual equation fitting performance, The analysis of the diagram can be categorized into four scenarios: (1) large r2,, and r2 mean great fitting; (2) small r2,, and r2M mean poor fitting: (3) large r2,, and small r2M mean overall good fitting but neglect of few species due to tiny scaling; (4) small r2,, and large r2ar means only few species predicted accurately.
For the intra-class scenario, it is evident from Figure 4 that the universal exhibits mediocre perfor- mance M5. Distributions for both M6 and M20 skewed notably towards the left, suggesting
8

---

Published as a conference paper at ICLR 2025
Bicatalytic Mechanism (M5)
Catalyst Activation (M6)
Catalyst Deactivation
Catalyst Activation (M6)
Catulyst Activation (M7)
Catalyst Activation (M8)
(M20)
KinForeer
KinFommer
10
12 04
M
Q2
04 06
60 08
Universa
Universal
(a) Distribution of R2-score under intra-class condition
(b) Distribution of R2-score under inter-cluss condition
Figure 4: Distribution diagrams of R2-score under out-of-domain conditions. Red curve represents
r2m and Blue curve represents r2 The vertical line represents mean value of R2-score.
Core Mechanism
Bicatalytic Mechanism
Catalyst Activation
Catalyst Deactivation
Figure 5: KinFormer distribution diagrams of R2-score under in domain conditions.
inability to generalize to these two mechanisms. However, KinFormer successfully shifts this distri- bution towards the right, demonstrating its capability to effectively generalize under the intra-class scenario. The inter-class scenario harder setting because the whole Catalyst Activation haven't 't been seen during training. As evidenced in Figure 4p, KinFormer achieves superior performance on both M6 and M7. Specifically, KinFormer improve the r2M distribution to the right obviously, which means KinFormer not only enhances the overall ODEs fitting performance, but also promote the species profiles fitting with small numerical scale. KinFormer and the universal strategy both fail on M8. We speculate that The dimension of most mechanisms are either or while the dimension for M8 is 6, thereby posing significant challenge in prediction. Meanwhile, KinFormer on M20, the universal strategy on M5 and M7 exhibit clear mismatches of the two R2-score, suggesting that the model still tends to overlook species at smaller scales.
5.5 MECHANISM CATEGORY ANALYSTS
We specifically analyze the performance of KinFormer based on the four reaction mechanism categories under in domain setting. From Figure [ firstly. it is obvious that KinFormer achieves the superior performance in Core Mechanism, because the first mechanism is relatively simple, whose dimension is only four and form is basic. Secondly, for Bicatalytic Mechanism, dimensions increase to five/six, raising the difficulty of the task. However, KinFormer still provides the moderate results, where r2., exceeds 0.6, which proves the effectiveness of KinFormer. Thirdly, Catalyst Activation Mechanism is the most difficult to predict with the lowest R2-score. Meanwhile, the two peaks of r2m of the category are close to 0 and 1, but its r2M is in the middle, indicating that the scenarios (3) and (4) mentioned in the previous section are both present. Fourthly, for Catalyst Deactivation Mechanism, which also a challenging task, the distribution of R22-ccree is obviously shifted the left and r2m, is almost 0.8. In addition, Kinformer also neglects small-scale species in this category.
5.6 GENERATION ORDER ANALYSIS
The top three generated sequences of MCTS and frequency count for each reaction category under in-domain scenario shown in Appendix. It worth noting that almost all reactions of catalyst.

---

Published as a conference paper at ICLR 2025
4098
R 4999
Concentration profiles
04
Substrate
Catalyst
Intermediate?
000
B 1O
Product
Intermediatel
Intermediate3
Time
Figure 6: Case study M20 illustrate E2 errors. LHS part is complete concentration profiles and predicted profiles of six involved species. The individual subfigures are presented in RHS part. Different color represents different species. Black represents ground-truie sound-truth profiles.
deactivation prefer specific generation order: [O, P. cat.S, cat]. where O represents another inter- mediate distinct from cat.. fact, except for O and P. the ODE coefficients of other neighboring species correlated (e.g. in Figure 1, the k [S]]]]]] term appears simultaneously in the equations of d[S) and d(cat)). Therefore, the relative sequence are more important and we infer that KinFormer tends to select relative generation order that makes terms and independent coefficients between two consecutive ODE equations similar.
5.7 ERROR ANALYSIS AND CASE STUDY
The prediction errors of KinFormer mainly stem from the predictions M3, M8, M10, M15, and M20. From a dimensional perspective, the errors on M8, M10, M15, and M20 originate from their highest dimensionality (D = 6), indicating that KinFormer's ability for cross-dimesioonal prediction needs improvement. When classifying error types, errors on M3 and M8 belong to the category of unpredictable errors (EI), while errors on M10 and M15 belong to the category overlooking small-scale species errors (F2). The errors on M20 exhibit characteristics of both categories. We further perform a case study to illustrate E2. Figure 6 the case from M20 has relatively high r2m but low r2M. Since the reaction reaches steady state within the first 10 timesteps, took the first 10 timesteps. This is typical E2 error. From Figure 6 Kinformer cannot effectively predict Intermediatel and Intermediate3. Digging into the reasons, the ground truth of these species' concentration profiles is severely disrupted by Gaussian noise. KinFormer is unable to reconstruct the expressions from noisy data, especially when the noise magnitude is similar to the data scale. This indicates that E2 errors stem from species severely affected by noise.
only
LIMITATIONS
Limitations include: (1) The performance both in- and -fttof--omain needs further improvement. (2) The cross- diosediimniionall transfer capability is weak. (3) MCTS faces the issue of high time complexity when dealing with a large number of species.
CONCLUSIONS
Kinetic equation prediction an essential task for understanding the chemical reaction mechanisms. Traditional DSR models fail to provide generalizable performance under chemical domain. In this paper, we propose KinFormer to solve the generalization problem in KEP, which promotes the development of chemical kinetics analysis. Specifically, KinFormer takes advantage of the conditional training strategy to model DSR under physical constraints. Meanwhile, MCTS is applied as a post- processing step select the optimal generation order. Experiments synthetic catalytic organic reaction dataset demonstrate KinFormer's 's generalization capability. In conclusion, KinFormer is a generalizable DSR framework for KEP, which can inspire future research in other scientific domains.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGE
We thank SJTU AT for Science platform for the computing support. This work was supported by the Shanghai Municipal Science and Technology Major Project (2021SHZDZZ01002, and the Fundamental Research Funds for the Central Universities.
REFERENCES
Sören Becker, Michal Klein, Alexander Neitz, Giambattista Parascandolo, and Niki Kilbertus. Predicting ordinary differential equations with transformers. In International Conference on Machine Learning, pp. 1978-2002. PMLR, 2023.
Anne-Catherine Bédard, Andrea Adamo, Kosi C Aroh, M Grace Russell, Aaron A Bedermann, Jeremy Torosian, Brian Yue, Klavs Jensen, and Timothy Jamison. Reconfigurable system for automated optimization of diverse chemical reactions. Science, (2448)))12220-122 2018.
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural symbolic regression that scales. In International Conference on Machine Learning, pp- 936-945. Pmlr, 2021.
Iwo Blędek and Krzysztof Krawiec. Counterexample-drivee genetic programming for symbolic regression with formal constraints. IEEE Transactions on Evolutionary Computation, 2022.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 1 113(15):3332-3337 2016.
Jordi Burés and Igor Larrosa. Organic reaction mechanism classification using machine learning. Nature, 613(7945):689--69 2023.
Trevor W Butcher, Jonathan Yang, Willi M Amberg, Nicholas B Watkins, Natalie D Wilkinson, and John F Hartwig. Desymmetrization of difluoromethylene groups by c-f bond activation. Nature, 583(7817)5548 15488-55,21 2020.
W. L. Cava, Patryk Orzechowski, Bogdan Burlacu, Fabr'icio Olivetti de Francea, M. Virgolin, Ying Jin, Michael Kommenda, and Jason H. Moore. Contemporary symbolic regression methods and their relative performance. Advances neural information processing systems, 2021 DB1:1-16. 2021.
François Charton. Linear algebra with transformers. arXiv preprint arXiv:2112.00 2021.
Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01530 2023.
Stéphane d' Ascoli, Sören Becker, Alexander Mathis, Philippe Schwaller, and Niki Kilbertus. Odeformer: Symbolic regression of dynamical systems with transformers. arXiv preprint arXiv:2310.05573. 2023.
Elliot HE Farrar and Matthew N Grayson. Machine learning and semi-empirical calculations: synergistic approach rapid, accurate, and mechanism-basee reaction barrier prediction. Chemical Science, 13(25):75947766 2022.
Yajuan Feng and Chao Wang. Surface confinement of finite-size water droplets for so3 hydrolysis reaction revealed by molecular dynamics simulations based on a machine learning force field. Journal of the American Chemical Society, (5(119:10631-1042 2023.
Connie W Gao, Joshua W Allen, William H Green, and Richard H West. Reaction mechanism gener- ator: Automatic construction of chemical kinetic mechanisms. Computer Physics Communications, 203:212-225 2016.
Xabier
Pablo Gareía Tahoces, Jesús Pérez-Ríos, and Emilio Martínez Núñez. Barrier height prediction by machine learning correction of semiempirical calculations. The Journal of Physical Chemistry A, 127(10):2274-2222 2023.

---

Published as a conference paper at ICLR 2025
Sébastien Gaucel, Maarten Keijzer, Evelyne Lutton, and Alberto Tonda. Learning dynamical systems using standard symbolic regression. In Genetic Programming: 17th European Conference, EuroGP 2014, Granada, Spain, April 23-25, 2014, Revised Selected Papers 17, pp. 25-36. Springer, 2014.
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. arXiv preprint arXivv2400.000282, 2023.
George Hutchinson, Carla Alamillo-Ferre and Jordi Burés. Mechanistically guided design of an efficient and enantioselective aminocatalytic a-chlorination of aldehydes. Journal of the American Chemical Society, 143(18 (111188))80868888 2021.
Kjell Jomee, Anna Tomberg, Christoph Bauer, Christian Sköld, and Per-Ola Norrby. Organic reactivity from mechanism to machine learning. Nature Reviews Chemistry, 5(4):240-255, 2021.
Kamienny, Stéphane d' Ascoli, Guillaume Lample, and François Charton. End-to- end symbolic regression with transformers. Advances in Neural Information Processing Systems, 15:69991-0088. 2022.
William La Cava, Kourosh Danai, and Lee Spector. Inference of compact nonlinear dynamic models by epigenetic local search. Engineering Applications of Artificial Intelligence, 55:292-306, 2016.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412, 2019.
Yanjie Li, Weijun Li, Lina Yu, Min Wu, Jingyi Liu, Wenqiang Li, Meilan Hao, Shu Wei, and Yusong Deng. Discovering mathematical formulas from data via gpt-guided monte carlo tree search. arXiv preprint arXiv: 4444442401.14424, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Annual Meeting of the Association for Computational Linguistics, 2022.
Kazuhiro Maeda, Aoi Hatae, Yukie Sakai, Fred C Boogerd, and Hiroyuki Kurata. Mlago: machine learning-aided global optimization for michaelis constant estimation of kinetic modeling. BMC bioinformatics, 23(1):455, 2022.
Nina Omejc, Boštjan Gec, Jure Brence, Ljupčo Todorovski, and Sašo Džcroski. Probabilistic grammars for modeling dynamical systems from coarse, noisy, and partial data. 2023.
Brenden K Petersen, Mikel Landajuela, T Nathan Mundhenk, Claudio P Santiago, Soo K Kim, and Joanne Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint drXiv:1912.0887,, 2019.
Chase Salazar, Kaylin N Flesch, Brandon E Haines, Philip S Zhou, Djamaladdin G Musaev, and Shannon S Stahl. Tailored quinones support high-turnover pd catalysts for oxidative c-h arylation with o2. Science, 1770065233:1544-144 2020.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data, science, 324(5923):81--88 2009.
Yao Shi, Paloma I. Prieto, Tara Zepel, Shad Grunert, and Jason E Hein. Automated experimentation powers data science in chemistry. Accounts of Chemical Research, 54(3):546-555 2021.
Parshin Shojace, Kazem Meidani. Amir Barati Farimani, and Chandan Reddy. Transformer based planning for symbolic regression. Advances in Neural Information Processing Systems, 36, 2023.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of without human knowledge. nature, 550(7676):344-599 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1 1140-1144 1144, 2018.
12

---

Published as a conference paper at ICLR 2025
Tony Tohme, Dehong Liu, and Kamal Joucef-Toumi. Gsr: A generalized symbolic regression approach. arXiv preprint arXiv:2205. 15566, 2022.
Leonardo Trujillo, Luis Muñoz, Edgar Galván-López, and Sara Silva. neat genetic programming: Controlling bloat naturally. Information Sciences, 333:21-43, 2016.
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: physics-inspires method for symbolic regression. Science Advances, 6(16):eaay6311, 2020.
Silviu-Marian Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. Advances in Neural Information Processing Systems, 33:4860 2020.
Lucy van Dijk, Ruchuta Ardkhean, Mireia Sidera, Sedef Karabiyikoglu, Özlem Sari, Timothy DW Claridge, Guy C Lloyd-Jones, Robert S Paton, and Stephen P Fletcher. Mechanistic investigation of rh (i)-catalysed asymmetric suzuki-mij muzuki-miyuura coupling with racemic allyl halides. Nature Catalysis, 4(4):284-292, 2021.
Martin Vastl, Jonáš Kulhánek, Jifí Kubalík, Erik Derner, and Robert Babuška. Symformer: End-to-end symbolic regression using transformer-based architecture. IEFE Access, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Marco Virgolin, Tanja Alderliesten, Cees Witteveen, and Peter AN Bosman. Improving model-based genetic programming for symbolic regression of small expressions. Evolutionary computation, 29 (2):211-233 2021.
Juliane Weilbach, Sebastian Gerwinn, Christian Weilbach, and Melih Kandemir. Inferring the structure of ordinary differential equations. arXiv preprint arXiv22107.07345, 2021.
Xin Yang, Jun Zou, Yifei Wang, Ying Xue, and Shengyong Yang. Role of water the reaction mechanism and endo/exo selectivity of 3-dipolar cycloadditions elucidated by quantum chemistry and machine learning. Chemistry-A European Journal, 25(35):8289-8803, 2019.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A KINETIC MECHANISM OF TWENTY CATALYTIC ORGANIC REACTIONS
Mechanism diagram describes the species transformation relations under a specific chemical reaction. Taking Figure as an example, the reaction is S cat, P and includes several elementary reactions represented by arrows with different color such as + cat ++ cat.S. The elementary reaction is reversible, with both the outer ring and inner ring representing this elementary reaction. The outer ring represents the forward process, which is the primary and represents the cat combining with the S. The inner ring represents the reverse process, which is secondary and represents the decomposition of catS. The mass-action law is a fundamental concept chemical kinetics, describing the rate of elementary reaction is proportional to the product of the concentrations of is decided by two parts: 1) combination of S and col;
involved species. For example, equation at 2) decompositon of cat.S; which shown by red arrow and purple arrow, respectively. Therefore, ds = kー [cat - k [S] [cat]. Each type of reaction mechanism defined by the reaction diagram. The ordinary differential equations (ODEs) correspond each diagram, where the kinetic constants control the rate of change of each species in the reaction.
ds
di
Figure iilustratts the core mechanism, bicatalytic mechanisms, and catalyst activation mechanisms. The core mechanism is the basis of catalytic organic reaction. It demonstrates the process that the substrate combines with the catalyst form an intermediate, which subsequently decomposes into the product. Bicatalytic reactions include additional steps of the combination of two catalyst molecules to facilitate the reaction. In catalyst activation reactions, it is necessary to activate the catalyst. Figure and Figure Sngureedenmee demonstrate catalyst deactivation mechanisms. In contrast catalyst activation, catalyst deactivation reactions inevitably lead to the generation of deactivated catalysts.
14

---

Published as a conference paper at ICLR 2025
caf:
cat 1V cat
cat
cat
Bicataly
Bleatalytie
cats
cats
cat.S
A.jjoss]- AjS]ead
di -eo.ges)---s))oon (hipjjonn)
dEa Aojont) Jeantp
doad
Ouff- Kulfi 2kujoaat
(ha Aglforgl) Ayooof (effi Aigh (aallo 4towr
Alos,)
41
Roljard)
hyjonts!
deaat
2kyleat)) 28. keat"
cat
cat
cat
S
Biratalyti
Ricataly ft
catP
cats
Activation
cats
cat
431
cat
joar (-koerr
RyAA) R.Jead)
42)
koker joaee (1P)
duat
E.jaar bajautt (RB)) (..[[[(or]
dial]
deat3]
oulil AalP)jond) (kuy AgjA))aao)
4
(ki8l coot 28 (ean
41
Rejontt (ne)) stll/jttt N.jtnt//oorr
4imet
(Nil .erppera Rajler-st
cat
cat
S
IV
catS
cat
Antivation
Acth
catS:
<15
4,3])on)
Ayentts) Ralons50
d3M Ajeat Augest-Pn
(h... Kallowddj A. (2)))ao))))
dters) sykand k...4ioonn- (his Agffear (R/5)) k.dP))eux)]
Aan5
duan (k18) Aug/tponts]] (Baa Agleat-5)
Figure 7: Core mechanism, bicatalytic mechanism and catalyst activation diagrams and corresponding ODEs (MI-M8)
15

---

Published as a conference paper at ICLR 2025
inactive cati
inactive
inactive cat
inhibitor
cat
cat
cat
S
Deaiclsation
Deactivatonn
catS
catS
MD A.jkes) kgjl(or))
413
(ont) kugbd(
tojond) k.deott7))
kyjeat (ejat))
bejoett k.oont)
"ua kolkess) (9)30) Ru221 2-11 JoMbiirr2ean
Rojoodd- (000) .Pi ((fot)
KugPteent) Raljont))
Allecttin wer)) (5goot)
d
qleaemeeet)) A (uahhnnner
inactive catP
inactive cat:
cat
S
cat
cat
Deactr
anartiaatio
cats
catS
inactive catS
dF j.jjad) -M55)mm
A.. ky[[]eed)
Aejentt) the (aaltt
471
doad
d(oa)
41
(R.5 AxSPDjoec) (Ro Rallont)
<jrer)]
ontil A.btr
dinanii cart) R.g
cat
cat
cats
cats
inhibitor
inactive catSI
inactive catS,
415)
4.5
Aajontf] kyF Aig
xtret]
(K.. T...---llllttt Rullndd (hIF) KalP (ewr]
Au, Sajaan) (k,pi+ A.
strai)
31
Figure 8: Catalyst deactivation diagrams and corresponding ODEs (M9-M16)
16

---

Published as a conference paper at ICLR 2025
cat
cat
Dractivation
cats
cats
cats
inactive catSP
inactive cat:S:
455
Aycas) R.jerr (aar (P)-ko
41P)
27
Kojarr jergP)
dyer]
4[aa]]
dicuts] Ro1f)po
deart
(ha NalPieett - dies Ry eagi
inactive cat:S
inactive
cats 1
cat
cat
Deactisation
Desctirtion
cat
inactive cat.S
inactive catS
2I"
K.jeett]]
Rglonzz Reg jeciP)
Rojoess R.z [ott
djaat dt
kallows) RuPP1 kaloss5)kos)
deal
4t
Rallass]
at
dastter ont (nntt)yve))
dllnacthre s/we]]
d1
Figure 9: Catalytic deactivation diagrams and corresponding ODEs (M17-M20)
B GENERATION ORDER SEARCH WITH MCTS
Algorithm 1: Generation Order Search
Input:
Kinetic data X, conditonal model Ps. hyper-parameters a, B, number of iterations m,
constant
Output: ODEs y
# Initialize the tree and execute the first expansion operation.
initialize the tree ir with the root r
initialize ((r) 0, V(r)= 0:
nodes r.cxpand (P,, X);
tr. tr.uuateenoodss r):
Execute MCTS. for 4- 1 to n
tr MCTS(tr, r, X, ae 3.
end
# Generate ODEs from the Monte Carlo tree. 10 y (arg max Net))-getEqqsi
return y
The execution of generation order search is shown in Algorithm I Given the kinetic data X and conditional model Py. the first step is to initialize the tree and execute the first expand operation. Afterwards, MCTS executed. Algorithm shows how adopt MCTS to search the best generation
17

---

Published as a conference paper at ICLR 2025
Algorithm 2: MCTS
Input: Monte Carlo tree tr, activated node noct. Kinetic data X, conditonal model Po; hyper-parametent a, 3, constant c
Output: Updated tree ir
if Oact = tr. gottott() then
nact = tr.select tr.select(Mact, c):
tr (CTSS(rr Mact X, Pa. a, B,
else
(r. BergeeChhidren(aat (nact)
then
if ((Ract) = 0 then
score = tr tr.simulate(Ract Pg, a, tr.backpropagate (nacts score) fr.updato(N,,
10
else
nodes Bact expand (Po, X)
12
saupdate(nodess nact))
Ract = tr tr.select(Ruct c);
14
score = tr.simulate(Pacct X, Po, 0. (,
tr. br.backpropagate(nn.. (Pacc, score)
16
tr.update(N, V)
17
end else
18
Ract tr tr.select e); tr = MCTS(tr, Poct, X, Ps. a, 3,c)
20
21
end
end
23
return fr
order, which is a recursive algorithm. When the activated node is not the root, it is necessary to determine whether it is leaf node. If the activated node is not leaf node, continue select the next node and perform recursion; otherwise, continue to judge whether the current node has been traversed. If the answer is "no", execute simulate operation and then backpropagate to the root node; otherwise, execute expand operation and then select the next node. After n iterations, select the sequence with the largest N as the final prediction.
Specifically, our MCTS algorithm is at the equation level rather than the token level. Taking ODEs Figure as an example, there are four eqaution components, denoted as dz0, dæl, dr3 and dz4, respecitvely. During initialization, we first allow the model to independently generate these four equation components as initial nodes. During selection, we use P-UCB (equation 6) to select the specific eqaution component (c.g. dr1 node). Because this node has not been traversed yet, will proceed directly with the simulation. During simulation, we use the dæl equation as the condition to generate the next equation (e.g. dz0). Then combine dx1 and dæ0 condition to generate the next, and so on. Simulation will end until we get the full ODEs. After that, by solving ODEs, can obtain numerical solutions for each reaction species. These numerical solutions can then be compared with the input data (e.g. temporal concentration profiles) to calculate the R2 score. During backpropagation, we compute the reward according R2 score and then record as the node value. When the dæl node has been traversed, we will do expansion. During expansion, we use the drl equation as condition to generate possible next equations (e.g. dr0, da.2 or dx3 ), respectively. The above processes are executed iterately. This is an easy example of our MCTS framework.
C DATASET CONSTRUCTION
Considering the requirement for data authenticity and large data volume, we construct a simulated dataset based the methods described in Nature work (Burés & Larrosa 2023). First, based on
the categorized mechanism types (20 in total), we identify the corresponding ODEs. To construct valid ODEs, We randomly pick all kinetic constants in the range of 105-10-5 A.U. and rounded
18

---

Published as a conference paper at ICLR 2025
to three significant figures. The chosen kinetic constants must fulfill the conditions defined for their corresponding mechanism and allow for reversible and irreversible reactions with maximum yields between 20% and 90% to match actual experimental records. The initial concentration of reactant is kept as and the product is O. The catalyst's initial state is sampled between 0.01 and 0.99. Then, reaction concentration profiles as input data. It is worth noting that in order to construct a computable ODEs, we utilize Networkx to construct tree structure diagram of the formula, which is used for computation with scipy. The complete symbolic mathematical expressions of ODEs is transformed into prefix notation sequence and are kept as training label. We have eliminated all generated equations that are unsolvable and equations where the rate of change in intermediate products is less than 0.2. Eventually, for each type of catalytic reaction, we generate 5,000 reaction samples for training and 500 reaction samples for testing, which results in 100,000 training data and 10,000 test data. Based on our data scale, we introduce the Gaussian noise to each training sample with a standard deviation of Le-4 to improve model's robustness.
solve_ive from the scipy Python package based
LSODA solver to obtain
For different traning strategies, we choose different data format. Specifically, for universal transformer, we keep time series {tt, XE))E(( 2..... input encoder and the model is taught predict the complete ODEs of a reaction sample. For independent transformer, we split ODEs into several independent equations and append an index prompt (e-g. dz0 for the first ODE of Substrate) after time series as encoder input. The independent ODE corresponding to the index prompt is utilized to train the model. For conditional transformer. we keep one of ODEs from each reaction sample as training label and randomly choose several left ODEs as condition. Combining corresponding index prompts, the model trained predict one ODE according other ODEs condition and learn the implicit constraints between ODEs.
Why Shuffling order:
Firstly, KinFormer aims to capture physically- -ootivated correlations estab- lished by mass-action law, instead of searching for the optimal, deterministic, physically thyssicall-mmaninggin generation order. Secondly, Shuffling the order helpful for the model to grasp poysically-motivater correlations instead of spurious correlations. Actually, the attention mechanism of Transformer Decoder can capture these inner correlations implicitly, but tends to be negatively affected by spurious correlations. For example, in Figure 1, the k1 [S][c term appears simultaneously in the equations of d[S] and d/cat]. indicating the physically- maaallyy-eaingfuul correlation from the mass-action law; The fixed generation order of (d[S), d P], d[cat), d|catt]) shows the spurious correlation, which is not necessary. If the order is not shuffled, the Transformer Decoder is likely to capture such a fixed generation sequence rather than physically-motivate correlations, thereby making it difficult to generalize to unseen reaction types.
D TRAINING DETAILS
Hyper-paramen parameters and training settings of three strategies are shown in Table D It can be seen that almost all settings are shared by three strategies, except Equations/Epoch and Batch size. Equations/Epoch means the number of the equation sampled in each epoch. For the independent strategy and KinFormer (and the conditional strategy), Equations/Epoch = Batch size x Steps Per Epoch holds because these models generate only one ODE for each step (sampling 4 equations for each set of ODEs). However, the universal strategy generates the whole set of ODEs so the total number of equation is different. Considering Batch size, due to GPU memory constraints, KinFormer adopts a smaller batch size. In principle, the total numbers of equations for each epoch are similar, although the universal strategy introduces more equations.
E DEFINITION OF EVALUATION METRICS
We define r2mm to evaluate the overall performance of one test sample, that is, to compare the generated curves of all species changes as a whole with the original profiles:
22)²
r2m
k=
(9)
(x) zz
1=
19

---

Published as a conference paper at ICLR 2025
Table 2: Training settings.
Settings
Universal
Independent
KinFormer
Embedding dim
64
64
64
Encoder dim
256
256
256
Decoder dim
256
256
256
Activation
silu
silu
silu
Loss function
CE
CE
CE
Dropout
0
0
0
Optimizer
Adam
Adam
Adam
Schedule
Cosine
Cosine
Cosine
Learning rate
2e
2e
2e -4
Warmup
10000
10000
10000
Clip
1.0
1.0
1.0
Weight decay
Dropout
Epochs
100
100
100
Equations/Epocd
128750
100000
100000
Batch size
50
200
100
1.0
0.5
0.5
Device
NVIDIA 3090 24G
NVIDIA 3090 24G
NVIDIA 3090 24G
Due to the minimal changes certain species during the reaction (such intermediate), the r2,, calculation might overlook the significance of these species. Therefore, define r2M to assess the individual fitting performance, that is, to calculate the /22-score for each generated species profile independently, and then take the average:
25)2
T2M D
(10)
(at
The reported R2-score is calculated by averaging all test samples:
Nisa
Necst
R2m
2
R2M
T2)
(11)
Neest
Neest
j=1
We consider test sample with R2-score greater than the threshold of 0.9 to be correct due to great fitting performance:
Nenat
Niest
Accm
I [r2/ > 0.9];
ACcM
12 Ms > 0.9
(12)
Niest
Ntest
j=1
j=l
F EXTRA RESULTS
Table 3 shows the results of KinFormer all twenty types of mechanisms and Figure 9 shows their distributions of R2-scores. KinFormer perform well on M1, M4, and M12.
It can be observed that in most mechanisms, the KinFormer performs quite well. The r2, consistently exceeds 0.6, and even surpasses 0.8 in some mechanisms. However, is noteworthy that the performance of the mechanisms in M3 and M8 is not satisfactory. We have analyzed the following
20

---

Published as a conference paper at ICLR 2025
Mociem (w
moliyye Serkee (4F)
Re (01)
Non day
Hiu Mecoidyye
Crabae (411
Carir 2015
Ductiartii 0415
Coldyo (
Deree (977)
der Newr pure
Figure 10: KinFormer distribution diagrams of R2-score for all twenty mechanism types under in domain conditions. Red curve represents r2m and Blue curve represents r2M- The vertical line represents mean value of R2-score.
Table 3: Results of KinFormer twenty types of mechanisms.
Mechanism
Accm
ACCM
AcCform
R2m
R2M
M1
82.8
76.8
88.8
0.855
0.841
M2
68.0
17.0
52.2
0.805
0.679
M3
49.2
0.012
0.088
M4
82.0
76.4
91.0
0.870
0.856
M5
61.8
22.8
77.6
0.779
0.672
M6
67.6
39.6
88.2
0,799
0,734
M7
53.0
17.2
68.4
0.648
0,553
M8
0.2
0
81.0
0.028
0.337
M9
66.6
36.0
74.0
0.812
0.745
MI0
71.4
0
31.0
0.786
0.512
MII
63.6
17.6
30.6
0.785
0.663
M12
85.2
39.6
58.0
0.887
0.791
M13
61.6
24.2
63.8
0.739
0.678
M14
59.0
30.6
44.8
0.733
0.665
M15
70.0
0
56.4
0.783
0.507
M16
65.2
18.8
60.0
0.768
0.643
M17
64.8
42.4
65.0
0.771
0.702
MIS
69.6
45.6
53.0
0.801
0,742
M19
62.6
39.4
70.4
0.726
0.688
M20
39.6
7.2
76.6
0.573
0.481
reasons. the M3 mechanism, the formula includes a squared term, which very uncommon in other types. This contributes to the difficulty in learning. In the M8 mechanism, the dimension of ODEs increased to 6, which is uncommon in other types. This led to unsuccessful learning. In addition, and B hyper-parameters only for MCTS.
21

---

Published as a conference paper at ICLR 2025
Table R2 and RMSE quantile distribution. We keep the original R2 value, instead of replacing the negative value of R2 with zero in the paper. KinFormer outperforms than other baselines under Iwo out-of-domain scenarios. Lower means lower quartile and Upper means upper quartile.
R2
RMSE
Methods
Lower
Median
Upper
Lower
Median
Upper
Universal 0.879
0.983
0.996
0.004
0.008
0.018
Independent -72.633
22.4466
4.596
0.137
0.247
0.412
ID
Conditional 0.402
0.915
0.986
0.007
0.018
0.041
KinFormer 0.276
0.951
0.991
0.005
0.011
0.022
Universal -16.977
-1.271
0.803
0.028
0.095
0.239
0OD
Independent -3942.36
33,4099
-8.605
0.193
0.304
2.431
(Intra)
Conditional 0.343
0.881
0.973
0.011
0.023
0.045
KinFormer 0.611
0.934
0.984
0.007
0.013
0.025
Univeral -5.036
-0.112
0.806
0.026
0.067
0.157
OOD
Independent -7584.96
-33,462
10.596
0.222
0.322
3.255
(Inter)
Conditional -11.744
0.452
0.942
0.016
0.044
0.188
KinFormer -0.907
0.689
0.977
0.007
0.012
0.024
We also provide the extra results of R2 and RMSE shown in Table We report the lower quartile, median and upper quartile metrics. We keep the original R2 value, instead of replacing the negative value of R2 with zero. The results are consistent with our conclusion in the paper.
The main packages of Python we used include: (1) Scipy 1.10.1 for ODE solving; (2) Networkx 2.8.2 for Monte Carlo tree construction; (3) Pytorch 2.0.1 for Transformer construction and training; (4) Scikit-learn 1.0.2 for model wrapping.
G TOP 3 ORDERS FROM MCTS
Table 5 shows the top 3 orders from MCTS. It can be seen that bicatalytic reactions and catalyst activation do not exhibit specific order. However, almost all reactions with catalyst deactivation share the same order: [O, P, cat., S, cat), where O represents intermediates distinct from catS.
22

---

Published as a conference paper at ICLR 2025
Table 5: Top 3 generation orders statistics from MCTS. The number after generation order is corresponding frequency count.
Mechanism
Top
Top
Top
M1
P. catS, S, cat]:57
[S, cat, P. ca4
[P,S, cat, cat
M2
[0 P. catS, S, cat):21
leat, O, catS):21
[catS, P,S, cat)
M3
[P,S, O, cal.S, cal]:19 :19
S, cal.S
JP,O, cul, cul.S):188
M4
[P, O, S, cat, cat.S):30
S,, 0
[s, P, cat, cat.S :223
M5
[eat.S, S, P, cat :37
[O, P. catS, caf):29 :29
[eatS, S, cot, P. 0):27
M6
[eat, O, S, P, ca2.5):21
[eat, O, P,
catS
[eat, S, catS, P, 0]:19
M7
[O, P. cat, S, cat.S)
caatS, cat, S, P, Pj:20
[S, cat, catS, P, P,O
M8
(cat, P, S, cat.,, 0]:29
cat, O S, cat.S, 0):25
[P, cat, O, S, cat.5)::5
M9
[0, P. catS, S, cat): :29
[0,S, catS, P, :27
[eat, O, S, P, cat
MIO
[O,O, P. catS, S, cat)::1
[0 0, P.S, cat, cat5]:10
[0, O P. catS, S, cat):9
MI1
[0, P, catS, S, cat]:31
[0, P, cat, S, :a
[S, O, catS, P, 11
M12
O., P, cat.S, S, c,
OO,S, cat.S, P, cat) :38
O, P. cat, cat.5) 26
M13
[0, P, catS, cat]:30
O S, catS, Pa :24
[0, P. cat, cat.S
M14
[O,S, S, Pa caf))
[0, P. cat, S, catSS):22
[0, P. catS, S, cat :22
M15
[0 0, P, catS, cat]:16
0, P,ctS, S, cat]:15
10 O, S, catS, P, cat):
M16
[0, P, cat.S, S, cat]::31
O, S, cat.S, P, cat] cat)::25
P. O, cat, S, cat.5) :222
M17
[0, P, cat.S cat)
[0, P, cat, catSS)2:28
[P, O, cat, S, cat.S) :24
M18
[0, P. catS, S, ca
[O, catS, S, cat) :20
[0, P. cat, S, cat S1:18
M19
[0, catS, :32
[0, P, S, cat, cuts):30 :00
10, P. cat, cat.S) :):28
M20
O, 0, S, catS, P, cat):9
0,0, P, eatS, S, cat]:9
0,0,, S, P, cat, cat
23


Reviews:
[{"review_id": "AjcS4sXhxq", "reviewer": "Reviewer_5fkZ", "summary": "This work presents a novel approach to solving dynamical symbolic regression through deep learning, with a specific emphasis on the dynamics of chemical reactions and kinetic equations. The central concept involves parameterizing the underlying ordinary differential equations (ODEs) using a conditional transformer that respects physical constraints aligned with the reaction dynamics. While this modeling technique performs effectively within the training distribution, it struggles with out-of-distribution scenarios.\nTo address this limitation, the authors introduce a conditional Monte Carlo Tree Search (MCTS) algorithm that seeks compatible solutions while adhering to the specified constraints. The proposed model, KinFormer, is evaluated across 20 symbolic regression benchmarks in organic chemistry, demonstrating competitive performance within the training distribution and significantly outperforming baseline models in out-of-distribution cases.", "questions": "- How did you collect the data used in your research? Are all the reactions you analyze simulated numerically? Why did you focus only on organic reactions?\n\n- Why did you choose to employ \"prefix notation\" for symbolic regression in your work?\n\n- Is there something unique about the kinetic equation or reaction dynamics that enhances the effectiveness of your method?\n\n- On average, how many simulation steps are necessary to derive the equations for the 20 organic reactions? Specifically, what is the average depth of the MCTS you observe? Is it essential to utilize all 100 steps in your simulations?\n\n- Have you considered implementing a hybrid approach, such as integrating a NeuralODE or a numerical simulator during the MCTS?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and enhanced by effective visualizations, aiding readers who may be familiar with ordinary differential equations (ODEs) and transformers, but not necessarily with reaction dynamics, in understanding the overall problem.\n\nThe methodology is interesting and clearly articulated. The central concept of parameterizing the dynamics using a conditional transformer, while ensuring that the model adheres to physical constraints, represents a promising research direction for integrating deep learning models into scientific and engineering contexts. Notably, although it has only been tested on relatively small benchmarks, the KinFormer framework seems to be scalable.\n\nThe approach of utilizing Monte Carlo Tree Search (MCTS) for out-of-distribution symbolic ODE generalization is both innovative and significant. Search strategies have proven essential in various fields, including discrete optimization, game theory, and reasoning within large language models. Leveraging search techniques is logical, provided that a consistent value/reward can be assigned to each node in the tree. While the underlying idea is straightforward, achieving effective implementation in such a technical domain is non trivial.\nAlthough the focus is on chemistry, the potential for generalizing the use of transformers and MCTS extends to a wide range of scientific and engineering problems governed by ODEs.", "weaknesses": "### Node Scoring\nThe reasoning behind the scoring mechanism presented in subsection 4.2 lacks clarity. It would be beneficial to provide some intuitive insights into why the heuristic employed for Go is also effective in the context of symbolic regression for chemistry. This could enhance the reader's understanding and strengthen the argument.\n\n### Limited Evaluation\nThe KinFormer has only been tested on 20 organic reactions that share similar properties. It would be valuable to conduct additional experiments across a broader range of fields, including inorganic chemistry. This would provide a more comprehensive assessment of the method's applicability and robustness.\n\n### Cost\nIn Table 1, the complete KinFormer method demonstrates strong performance out-of-distribution (OOD); however, it is also 10 to 20 times more resource-intensive compared to the baseline methods. A deeper analysis of this trade-off between performance and computational cost would be insightful for readers considering its practical implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OYzwoZAUEI", "reviewer": "Reviewer_vrt5", "summary": "This paper develops dynamic SR for catalytic organic reaction kinetics from experimental time series, using a transformer approach.", "questions": "Can this technique be useful in other kinds ODEs and/or PDEs?\n\nConsidering that this is *symbolic* regression, what does an RMSE error mean? For example, if it should be a +, but is instead a *, what is the RMSE?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The approach is reasonable. The paper seems well-written and well-presented. The appendix is detailed.", "weaknesses": "Overall, I think the paper has not sufficiently presented a convincing case of innovation and novelty. This may partly be due to the fact that I found the paper hard to understand. For example, in Figure 1a, the inner arrows is not explained in the caption, and I could not find it in the text. In Figure 1b, the constants with negative subscripts such as $k_{-1}$ are not explained.\n\nThe paper seems to define generalizability in the intro as being not domain-specific, but as far as I can tell, they then go on to only show results for a very narrow domain, but only for inter/intra class.\n\nMy understanding is that the physical constraints R are the mass-action law. I think a sentence or two more on how they maintain that would be helpful. And doesn't that then also preclude generalizability? How would they support different physical constraints (if they claim generalizability)?\n\nDoes the numerical tokenization scheme represent, say, the difference between 1.23 and 1.24, as being less than the difference between 4.21 and 8.21? How does it compare to things like XVal [1]?\n\nI think my main issue with the paper is the significance of it. I would imagine that collecting the experimental data is often the largest effort. Once collected, is it difficult for a human to look at the data and fit the kinetics?\n\nIf this technique could be generalized to general ODEs, then I think it would be of significantly greater interest.\n\n[1] Golkar, Siavash, et al. \"xval: A continuous number encoding for large language models.\" arXiv preprint arXiv:2310.02989 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8od03Uv8Lc", "reviewer": "Reviewer_tWZW", "summary": "# Modelling reaction kinetics with (language model based) dynamic symbolic regression and Monte-Carlo tree search\n\nThis paper presents a method to model reaction kinetics for a reaction between a given number of reaction partners (substrates, catalysts, products). In particular, the proposed method infers a symbolic form of the reaction mechanisms (as system of ODEs, in accordance with the mass action law), from given measurements of the concentration of each reaction partner, measured at several discrete timepoints along.\n\nCompared to previous approaches, the current method does not generate all equations governing the simultaneously (referred to as *universal strategy* by the authors) or sequentially and independently (referred to as *independent strategy*). Instead, the authors propose to generate the order of the equations in the system of ODEs sequentially, but whilst conditioning subsequent equations on the previously generated equations within one prediction (ie. one system of ODEs). The prediction order is determined via Monte Carlo tree search. \n\nThe authors test their ideas on a set of 20 reaction mechanisms, from which synthetic temporal concentration profiles are simulated and then used as training and inference input data. \n\nThe contributions are:\n1. The proposed strategy (in particular the conditioning of subsequent ODEs on previous ODEs within one reaction mechanism - ie ODE system and the MCTS post-processing to search for the optimal generation order).\n2. The performance analysis and comparison of in-distribution, out of distribution (but within a given reaction class) and out of distribution for a different reaction class generalization. Here the authors compare to two prior methods and find significant improvements in generalization.", "questions": "1. I did not quite understand to what extend MCTS is a post-processing procedure? Is it not also used during training in order to come up with the one desired prediction order in the first place, since that order is needed for the conditioning you are doing of one equation by the next?\n\n2. On page 6, by dimension, presumably you mean the number of reaction partners? If so, it might help readers from a loosely adjacent field (like myself) to state this explicitly.\n\n3. For your comparison against beam search, would you be able to go into a bit more detail on what your n=20 is here?\n\n4. Figure 6: I'm afraid I cannot follow what this is meant to represent. Is the left hand side concentration profile the input you got from the timepoints of the exact simulation of your synthetic system (on the RHS)? And are the dotted lines the inferred symbolic estimates? \n\n\nVery minor suggestion: There are various articles missing or at wrong places. I would kindly suggest the authors to run the text through a grammar checker to enhance the reading flow. Other than that it is very readable!", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Disclaimer: I am not directly researching in this field and therefore not familiar with the latest literature.\n\nAs someone familiar with the overall research area, but unfamiliar with this task and formulation -- and the task specific literature -- I found this paper a refreshing read with interesting ideas and discussions.\n\nThe problem description and statement were very clear and well readable, and the method was also well explained. \n\nI found the study with the in-distribution, out of distribution (but within reaction class) and across reaction classes well designed and interesting. \n\nBased on what the authors describe, the proposed conditioning scheme & combination with MCTS appears novel to me.", "weaknesses": "While the description of the problem and the method were great, I found the discussion of the results comparatively harder to follow. This may be due to me coming from an adjacent field, but I would appreciate if the authors could refine this discussion further (c.f. my questions below for guidance).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "btbPFPLsVZ", "reviewer": "Reviewer_c4Q1", "summary": "The authors address the kinetic equation prediction (KEP) problem for discovering symbolic equations from temporal concentration profiles of species while satisfying mass-action law constraint. They propose to combine a Transformer encoder/decoder, as used by prior work, with a novel MCTS search strategy over the generation order of the ODE equations following a conditional training strategy. They show the approach is able to generalize from the training data, out-perform two classical symbolic regression methods, and generalize to new reaction types. They provide in-depth analysis of generalization across various reaction mechanisms.", "questions": "- How much does performance drop if you take random/bad generation orders for the conditional strategy?\n- How can the robustness to noise be improved?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an essential task for understanding chemical reactions. \n- The methodology and analysis sections show a deep understanding of the problem.\n- The generalization favors traditional SR methods", "weaknesses": "- There needs to be more baselines than traditional SR approaches and the three training strategies (which is more of an ablation)\n- The topic does not seem very relevant for ICLR\n- The approach does not seem scalable or robust to noise", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)