ChemAlgebra : Algebraic Reasoning on Chemical Reactions
Authors: Andrea Valenti
Davide Bacciu
Antonio Vergari
Venue: Submitted to ICLR 2023 (2023)
Keywords: algebraic reasoning
datasets
benchmarks
learning-reasoning integration
chemical reactions

Abstract: While showing impressive performance on various kinds of learning tasks, it is yet unclear whether deep learning models have the ability to robustly tackle reasoning tasks. Measuring the robustness of reasoning in machine learning models is challenging as one needs to provide a task that cannot be easily shortcut by exploiting spurious statistical correlations in the data, while operating on complex objects and constraints. To address this issue, we propose ChemAlgebra, a benchmark for measuring the reasoning capabilities of deep learning models through the prediction of stoichiometrically-balanced chemical reactions. ChemAlgebra requires manipulating sets of complex discrete objects – molecules represented as formulas or graphs – under algebraic constraints such as the mass preservation principle. We believe that ChemAlgebra can serve as a useful test bed for the next generation of machine reasoning models and as a promoter of their development.

Under review as a conference paper at ICLR 2023
CHEMALGEBRA: ALGEBRAIC REASONING ON CHEMICAL REACTIONS
Anonymous authors Paper under double-bling blind review
ABSTRACT
While showing impressive performance on various kinds of learning tasks, is yet unclear whether deep learning models have the ability to robustly tackle reasoning tasks. Measuring the robustness of reasoning in machine learning models is chal- lenging needs to provide a task that cannot be easily shorteut by exploiting spurious statistical correlations in the data, while operating on complex objects and constraints. To address this issue, we propose CHEM measuring the reasoning capabilities of deep learning models through the predic- tion of boichiometriically-balaaa manipulating sets of complex discrete objects molecules represented as formulas or graphs under algebraic constraints such as the mass preservation principle. We believe that CHEMALGEBRA can serve as a useful test bed for the next generation of machine reasoning models and as promoter of their development.
benchmark for
chemical reactions. CIEMALLEBEE requires
INTRODUCTION
Deep learning models, and Transformer architectures in particular, currently achieve the state-of-the- art for a number of application domains such as natural language and audio processing, computer Andonie,
vision, and computational chemistry (Lin et al., 2021; Khan al., 2021; Braşoveanu 2020). Given enough data and enough parameters to fit, these models are able to learn intricate correlations (Brown et al., 2020). This impressive performance on machine learning tasks suggests that they could be suitable candidates for machine reasoning tasks (Helwe et al., 2021).
Reasoning is the ability to manipulate a knowledge representation into form that is more suitable to solve a new problem (Bottou, 2014; Garcez. et al., 2019). In particular, algebraic reasoning includes a set of reasoning manipulations such as abstraction, arithmetic operations, and systematic composition over complex objects. Algebraic reasoning is related to the ability of learning system to perform systematic generalization (Marcus, 2003; Bahdanau et al., 2018; Sinha et al., 2019), i.e. to robustly make predictions beyond the data distribution it has been trained on. This is inherently more challenging than discovering correlations from data, as it requires the learning system to actually capture the true underlying mechanism for the specific task (Pearl, 2009; Marcus, 2018).
Lately, much attention has been put on training Transformers to learn how to reason (Helwe et al., 2021; Al-Negheimish et al., 2021; Storks et al., 2019; Gontier et al., 2020). This usually done by embedding an algebraic reasoning problem in a natural language formulation. Natural language, despite its flexibility, imprecise and prone shortcuts (Geirhos al., 2020). As result, it is often difficult to determine whether the models' performance on reasoning tasks is genuine or it is merely due to the exploitation spurious statistical correlations in the data. Several works in this direction suggest (Agrawal et al., 201 6; Jia & Liang, 2017; Helwe et al., 2021) the latter is probably the case.
In order to effectively assess the reasoning capabilities of Transformers, we need to accurately design tasks that i) operate on complex objects ii) require algebraic reasoning to be carried out and iii) cannot be shortcut by exploiting latent correlations in the data. We identify chemical reaction prediction as suitable candidate for these desiderata. First, chemical reactions can be naturally interpreted as transformations over bags of complex objects: reactant molecules are turned into product molecules by manipulating their graph structures while abiding certain constraints such as the law of mass conservation. Second, these transformations can be analysed as algebraic operations over (sub-) )graphs (e.g., by observing bonds forming and dissolving (Bradshaw et al., 2019)), and balancing them to

---

Under review as a conference paper at ICLR 2023
preserve mass conservation can be formalised as solving linear system of equations, will show in Section Third, the language of chemical molecules and reactions is much less ambiguous than natural language and by controlling the stoichiometrie coefficients, i.., the molecule multiplicities, at training and test time we can more precisely measure systematic generalization. Lastly, Transformers already excel at learning reaction predictions (Tetko et al., 2020; Irwin et al., 2022).' Therefore, we think this can be a solid test bed to measure the current gap between learning and reasoning capabilities of modern deep learning models.
The main contributions of this paper are the following:
We cast chemical reaction prediction as a reasoning task where the learner has not only to predict set of products but also correct stoichiometric coefficient variations (Section 2).
2. We evaluate the current state-of saae-oof-hhas of-the-art Transformers for chemical reaction predictions, showing that they fail to robustly generalise when reasoning simple variants of the chemical reaction dataset they have been trained on (Section 3).
3. We introduce CHEMALGEBRA as novel challenging benchmark for machine reasoning, in which we can more precisely measure the ability of deep learning models to algebraically reason over bags of graphs in-, cross- and aat-of-distribution settings (Section 4).
PREDICTING CHEMICAL REACTIONS AS ALGEBRAIC REASONING
To illustrate point, let us consider the Sabatier reaction: it yields methane (CH,) and water (H₃O) out of hydrogen (H₂) and carbon dioxide (CO₂), in the presence of nickel (Ni) as a catalyst. In chemical formulas:
ICO, + 4H2 Ni 1CH4 + 2H
(1)
where formulas encode complex graph structures where atoms are nodes and chemical bonds edges:
+
Ni
== ==
A reaction prediction learning task hence consists of outputting bag of graphs, the products (right hand side), given the bag of graphs consisting of reactants (left hand side) and reagents (i.e. the catalysts, over the reaction's arrow).
The multiplicities of the molecules, also called their stoichiometric coefficients, express the fractional proportions of reactants to yield certain proportion of products. For example, one needs 4:1 ratio of hydrogen molecules and carbon dioxide to produce a 1:2 ratio of methane and water. A reaction is (mass) balanced when its stoichiometric coeffients are well placed such that the sum of the number of atoms for every element across products shall be the same of that across reactants, i.c., it satisfies the principle mass conservation (Whitaker, 1975). Unbalanced reactions, the other hand, would be chemically implausible.
This constraint over atoms of the molecules underpins the true chemical mechanism behind reactions: bonds between atoms break and form under certain conditions but atoms do not change. In reasoning terms, this a symbol-mmnipplattin of graphs. A machine reasoning system that would have learned this true chemical mechanism, would be able to perfectly solve the chemical reaction prediction task for all balanced reactions and for all possible variations of stoichiometric coefficients.
process where bags of graphs are deconstructed into other bags
As humans, we can balance fairly complex chemical reactions quite easily.? For machines, this process can be formalised as finding a solution of potentially undetermined system of linear equations. For example, we can write the Sabatier reaction as:
11 CO2 + r2 Hg + r3s Ni = P1 CH2 + H2O Ps
An extended overview of the related works in chemical reaction prediction is given in Appendix A. We learn to do from very few examples, e.g. a handful reactions taken from chemistry textbooks in high school. Without following an explicit algorithm, we can usually perform balancing in an intuitive way, by leveraging our quick arithmetic skills to count the atoms for an element and match the numbers on both sides of the equation, iteratively changing the stoichiometric coefficients until all elements balanced.
2

---

Under review as a conference paper at ICLR 2023
where the variables T1, F2, r3, Pl, P2. Ps represent the stoichiometric coefficients of the molecules they refer to. Molecules of reagents act as a special kind of confounders: since they are not changed during the reaction, they must appear on both sides of the equation with the same coefficient (i.e., ra Pa)- Under this rewriting, it becomes even more evident how a full chemical reaction can be interpreted an algebraic equation where stoichiometric coefficients
the unknown variables.
Then, we can represent the molecule CO2 with the vector [1, 0, 2, 0]. indicating one atom of carbon, zero hydrogens, two oxygens, and zero nickles. Analogously, H2 can be encoded as [0, 2, 0, 0]. and so on. Therefore. Eq. (1) can be rewritten as the following linear system:
1 0
[
0
07
0
2
r1
4
2
P1
2 0
0
1
0
P2
(3)
0 1
0
0
1
P3
is straightforward to verify that the minimum norm solution of Eq. (3) r
[1,4.1].p
[1,
thus conforming to Eq. (1). We now devise set of reasoning tasks exploiting this perspective.
2.1 "TYPE 1" AND "TYPE 2" CHEMICAL REACTION REASONING TASKS
We propose cast chemical reaction prediction reasoning task, where the model has to predict both the graphs corresponding to the product molecules (i.e. the right hand side of the reaction) and the exact multiplicities of such molecules, given particular input consisting of reagents and reactants equipped with varying stoichiometric coefficients. This is in stark contrast with the vanilla reaction prediction learning setting: in models are trained on reactions without stoichiometric information, largely on unbalanced reactions (see Section 3.2); in our setting, stoichiometric coefficients are not only present but, at prediction time, can greatly differ from those the model has seen at training time. By doing so, we can better control and measure systematic generalization.
For example, given a reference reaction, we can multiply by a factor all the stoichiometric coefficients in it. For Sabatier reaction, we would obtain the following input-output pair for a factor of two:
2C0₂ + 8H2 + 2Ni (INPUT)
2CH, + 4H20 + 2Ni (OUTPUT).
(4)
In the rest of this paper, we will refer to this type of reasoning task Type 1 task. Alternatively, we can just add a certain number of molecules on the left hand side. Some these additional molecules might not take part in the reaction, since there might not be enough reactants to bond with. For example, if we add two CO2 and two Ni to Sabatier reaction we expect a model to predict them in the right hand side in addition to its usual outputs:
3 C02 + 4H2 Ni (INPUT) CH2 2H,0 + 3i 2CO2 (OUTPUT).
(5)
We refer to this Type 2 task. Type reactions harder reason with than Type 1, but should still be easy for a machine learning model that has learned the true underlying chemical mechanism.
There is final aspect consider: if the test coefficients are sampled from the same data distribution as the training coefficients, there is still a chance that the model would be able to predict them just by pattern matching. Conversely, learning the actual algebraic reasoning behind the stoichiometry of chemical reactions, e.g., by solving the linear system of Eq. (3), empowers the model to solve any stoichiometry problem, regardless of the actual values of the coefficients observed during training.
To control for these aspects, in addition to the usual in-distribution scenario where both training and test coefficients are sampled from the same set of integer numbers S,,, we consider also an out-of-distribution setting where the training and test reactions are instantiated with coefficients coming from the disjoint sets of integers Siz and Sout, respectively. Analogously, we can test a cross-dae wross-distribution scenario, where the training set is divided into two halves. The reactions of the first half are instantiated with coefficients selected from Sm, while the second half's coefficients are selected form Sout. The cross-distribution test set will contain the same reactions of the training set but with "swapped stoichiometry": that is, the first half of the test set will have coefficients from Sout- the second half from S.n- We argue that well-traincs reasoning model should have no problem to semantically disentangle the stoichiometric coefficients from the molecules, thus achieving the same performance in both the in-distribution, cross-distributiin, and nut-of-distribution scenarios.
3

---

Under review as a conference paper at ICLR 2023
Table 1: Statístics of the training, validation and test splits of the USPTO dataset before and after rebalancing. Percentages in parenthesis are w.r.t the original dataset.
TOTAL
BALANCED (%)
RE-BALANCED (%)
USPTO-BAL (%)
TRAINING SET
409035
18815 (4.59)
162220 (39.66)
181035 (44.26)
VALIDATION SET
30000
1292 (4.31)
11868 (39.56)
13160 (43.87)
TEST SET
40000
1809 (4.52)
15973 (39.93)
17782 (44.45)
CAN TRANSFORMERS PERFORM ALGEBRAIC REASONING?
In this section, question whether current deep learning models effectively solve the algebraic reasoning tasks induced by the stoichiometry of chemical reactions, as discussed in Section 2. In order to do so, first need to build suitable dataset of chemical reactions that can be processed by state-of-the-es models for reaction predictions. We then evaluate these models on some meaningful variations of the task and discuss their limitations.
3.1 HOw TO BUILD A BENCHMARK OF BALANCED REACTIONS
A natural candidate is the USPTO-MIT dataset (Lowe, 2012), large collection of chemical reactions extracted from US patents from 1976 to 2016 and represented in the SMILES format (Weininger, 1988). SMILES reactions text strings encoding linearization of the molecular graphs of each molecule participating in the reaction. We employ the version popularized by Jin et al. (2017), often referred to USPTO-MIT. This version contains subset of polished reactions, obtained by removing duplicate and incorrect ones. For simplicity. in the rest of this paper we will refer to "USPTO-MIT" simply "USPTO". This dataset composed of 409k training reactions, 30k validation reactions and 40k test reactions (see Table 1).
Unfortunately, reactions the USPTO dataset are mostly unbalanced: we found that less than 4.6% of the reactions in the dataset are mass balanced and readily usable for our tasks. For the remaining reactions, in many cases only the major product of the reaction was recorded, while disregarding minor byproducts, such as H2O or HCI, probably not deemed interesting from patent perspective. Examples of these reactions are illustrated Fig. While this does not impact patents, it can affect the generalization capability of the learned models,
will show next.
Fortunately, the missing byproducts can be sometimes deduced from the unbalanced reaction: if the set of missing atoms of one side of the reaction is enough to unambiguously reconstruct one (or more) valid molecules on the other side, we add such reconstructed molecules to that side of the reaction. The rightmost column in Table shows the number of reactions contained in the balanced subset of USPTO that we used as the basis for the augmentation procedures described in Section 4. Using this strategy, we were able re-balance about 44% of the original reactions for the training, validation and test sets. We denote this re-balanced version of USPTO as USPTO-BA -AL and use it next.
3.2 TYPE AND TYPE 2 VARIANTS OF USPTO
We build two variants of the original USPTO dataset, called USPTO-TI and USPTO-T2, as instances of the Type 1 and Type 2 reasoning tasks introduced in Section 2.1. For TI, multiply all coefficients by two, in practice duplicating the molecule representations in the SMILES string of a reaction (as the SMILES format does not allow to represent stoichiometric coefficients). For T2 we add randomly-selette molecule to the reactants. The detailed procedure is discussed in Appendix B.
These two variants should not, in theory, pose significant challenges for chemical reaction models that learned the true reaction mechanisms of USPTO, since despite shifting from the original distribution, they still use the same molecules and reaction mechanisms of USPTO.
In the case of US,,O-TI, the reactants contain all the required additional molecules to perform the reaction twice, so the output should correspond the same multiple of the original products. This even easier considering that the original reactions in USPTO generally involve a single molecule per
Note that SMILES strings represent hydrogen implicitly, when the resulting molecule ambiguous.
4

---

Under review as a conference paper at ICLR 2023
(H)
Figure I:
Examples of re-balanced reaction of the USPTO-BAL dataset. The inferred byproducts shown between brackets (the reagents have been omitted for simplicity).
Table 2: Results of state-of-the-art Transformers for reaction prediction on the USPTO dataset and the BAL, TI and T2 variations. We report the top-1 accuracy (ACC) for all models. We also report the at-least-one (ALO) accuracy for the T1 and T2 variants. All metrics are reported as percentages.
USPTO
USPTO-BIL -AL
USPTO-TI
USPTO-T2
STATE-OR COETE-OF-THE--TEE ART MODEL
ACC
ACC
ACC
ALO
ACC
ALO
MOLTRANS, (ScHWALLER ET AL., 2019)
90.4
1.39
0.04
56.66
0.03
28.32
CHEMFORMER (IRWIN BT AL., 2022)
92.8
10-2
0.23
34.41
10-2
2.61
G2S (DGCN) (Tu & COLEY, 2021)
90.3
1.37
10
83.02
10
40.79
G2S (DGAT) (TU & COLEY, 2021)
1.40
10
84.21
10-2
41.43
type. On the other hand, in the case of USPTO-T2, the added molecule is not sufficient to trigger multiple reactions, so it should just be copied in output by the model. As will show in the next section, in practice this is not the case.
3.3 CURRENT CHEMICAL MODELS ARE ACTUALLY ALCHEMICAL MODELS
We evaluate the systematic generalization of state-of-the-art models for reaction prediction trained on USPTO on the chemically-soune variation and the reasoning variants introduced in Section 3.1 and Section 3.2. We focus on Transformer-bias SMILES representations (Schwaller et al., 2019; Irwin et al., 2022) as well Transformers that employ graph neural networks (GNNs) to parse the molecules structures and be permutation invariant (Tu Coley, 2021). For additional details about these models, we refer the reader to Appendix B.
language models that operate on
For all models, we report the top-1 prediction accuracy (ACC) on the BAL, T1 and dataset variants as well as the original USPTO. On the T1 and T2 variants, a correct prediction would involve the model output both the right molecular graph and the right multiplicity each product molecule. This proved extremely challenging for all models. As such, also report the "at-least-one" accuracy (ALO) score which consider the prediction correct if at least one of the ground truth molecules is being predicted, despite its multiplicity being inaccurate.
The results in Table are striking: while all models are able to reach over 90% of top-1 accuracy over the standard unbalanced USPTO test set, the performance drops sharply below 2% on USPTO-TI and USPTO-T2 but also on USPTO-BAY BAL, despite this variant only adds small number byproduct molecules. This fall in performance is also evident for the ALO metric.
5

---

Under review as a conference paper at ICLR 2023
Reactants
Prediction
Ground Truth
Figure 2: Examples alchemical reactions as predicted by state-of-the-ard Transformer models on the original USPTO dataset. (see Table 2 and Appendix B) where atoms of chemical elements appear or disappear between reactants and predicted products regardless of the mass preservation principle. For example, in the first reaction, sulphur (S) is turned into oxygen (O), and in the third selenium (Se) into sulphur. In the second and fourth, nitrogen (N) and sulphur disappear as elements from the products, while in the last only two atoms of oxygen out of four remain.
The distribution shifts in the original training distribution introduced in our variants highlight that these state-of-the-art models are not learning the true chemical mechanism, but a brittle function approximation that leverages statistical correlations present in the data. This is somehow expected as these models have been trained on unbalanced reactions and their predicted products often do not satisfy the mass preservation constraint. In fact, predicted product molecules might contain not only an unbalanced number of atoms per element, but also introduce atoms of elements that were present in the reactants and reagents making other elements disappear from the products. We show some qualitative examples of these "alchemical" predictions reported on the original USPTO dataset," in Fig.
So, why are current state-of-the-art models not able to perform basic forms of reasoning over chemical data? We think there are several reasons: part of the problem might reside in the training data itself, not containing chemically plausible reactions. Datasets based on USPTO, in their current form, are not able pose a significant learning challenge that is also sound from chemical point view. Another part of the problem resides in the intrinsic limitations of deep learning architectures that are currently used for this kind of tasks. While efficient and flexible, these models are too keen on leveraging statistical patterns of the data, and have hard time to perform out-of-distribution predictions even on simple variations the original task, thus failing to achieve systematic generalisation.
One could argue that, if we added the Type I and Type reaction variations in novel training set, the models learned on would robust to them. This not a satisfying answer for models that should systematically generalize, as could always test their performance new batch of reaction
In nuclear reactions the appearance or disappearance of elements would be allowed, but this kind of reactions out of the scope USPTO and our work.

---

Under review as a conference paper at ICLR 2023
Table 3: Example of the CHEMALGGERRA input-output variations obtained from the same initial reference reaction and represented in our stoichiometticall-aagmeeee SMILES and raw chemical formulas (FORMULA) languages.
REACTANTS
PRODUCTS
ORIGINAL
CI HCI + C,H,,O + C2H
HCI + C2H220O
(3)C1, (3901.33) [ci-].
(3) [C1-1] (3)
SMILES
cC0cc0oc11, (3)ccccce)())
(3)00C0C (CO) cloce (C) colC
TI
[3JHC1.13JC1-. (3)C1-.
FORMULA
[3)668120. (3)c8=10
(3:C148220
(3)C1. (5) (c)-)
[3)C1. (5) [C1-].
SMILES
(5)Celcccc cC) cl. (3) cocccic11.
cl, (1)cccc0c11110000
T2
(2)Co (CO) (C) ccIC
(3)8C1. [1JC14H2200 (1:06122000:11122,0
(3) (0]08800.
FORMULA
variants involving slightly different stoichiometric coefficient manipulations, e.g., multiplying them by three instead of two. Designing suitable benchmark suite systematically train and evaluate models over several variations of these manipulations poses certain challenges in terms of input specification and controlling potential shortcutting. We discuss and overcome these challenges in the next section.
4 THE CHEMALGEBRA BENCHMARK
Building on the previous considerations, we now design the CHEMALGEBRA benchmark. Starting from the balanced variant of USPTO, described in Section 3 (USPTO-BAL), we devise eight tasks for assessing the reasoning capabilities of deep learning models. First of all, we need to find way to explicitly represent the stoichiometric coefficients directly (as duplicating SMILES sub-strings becomes impractical for long molecules and large coefficients numbers, which are needed for this end, we define a aatichiommtrrccllly-agmeeeee language, where each coefficient is encoded an integer surrounded by curly braces preceding each molecule in the reaction. For example, Sabatier reaction from Eq. (1) becomes
challenging benchmark).
SMILES string
(1)0-C-0. } [HH] {1) [Ni] > > (1)C.12)0) {1} [Ni]
in this new SMILES dialect. To better understand how reasoning task of predicting the correct reaction mechanisms affects the performance, we devise second, simpler, representation where molecules the reactions are encoded as raw chemical formulas. Sabatier reaction encoded this way is:
{1) C00. H2. NNi
(1 CH4. (2) (1 N1)
(7)
Note that in both cases the reagent Ni is copied explicitly on both sides of the reaction. These alternative notations in CHEMALGERRA pose different challenges for deep learning models. The SMILES representation is more verbose and lower-level, allowing for a complete reconstruction of the original molecular structure. However, SMILES strings can be ambiguous (Arús-Pous et al., 2019), with more than one string representing the same molecule." They also do not explicitly represent Hydrogen atoms, so the models would need to extract that implicit information by its own and properly balance The formula notation, on the other hand, more compact and more abstract. It represents all the atoms, and their multiplicities, in explicit way, but the correct reaction mechanism is harder to predict, as the formula does not contain structural information.
We now instantiate the two reasoning tasks defined in Section 2.1 with the new notation for stoichiometric coefficients in CHEMALGEBRA For Type 1 reactions, we randomly select a single multiplication factor from a set and apply it to all molecules in the reaction. Since the reactions in USPTO-BAL are balanced, but rarely contain molecules with coefficients greater than one, Type 1 reactions will likely contain molecules sharing the same random stoichiometric coefficient. We
We use the canonized molecular string representation (Arús-Pous al., 2019).

---

Under review as a conference paper at ICLR 2023
employ this variant to precisely measure how much the learning model can shorteut the reaction reasoning task by just predicting the same coefficient of the input reagents for the output products.
Instead, for Type 2 reactions, each molecule in the reaction reagents, reactants and products is randomly assigned a different coefficient and allowed to appear on the left or right side in addition to its original placement, thus creating confounder molecules that might not react. This process requires taking some precautions in order to ensure that the resulting reaction is still balanced. We detail the complete procedure create Type reactions in Appendix C. As this would make it difficult for a model to shortcut coefficient prediction, Type 2 reactions pose greater challenge than Type Table 3 illustrates examples of Type and derivations from a reference reaction, encoded in stoichiometric SMILES and raw formula representations.
We now have to decide how many times shall we create Type and variants of a reaction from USPTO-BAL at training time. We control this factor of variation by devising two augmentation strategies to measure how different training sizes could impact generalization:
x1 augmentations each reaction of USPTO-BAL is used to create a single stoichiometric reaction in CHEMALGEBRA.
x5 augmentations each reaction of USPTO-BAL is used to create five stoichiometric reactions in CHEMALGEBRA. All the stoichiometric reactions contain the same input/output molecules, but differ by the sampled values of the stoichiometric coefficients.
This leaves us with total of 8 datasets variants, one for each combination of the reasoning types, 2 molecule encodings and 2 augmentation amounts. A comprehensive overview of the eight CHEMALGEBRA datasets included in Appendix G. Lastly, we need to determine the different sets of coefficients to sample from across train and test distributions. We thus instantiate the chemical reactions in GEBRA by sampling the stoichiometric coefficients the interval S,,, = [1, 5]. We retain the same train, validation and test split as the original USPTO-BAL dataset. As outlined in Section 2.1, order to test systematic generalisation, we build three different test sets for each dataset variant:
In-distribution
test set: the test reactions are instantiated with stoichiometric coefficients drawn in the interval Sin. This is the "easy" scenario where the model should be expected to use the shortcuts for Type I reasoning.
Out-of-distribution test set: the test reactions instantiated with stoichiometrie coefficients drawn in the Sout = 6, 10] interval. This is the "hard" prediction task where to succeed a model would also require the arithmetic knowledge to derive coefficients in Sout from those in Sin- This challenging setting is introduced explicitly to stimulate future research to inject external knowledge or ways to bind symbols concepts.
Cross-distribution test set: half of training and test reactions is instantiated with stoichiometric coefficients in S.n. the other half in Sout- This a "medium" difficulty scenario where the model can infer arithmetic knowledge from the training data and still shortcut predictions for Type reactions.
4.1 BASELINE EXPERIMENTS
As part of our benchmark, we provide reference baseline results for the eight variations of CHEMAL- GEBRA described in Scction 4. We employ a Transformer architecture" akin to the sfte-offthe--a models of Table (see Appendix D for the details about the architecture and the training procedure). Our aim with these baselines is to analyse the performance variations with respect to different molecule representations, coefficient assignment types, and dataset sizes.
We measure the performance of the model as in a multilabel classification setting, where the task is to predict not only the exact bag of product molecules but also their stoichiometric coefficients." To do
"We chose to focus the Transformer for our baseline because i) the models with the highest accuracy on forward reaction prediction are all based on Transformers, ii) Transformers already currently used for many types reasoning tasks (Helwe al., 2021) and iii) while other specialized approaches, such as MEGAN (Sacha et al., 2021). can bake simple chemical rules into the model by design, they cannot be used to solve different kinds of reasoning tasks.
We measure the "moleculc-only" performance in Appendix
8

---

Under review as a conference paper at ICLR 2023
Table 4: Exact match (EM), Jaccard (JAC) and Fl scores evaluating the prediction of both molecules and coefficients on the variations of CHEMALGEBRA for in-, cross- and out-of-distribution general- ization.
IN
CROSS
OUT
TYPE
AUG.
EM
JAC
F1
EM
JAC
F1
EM
JAC
F1
FORMULA
29.99
63.03
72.42
37.81
67.40
75.72
0.36
3.89
6.48
T1
X5
28.87
62.10
71.69
25.85
60.64
70.59
0.38
7.30
12.15
x1
1.10
39.87
53.53
0.03
31.43
44.84
0.00
13.71
23.05
T2
X5
1.30
41.01
54.71
0.06
32.57
45.97
0.00
13.29
22.37
X1
4.17
44.43
57.49
5.96
45.74
58.55
0.04
1,88
3.18
SMILES
T1
X5
2.62
41.86
54.99
2.77
42.01
55.18
0.08
1.87
3.38
X1
0.03
23.37
35.37
10-2
22.73
34.57
0.00
8.90
15.61
T2
X5
0.04
24.00
36.18
10
22.73
34.67
0.00
7.44
13.08
so, we consider the bag of labels to be the bag of product molecules, in which a molecule appears as many times its correct coefficient states. We therefore employ classical multi-label performance metrics (Tsoumakas & Katakis, 2007) for the top-1 generated bags of products. match (EM) score the multi-label accuracy, also called Jaccard (JAC) score, and the Fl score metrics (see Appendix E for details). These metrics are differently forgiving, with the EM score being the most strict and Fl the most permissive.
use the exact
The results Table 4 show some interesting trends. First, as expected, the EM scores drops sharply setting. Not only the model is not able to recover the exact multiplicity of product molecules, but also their presence, as the F1 and JAC scores drop to much less than half the in-distribution scores. Type 2 tasks are confirmed to be much harder than Type 1, highlighted especially by EM scores.
the
Second, the baseline can actually achieve better dutti--distribution JAC and F1 scores Type 2 than Type tasks. This can be explained by considering that JAC and Fl scores are more permissive than EM and can increase if the prediction contains more instances of one molecule labels, even if the complete configuration does not exactly match.
Third, augmenting the training data fivefold (x5) does yield any breakthrough performance improvement. On the contrary, for Type reactions, it seems that additional augmentations can slightly confuse the model on which shorteut to use. Overall, this suggests that the CHEMALGEBRA tasks are challenging and robust to the usual data augmentation techniques that are usually used to improve the performance of Transformer-basee models (Schwaller al., 2019; Irwin et al., 2022). We recommend running CHEMALGEBRA x1 if one needs to save computation.
Finally, balancing raw chemical formulas reported to easier than doing that on SMILES. seems that predicting the stoichiometric coefficients is easier when using formulas, while even Type I in distriibuution reaction predictions are challenging when using graph structures, encoded as SMILES, inputs.
5 CONCLUSION
Our capacity to measure the reasoning capabilities for current machine learning models is bounded by the benchmark tasks that we have at our disposal now. Many of the current datasets used for this purpose cast reasoning as a subroutine of more complex problems, often expressed in natural language (Helwe et al., 2021: Wei et al., 2022). By doing it becomes hard to disentangle the systematic generalization ability statistical correlations in the data (Zhang et al., 2022; Branco et al., 2021).
model from a number confounding factors such as mere
We introduced CHEMALGEBRA to have a solid benchmark for robustly measuring the reasoning abilities of deep learning models over complex objects, such bags of graphs. CHEMALGEBRA offers a more controlled and versatile experimental setting including different reasoning tasks and graph encodings while remaining highly challenging in both the in- and out-of-distribttion
9

---

Under review as a conference paper at ICLR 2023
settings. In fact, it requires to combine statistical learning over graphs with algebraic reasoning the (sub-)graph structures and their multiplicity. As such, we think that the CHEMALGEBRA task can serve useful test bed for the upcoming generation of machine reasoning models.
5.1 LIMITATIONS AND FUTURE DIRECTIONS
While completely disentangling learning and reasoning might not be possible (as it is not in many real- world applications), we stress that the intended use of CHEMALGGERA is mainly for the evaluation of the reasoning abilities of neural models and specifically Transformers. Therefore, should not be considered to be dataset to train real-world models for pure chemical reaction prediction. We remark that other neural reaction prediction, such MEGAN, which inject prior knowledge about chemical reaction predictions, might possess more favourable inductive bias to solve the reasoning task. Our focus however on Transformers, which are emerging as machines. We believe CHEMALGEBRA can be solid benchmark to measure their capabilities and offer a more controlled environment where shortcutting can limited.
purpose learning and reasoning
We recognize the limitations of the re- re-balancing technique introduced in Section 3, which is simple heuristic and may abstracts-away several chemical nuances that can be relevant for pure chemical reaction predictions. At the same time, we believe that can it be simple initial solution to propel more rigorous and challenging algebraic reasoning benchmarks. Type 2 augmentation, while sound from an algebraic point of view, might also introduce noise in the chemical reaction prediction task.
We plan to design additional variations of CHEMALGEBRA to take into account the different facets of machine and human reasoning well more chemically and physically plausible constraints such as modeling redox reactions.
REPRODUCIBILITY STATEMENT
The dataset files, code to run the experiments scripts to compute the results of this paper can be found https: annnymous 4open. www.scceeeeeeeeeee.ee hhth/hheemmaaaaaaammec The construction of the BAL, TI and T2 variants of USPTO is described in Section 3 of the main paper. Additional details about the USPTO variants and the state-of-the-ee describes the eight dataset variants included in CHEMALGEBRA The detailed algorithm used to sample the stoichiometric coefficient of Type 2 tasks is reported in Appendix C. Appendix D contains additional information about the architecture and training procedure of the Transformer model used to compute the baseline results of Section 4.1. An extended description of the multilabel metrics used for evaluation is included in Appendix E. Finally, extensive overview of the CHEMALGEBRA dataset variants is given in Appendix G.
models are reported in Appendix Section
ACKNOWLEDGMENTS
We wish to thank Adrian Javaloy and Frank Mollica for their precious discussions and suggestions, which were crucial to improve the quality of the final manuscript.
REFERENCES
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh, Analyzing the behavior of visual question answering models. arXiv preprint 1606.073566 2016.
Hadeel Al-Negheimish, Pranava Madhyastha, and Alessandra Russo. Numerical reasoning in machine reading comprehension tasks: are we there yet? arXiv preprint arXiv:2109.08207 2021.
Josep Arús-Pous, Simon Viet Johansson, Oleksii Prykhodko, Esben Jannik Bjerrum, Christian Tyrchan, Jean-Louis Reymond, Hongming Chen, and Ola Engkvist. Randomized smiles strings improve the quality of molecular generative models. Journal of cheminformutics, 11(1::113, 2019.
Davide Bacciu and Marco Podda. Graphgen-redus a fast and lightweight recurrent model for labeled graph generation. In 2021 International Joint Conference on Neural Networks (IJCN), pp. 1-8. IEFF, 2021.
10

---

Under review as a conference paper at ICLR 2023
Davide Bacciu, Federico Errica, Alessio Micheli, and Marco Podda. A gentle introduction to deep learning for graphs. Neural Networks, 129:203-221, 2020.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: what is required and can it be learned? arXiv preprint arXiv: 1. 1.1289,, 2018.
Hangrui Bi, Hengyi Wang, Chence Shi, Connor Coley, Jian Tang, and Hongyu Guo. Non- autoregressive electron redistribution modeling for reaction prediction. International Conference on Machine Learning, pp. 904 913. PMLR, 2021.
Léon Bottou. From machine learning to machine reasoning. Machine learning, 94(2)133-1449 2014.
John Bradshaw, Matt J Kusner, Brooks Paige, Marwin HS Segler, and José Miguel Hemnándzz-Lobtt.. A generative model for electron paths. arXiv preprint arXiv:/805 10970, 2018.
John Bradshaw, Matt J Kusner, Brooks Paige, Marwin HS Segler, and José Miguel Hernández-Lobato. Generating molecules via chemical reactions. 2019.
John Bradshaw, Brooks Paige, Matt Kusner, Marwin Segler, and José Miguel Hernández-Lobato. Barking up the right tree: an approach to search over molecule synthesis dags. Advances in Neural Information Processing Systems, 33:6852 6866, 2020.
Ruben Branco, António Branco, Joao Rodrigues, and Joao Silva. Shortcutted commonsense: Data spuriousness in deep learning of commonsense reasoning, In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 1504 1521, 2021.
Adrian MP Braşoveanu and Rázvan Andonie. Visualizing transformers for nlp: a brief survey. In 2020 24th International Conference Information Visualisation (IV), pp. 270-279. IEEE, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Connor W Coley, Wengong Jin, Luke Rogers, Timothy Jamison, Tommi $ Jaakkola, William H Green, Regina Barzilay, and Klavs F Jensen. A graph-convolutional neural network model for the prediction of chemical reactivity. Chemical science, 10(2):370-377, 2019.
Hai Dai Nguyen and Koji Tsuda. A generative model for molecule generation based on chemical reaction trees. arXiv e-prints, pp. arXiv-2106, 2021.
David Fooshee, Aaron Mood, Eugene Gutman, Mohammadamin Tavakoli, Gregor Urban, Frances Liu, Nancy Huynh, David Van Vranken, and Pierre Baldi. Deep learning for chemical reaction prediction. Molecular Systems Design & Engineering. 3(3):442 452, 2018.
Wenhao Gao, Rocío Mercado, and Connor Coley. Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. arXiv preprint arXiv:2110.06380 2021.
Artur d' Avila Garcez, Marco Gori, Luis C Lamb, Luciano Serafini, Michael Spranger, and Son N Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. arXiv preprint arXiv: 000880 2019.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1))665-673, 2020.
Nicolas Gontier, Koustuv Sinha. Siva Reddy, and Chris Pal. Measuring systematic generalization in neural proof generation with transformers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp- 22231-22242.
Curran Associates, Inc., 2020.
Nikhil Goyal, Harsh Vardhan Jain, and Sayan Ranu. Graphgen: scalable approach to domain- agnostic labeled graph generation. In Proceedings of The Web Conference 2020, pp. 1253-1263, 2020.
11

---

Under review as a conference paper at ICLR 2023
Ryan-Rhys Griffiths and José Miguel Hemááddez-Lobato Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical science, 11(2):577-586 2020.
Chadi Helwe, Chloé Clavel, and Fabian M Suchanek. Reasoning with transformer-based mod- els: Deep learning, but shallow reasoning. In 3rd Conference on Automated Knowledge Base Construction, 2021.
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 3(1): 015022, 2022.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint vrXiv:1707.07335 2017.
Wengong Jin, Connor Coley, Regina Barzilay, and Tommi Jaakkola. Predicting organic reaction outcomes with weisfeiler-lehman network. Advances in neural information processing systems, 30, 2017.
Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecular optimization. arXiv preprint arXiv: / 2018.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Computing Surveys (CSUR), 2021.
Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing. Chembo: Bayesian optimization small organic molecules with syn- thesizable recommendations. In International Conference Artificial Intelligence and Statistics, pp. 3393-3403. PMLR, 2020.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. arXiv preprint arXiv:2106.048: 04554, 2021.
Jian Liu, Leyang Cui, Hammeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020.
Daniel Mark Lowe. Extraction of chemical structures and reactions from the literature. PhD thesis, University of Cambridge, 2012.
Gary Mareus. Deep learning: critical appraisal. arXiv preprint cxXiv:1801.00633 2018.
Gary Mareus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003.
Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Learning: Science and Technology, 2(2):055023, 2021.
Dai Hai Nguyen and Koji Tsuda. Generating reaction trees with cascaded variational autoencoders. The Journal of Chemical Physics, 2022.
Judea Pearl. Causality. Cambridge university press, 2009.
Wesley Wei Qian, Nathan T' Russell, Claire LW Simons, Yunan Luo, Martin D) Burke, and Jian Peng. Integrating deep neural networks and symbolic inference for organic reactivity prediction. 2020.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:220..07200 2022.
Mikolaj Sacha, Mikolaj Blaz, Piotr Byrski, Pawel Dabrowski-Tumankke Mikolaj Chrominski, Rafal and Stanistaw Jastrzebski. Molecule edit graph atten- tion network: modeling chemical reactions as sequences of graph edits. Journal of Chemical Information and Modeling, 61(7)33233-3284 2021.
Loska, Pawet
12

---

Under review as a conference paper at ICLR 2023
Timothy D Salatin and William L Jorgensen. Computer-assisted mechanistic evaluation of organic reactions. 1. overview. The Journal of Organic Chemistry, 45(11):2043-2000 2051,, 1980.
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Costas Bekas, et al. Molecular transformer for chemical reaction prediction and uncertainty estimation. 2019.
Marwin HS Segler, Mike Preuss, and Mark Waller. Planning chemical syntheses with deep neural networks and symbolic ai. Nature, 555(7698):604 610, 2018.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autocncoders. International conference on artificial neural networks, pp. 412-422. Springer, 2018.
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint 1rXiv:1908.06177, 2019.
Tiago Sousa, João Correia, Vitor Pereira, and Miguel Rocha. Generative deep learning for targeted compound design. Journal of Chemical Information and Modeling, 6111):53433536,, 2021.
Shane Storks, Qiaozi Gao, and Joyce Chai. Commonsense reasoning for natural language under- standing: A survey of benchmarks, resources, and approaches. arXiv preprint arXiv:190.00117, pp. 1-60, 2019.
Mohammadamin Tavakoli, Alexander Shmakov, Francesco Ceccarelli, and Pierre Baldi. Rxn hy- pergraph: hypergraph attention model for chemical reaction representation. arXiv preprint arXiv:2201.01 2022.
Igor V Tetko, Pavel Karpov, Ruud Van Deursen, and Guillaume Godin. nlp transformer models for direct and single-step retrosynthesis. Nature communications, I1(1): 1-11, 2020.
augmented
Truyen Tran, Vuong Le, Hung Le, and Thao M Le. From deep learning deep reasoning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 0076-4077, 2021.
Grigorios Tsoumakas and Ioannis Katakis. Multi-label classfficattn An overview. International Journal of Data Warehousing and Mining (IJDWM), 133) 2007.
Zhengkai Tu and Connor W Coley. Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction. arXiv preprint 10.00661,, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Cunxiang Wang, Boyuan Zheng, Yuchen Niu, and Yue Zhang. Exploring generalization ability of pretrained language models on arithmetic and logical reasoning. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 758-769, Springer, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou Chain of thought prompting elicits reasoning in large language models. arXiv preprint 1701 2022.
David Weininger. Smiles, chemical language and information system. 1. introduction methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Robert D Whitaker. An historical note on the conservation of mass. Journal of Chemical Education, 52(10):658, 1975.
Honghua Zhang. Liunian Harold Li, Tao Meng. Kai- Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11500 2022.
13

---

Under review as a conference paper at ICLR 2023
A RELATED WORK
ML for chemical predictions. Computational methods for chemistry have a long history of applications. While the first methods for chemical reaction predictions relied mostly on hand-crafted symbolic rules (Salatin & Jorgensen, 1980), in recent years the computational the community interests shifted towards the use of deep learning techniques and derivatives. Nowadays, deep learning is used for big variety of chemically related tasks, such molecular graph generations (Sousa et al., 2021; Goyal et al., 2020; Bacciu Podda, 2021; Simonovsky & Komodakis, 2018; 8; Mercado et al., 2021), molecule optimisation (Jin et al., 2018; Griffiths & Hernández-Lobato 2020; Korovina et al., 2020), and chemical reaction predictions. Focusing on the latter, we can find two related approaches: forward prediction, where the goal is to infer the product molecules given set of reactants, and retrosynthesis, a specular problem to forward synthesis consisting into predict the original reactants given a final product.
DL for chemical predictions. Initial attempts to use use deep learning models for reaction predic- tion used neural network to first identify the reaction centre in the reactants molecules (Coley et al., 2019) or pairwise reactivity of atom bonds (Jin et al., 2017; Fooshee et al., 2018). The proposals of the models were then used to algorithmically construct the final product of the reaction. A relevant collection of works aim at generating the whole reaction trees by iteratively selecting predefined reaction templates (Bradshaw et al., 2018; Segler et al., 2018; Bradshaw et al., 2020; Gao et al., 2021; Dai Nguyen & Tsuda, 2021; Nguyen & Tsuda, 2022). Reaction templates are helpful for preventing the models to generate incorrect molecules that violates chemical rules, however they tend to be too restrictive in general, making more difficult for the models to generalise to new types of reactions.
In the attempt to overcome the limitation of fixed reaction templates, it was proposed to combine powerful Transformer-basee (Vaswani et al., 2017) language models with the text SMILES represen- tations of molecules in order to generate reaction outcomes in one step. These models leverage the flexibility of their architecture to directly learn a mapping between the reactants and the products. The first work of this kind is the Molecular Transformer (Schwaller et al., 2019), able to beat the previous state of the art on both forward and backward prediction. Further improvements quickly followed with the Augmented Transformer model (Tetko et al., 2020). In this work, the authors focused on augmenting the original dataset by computing different variations of SMILES strings representing the same molecule.
The current t 1 accuracy state of the art for text-based, template- free models is represented by the Chemformer (Irwin al., 2022), where the authors use a supervupe pre-training large variety of known molecules in order to increase performance on the reaction prediction downstream task. An interesting line of research tries to leverage the natural graph structure of molecules. Tu & Coley (2021) combine Transformers with Deep Graph Networks (Bacciu et al., 2020) to get better initial molecular embeddings, while (Tavakoli al., 2022) build an hypergraph representation for the reaction by augmenting the disconnected molecular graph with "hypemodes" that represent molecules and reactions sides as a whole.
Finally, an orthogonal approach to chemical reaction prediction tries to leverage natural chemical constraints for ensuring the generation of chemically-plausibll reactions. Bradshaw al. (2018) model low-level reaction mechanisms for linear electron flow heterolythic reactions. This model generate the final products by predicting the linear electron flow on the initial set of reactants. Qian et al. (2020) combine neural networks with integer level programming constraints for expressing simple chemical constraints. Other works try to predict simplified version of the actual reactions mechanisms (often called grseudo-mechanisnee the molecular graphs (Sacha et al., 2021) or single-step predicting all possible bonds formation and deletion using multi-pointer decoding network (Bi et al., 2021). The architectural constraints of these models generally ensure that the products resulting from a reaction are sound from a chemical perspective.
either via auto-regressively generating edits to
Benchmarking and studying logical and algebraic reasoning. After the recent impressive results of large scale pre-trained Transformers, several works started to investigate the reasoning abilities of these models (Geirhos et al., 2020; Helwe et al., 2021: Tran et al., 2021). Wang et al. (2021) show that large scale language models can only generalise well when the test distribution is the same of the training distribution, while they struggle in cross-distribution and distr--iisstssuttoons scenarios. Liu
14

---

Under review as a conference paper at ICLR 2023
al. (2020) introduce dataset based natural language multiple-choice questions. Each question requires a certain amount of logical reasoning in order to reach the correct answer. The baseline results show that the performance of current language models is still far behind human beings. The out-of-distribution problem is also highlighted Razeghi et al. (2022), showing that the accuracy of these models on certain training item is proportion to the number of times that that item as been seen in during training.
15

---

Under review as a conference paper at ICLR 2023
B ADDITIONAL RESULTS AND DETAILS FROM SECTION 3
The USPTO-TI and USPTO-T2 variations are built in the following way:
USPTO-TI:
we duplicate all the molecules on their respective sides of the reaction (double reactants, and double products).
USPTO-12: for each reaction, we randomly select molecule from either the reactants or the products. We then replicate the selected molecule on both sides of the reaction.
Please note that, with USPTO data, there is straightforward way represent stoichiometry: in order to represent a double molecule in the reaction, we explicitly need to copy it twice in the data. Due to architectural limitations of the considered models, are constrained perform only limited amount of augmentations (e.g. it would be impossible to replicate the entire reactants more than twice without exceeding the maximum sequence length). Table report, as an example, the Sabatier reaction re-adapted to the BAL, Tl and T2 variants.
For the experiments, we consider the following Transformer-
sate-of-theearr models:
Molecular Transformer (MOL.T) (Schwaller al., 2019): the first wansformer-basse model that framed the problem of reaction predictions as sereence-o-sequenee text translation.
Augmented Transformer (AUG.T) (Tetko al., 2020): extension of the Molecular Trans- former, it was trained using different data augmentation schemes in order to boost its performance.
Chemformer (CHEMF) (Irwin al., 2022): yields the current state of the of top-1 accuracy on the reaction prediction tasks based on USPTO.
Graph2SMILES (G2S) (Tu & Coley, 2021): Transformer- based neural encoder for learning more expressive representations of the input molecules. The authors of G2S presents two versions (dgen and dgal) of the model employing a different type of neural encoder.
model that uses a graph-
Tables 6, 7, ad report the detailed results of these model over the USPTO, USPTO-BAL, USPTO- Ti and USPTO-T2 variants. We compute the following metrics:
Validity rate (VAL)): ratio of predictions that have a correct molecular structure.
Accuracy (ACC): ratio of correct predictions.
(ALO): define a prediction be "at-least-one accurate" if it contains all the ground truth molecules at least once. can therefore be seens as "molecule- only" accuracy, disregarding the multiplicity of the molccules.
st-least-one-acee
Balanced predictions rate (BAL): ratio of predictions that are balanced (i.e. that contains the same atoms as the reactants).
Deficitary predictions rate (DEF): ratio of predictions that do not contain atoms than are present in the reactants.
Exceeding predictions rate (EXC): ratio of predictions that contain atoms that are not present in the reactants.
Deficitary and exceeding prediction rate (D+E): ratio of prediction that are both exceeding and deficitary.
The table shows that, while the number of valid and accurate predictions is over 90% for all models, the balanced predictions are just minority, less than 10%. Most of the unbalanced predictions are of the deficitary type, while relative minority of the exceeding type. One of the causes for the bad performance can be identified in the training data: shown in Sec. 3.1, many of the reactions contained USPTO are already unbalanced. This unbalance is reflected on the outputs these models, another sign that these models relying memorising the data rather than trying to actually retrieve the correct underlying chemical mechanisms.
16

---

Under review as a conference paper at ICLR 2023
Table 5: Example of the the Sabatier reaction adapted to fit the BAL, T1 and T2 variations of the original USPTO. In TI, the reactants and products doubled. In T2, molecule (CO2) is added on both side of the reaction.
Reactants
Products
Original
CO2 + H₂
CH4 H2O
BAL
CO2 H2 + H2 + H2 H2 Ni
CH4 H2O + H₂O Ni
TI
CO2 + CO2 H2 + H2 +
CH4 + CH2 + H2O H2O +
T2
CO2 + CO2 + H2 + Ni
CO2 + CH2 + H2O Ni
Table 6: Results of predictions state-of-the- art models on the USPTO dataset.
VAL
ACC
BAL
DEF
EXC
D+E
MOL.T
100.0
90.4
9.06
90.86
0.28
0.21
AUG.T
99.97
91.1
9,44
90.40
0.59
0.43
CHEMF
87.20
92.8
6.71
76.33
48.28
31.33
Graph2SMILES (dgen)
99.98
90.3
8.83
90.62
1.60
1.06
Graph2SMILES (dgat)
99.98
90.3
8.88
90.52
1.67
1.08
Table 7: Results on predictions of state-of-the-arr models on USPTO-BAL.
VAL
ACC
BAL
DEF
EXC
D+E
MOL.T
99.85
1.39
1.50
98.48
0.30
0.29
CHEMF
78.20
0.0
4.51
92.39
22.52
19.43
G2S (dgat)
99.89
1.37
1.48
98.50
0.15
0.14
G2S (dgen)
99.85
1.40
1.46
98.53
0.16
0.15
Table 8: Results on predictions of state-of-the-art models on USPTO-T1 dataset.
VAL
ACC
ALO
BAL
DEF
EXC
D+E
MOL.T
99.03
0.04
56,66
0.18
99.76
1.41
1.37
CHEMF
91.88
0.23
34.41
0.29
99.22
6.31
5.83
G2S (dgat)
99.60
0.0
83.02
0.01
99.97
0.67
0.66
G2S (dgen)
99.67
0.0
84.21
0.01
99.97
0.64
0.64
Table 9: Results of predictions of state-of-the-art models USPTO-T2 dataset.
VAL
ACC
ALO
BAL
DEF
EXC
D+E
MOL.T
99.03
0.03
28.32
0.38
99.08
5.76
5.23
CHEMF
67.42
0.0
2.61
2.48
90.00
25.06
17.55
G2S (dgat)
99.66
0.005
40.79
0.06
99.88
1.35
1.30
G2S (dgen)
99.71
0.0025
41.43
0.04
99.90
1.19
1.14
17

---

Under review as a conference paper at ICLR 2023
C COMPLETE ALGORITHM FOR BUILDING THE TYPE-2 STOICHIOMETRIC AUGMENTATIONS
Let M be the of molecules in the reaction, first sample before random coefficient km E [1,5] for each molecule m E M. We then compute
k = min Am-
(8)
MEM
The final coefficients assigned as follows: the reactants side of the reaction, we have
km
if m is a reactant or reagent,
Vm E M.
km
(kim
k)
if m is product, and (km k) > 0,
(9)
0
otherwise.
Similarly, the products side, we have
kim
if m is product a reagent, if m is reactant, and (km k) > 0,
Vm E M. km
(km
k)
(10)
0
otherwise.
This assignment strategy takes into account the fact that specific quantities of reactants and reagents molecules are needed in order to get the respective products. The excess amount of molecules is just copied the other side (a specular argument can be done for products). Please not that, following this algorithm, the new added molecules are chosen among the molecules that are already present in the original reaction.
18

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL DETAILS ABOUT THE MODEL AND TRAINING PROCEDURE
We train a Transformer model (Vaswani et al., 2017) with 6 encoder and decoder layers. The number of attention heads is set to Both the embeddings size, the internal model size and the size of the fully connected layers is set to 512. Loss function used is the cross-entropy between the predicted and the target strings.
We use a batch size of 64 for all experiments. the Adam optimizer is used for training, with an initial learning rate 10 5, The learning rate is halved when no loss improvement over the validation set is observed for 1000 steps, while the training is stopped after no loss improvement over the validation set is observed for 5000 steps. We use label smoothing with smoothing parameter 0.1.
For evaluation, we auto-regressively generate the model's prediction one character at time, stopping the generation either when the model predicts an "end sequence" token or after 200 timesteps. The model is implemented and trained using the snsorch_lighnnngg library.
19

---

Under review as a conference paper at ICLR 2023
E MULTILABEL METRICS FOR EVALUATION
In CHEMALGEBRA, the model needs to predict a bag of product molecules, given an initial bag of reactants. This problem can be framed multilabel classification task, since multiple ground truth molecules (and their stoichiometric coefficients) have to be predicted at the same time. The computation of the usual classification metries (such as accuracy, precision, recall, etc.) in the multilabel setting is not trivial, as we need to take into account different failure modes of the model at the same time. For example, the model could either predict the wrong molecule, or the right molecule but with higher/lower coefficient than expected. For our tasks, we consider the Exact Match (EM), Jaccard (JAC) and Fl scores as multilabel metries:
EM N Σ T 10), JAC N
De Di
F1
2
O
Uil 10
(11)
N
Va
where Y4 and S are the k-hot binarized vectors of the ground truth and the prediction, respectively. N is the number test samples and is the indicator variable checking for exact equality between y: and ú;-
Due to computational efficiency reasons, we do not explicitly binarize the vectors. Instead, we compute the number of true positives, false positives and false negatives by comparing the coefficients of corresponding molecules in both the prediction and the ground truth.
For example, if the model predicts 3 H2O 2 HCC CO2 while the ground truth is 2 H2O 2 HHI CH2, we count true positives (two H₂O and two HCI), false positives (one excess H,O and one CO2), and I false negative (the CH2 in the ground truth). These counts are then combined get the final multilabel metrics.
20

---

Under review as a conference paper at ICLR 2023
F ADDITIONAL BASELINE RESULTS (MOLECULE-ONLY ACCURACY)
Table 10 contains the baseline results the CHEMALGEBRA benchmarks, considering only the molecular-level accuracy (i.e. taking only into account the prediction of the correct molecules, disregarding stoichiometric coefficients). Using the same example of Appendix E, where the model predicts H₂O + HCI CO2 while the ground truth is 2 H2O + HCI CH2, we now count true positives (the H2O and HCI molecules), false positives (CO2), and false negative (the CH₃ in the ground truth). In practice, here we are measuring only the predicted molecular structures, disregarding the stoichiometric coefficients.
We can observe that, for Type tasks, the performance are very similar to Table 4. This is not surprising, as the model can easily learn to copy the same coefficient to the output, leaving only the task of molecular graph prediction to learned. On the other hand, the performance on Type 2 variants are higher than Table 4. This can be explained the main challenge of Type 2 tasks is the prediction of the correct coefficients, which is disregarded by these "molecule-only" metrics. The consequence of the different choice of metrics can be also observed in the out-of-distributie performance, that in Table 10 is consistently higher than Table 4: not considering the stoichiometric coefficients makes the out-of-disrribution task very similar standard reaction prediction tasks (with the out-of-distribure coefficients acting as confounding tokens).
Table 10: Jaccard (JAC) and F1-score (F1) on the CHEMALGEBRA datasets.
test-in
test-cross
test-out
JAC
F1
JAC
F1
JAC
FORMULA I
Tix1
63.03
72.43
67.39
75.71
23.96
35.79
FORMULA T1x5
62.09
71.69
60.64
70.59
26.03
38.11
FORMULA T2x1
48.89
61.48
35.10
48.44
26.35
39.27
FORMULA T2x5
50.33
62.80
36.29
49.54
27.25
40.26
SMILES TIxl
44.43
57.49
45.74
58.54
24.44
36.52
SMILES_TIA
41.85
54.99
42.00
55.17
24.50
36.28
SMILES T2x1
27.63
40.57
24.64
36.97
18.03
28.45
SMILES T2x5
28.47
41.54
24.56
36.98
14.59
23.50
21

---

Under review as a conference paper at ICLR 2023
G OVERVIEW OF THE CHEMALGEBRA VARIANTS
Table contains an overview of the different CHEMALGEBRA variants that are described in Section Each variant contained separated folder. The filenames given according the following conventions:
Files with prefix "src" contain the input molecules, while files with prefix "tgt" contain the target molecules.
Substrings "train", "valid" or "test" refer to the respective subsets of data to be used during cross-validation.
Test files have the additional suffixes "_in" or " "_out", referring respectively to the in- distribution test set and out-of- offooff--tiittiionn test set, as described in Section 4.
Files with the cross" suffix refer to the cross-distribue Note that the "rrsss" tasks have their own dedicated training and validation
setting described in Section 4.
In order to provide visual intuition of the actual data contained in CHEMALGEBRA, show in Tables 12, 13, 14, 15, 16, 17, and 19 the first five training reactions for each CHEMALGEBRA variant.
Table 11: Summary of all eight variants of CHEMALGEBRA. "Name" is the name of the variant in the dataset's s files. The "Representation" column specifies which representation is used for the reactions of the dataset. The "Task Type" column determines the choice coefficients (according to the reasoning tasks defined in Section 2. The "Augmentation" column specifies how many stoichiometric assignments are sampled for each original USPTO reaction in the training set.
Name
Representation
Task Type
Augmentations
Chemalggebra___11__x
Chemical formula
Type
xl
Chema_g@ebra___111_55
Chemical formula
Type 1
x5
ChemaAllgebra____2__1
Chemical formula
Type
x1
ChenA_ggebra___12_.55
Chemical formula
Type 2
x5
ChemA_geebra____1__.
SMILES string
Type
xl
Chemklgebra____1_.x
SMILES string
Type
x5
ChemAlgebra___12_..
SMILES string
Type
x1
SeenAlgebra____2_x
SMILES string
Type
x5
22

---

Under review as a conference paper at ICLR 2023
Table 12: Sample of training chemical reactions of ChemALgebra___AL x1.
INPUT
TARGET
(5)C784C1806, 831 (5)CH5N
(5)822
(3)C358H204
15JC/N/CU2N, 15 R3CCCC3
(1)8. [1]]6]1]_C1_00 (13820. (1 61111111 (1)C28402
(15228402.(1)888)) [2]GG, (1))1/00000)11111
(41C4HBO. (4:E. 011268332223330001 (4)00000....-...... 1418
11022H0. (4) (4)
[4)82045. CH20, (4)42H3--0-00 [4]H4B-
(4)44-.(64)2268) (4 (4)CH20
(4JC5.H... CC-CSCCCCO222.. (418C1. (S/CEECCINO
16)C5R5N, ( M1331
Table 13: Sample of training chemical reactions of
TARGET
[31C/HACINDS, 331895N.13331
(3: R20
13TCH5H, (3J6784C1804, (31820
[31820, (3) [3)
131820. (3)CTECCLNO1. (3)CH5N
13)C9H9N204, (3]320, HCL
121820. (2)07F4C1N04. (23CH5N)
(2]C3HBN2C4, 12 18C1- 820
13)C784C1N06. (31CSEN. 13)820
(3]HC1. CENEX22...
Table 14: Sample of training chemical reactions of Chemalgebra___r2_x
INPUT
TARGET
(4)277441104, C2)88280 H2).22. (31CH5N
1820, (3)C338N2C44 (3177/6611778880...08 12) B1)
(3)31183122335. 111. (1)C7E7C12H, 111118811C13905
1218C1, (0)C113311211 (21C19811C1366 131C787112
(2) :318. 0011121111111 9C17884 C3)
1218. 111H.14CHH... 12)C2H102. (2)
(2) (111:2802222)01188800 111101011000. H5]HC1
(LIC15813CLENO - (3)8CL, 11C.LH.1111111111100 (811FER)
(3)226H33N363. 33C883
(13132048, (13C4H8O. (5:848
[1]CHHBO.
155 (1:CH20. (3382048. (2)CH03
(5/ 14).. 67)26H33530000 C120
(51Ne+.15 1226661
(2)25844CX022
(3)c5es HC)
C1)CEEAC (3:C1137CIN225.. 1418c1
141C1187CIN203, (3) C5355,
(1 13JC54CCN02.
[4)8C1, (4)8c1
Table 15: Sample of training chemical reactions of ChemAlgebraa__122
INPUT
TARGET
111G9R99244, 44 1820. / /CC.IIGGGAAO ICMSN. (2C
+4 1820, 141HG1 . 3-CHH8N204
13]C3H9M204, (3)878421114444... 131320. (2). 151CHEN
(3)CH (5) CEHEN204 BC) (31820
13/C/RSCIND9, CB.. H20 102280204
1210784C1IO0 (111L 15
(2)
[2]CH5N, (3J678401804,
(3) 122.002004...
(1YCH5N. (3)C8HEN204. H2: (4)8C1
H2) (2)CH3N.:1 :1/811 11ICTRCCLNN
(1)820, (1IC9992244 (218C1
23

---

Under review as a conference paper at ICLR 2023
Table 16; Sample of training chemical reactions of ChemAlgebra___TI_XX
INPUT
TARCarT
(C) s0mc 10y clecs ((N)) (-0) :410
1411-11121144
coe tell cll - (illltooeee (o1 Boence no2xl
(1:0), (icl Je careee cl
(4)
c2cce c2) 14t.c.........e... [4]62,
(4) [4) (C1) oi (4:
(4)20(=0)0. (4; [H]
(6000
[3] [BB0-1-;5]0m (+0) (010.
(5:0+B (=0) 10)0. [8a=] [B8[-]
(5) [Na (18 150.
18l I]
(5)CN (CC) (0) CDN (CDo7ocu cc2)c011 cloce (-d) (e1C)ee).
(5) (CI (C1 (-016l000 (XCC2 101 ObN (C (310c2))))
(5) [Na+ 1 151 101000-11151110000
153 [M]
(3) Declcc (0) a c2
[3:0-(00elcc (wd) [a3] eecteeeeee (3)c1. 13:clecncet 1316)
(3 ccnce"
Table 17: Sample of training chemical reactions of ChemAlgebra_ s_ ___toora________________________ x5.
TARGET
1410, [110 (110 ((-11
e (-0) (0-1)c1
[0-) 141C1
10w 101 clees ((X)) (60) (o 1el.0-00...000 (230..2000....... 1230
12SCNatope (0 [X]] (=0) 10 19 (21C) (230
14108, (4 (4:00 iet.://oeoeoooeeeeeeeeeeeeeeeee. 10-1201-4410
1elo: (M (-0) 10- 1416)
05) (3) clocs (C8+
(JCC,,/1ccccccccec(ooneeeeccc [X+] (#0) [0-1, (5)0
13,0. 1310=0 100 ol (C1))
(=0) (0-)) (0-))at.
13yCNcleec ID 00 (=0) 1310, 131C1
Table 18: Sample of training chemical reactions of ChemAlgebra_s_____
INPUY
TARGET
Nole (=0)
[0-1,
(0)) col (=0)
(5) 2..... 14]0nC (0) (c1) (+0) s2.
(3:0-0101 ciese (CL)
[0-]] [4)CN. [3]0.
[4) MCC (Gl) (3)ciciee (CL) (=C3ccaac))
(2:0). 101 eleete (G1) (C)))))n (-c 3 notst,
(3 [3:CCecce: (C1) (C1) 21.1ccle:2ee (C1))c (-C3emnec3)
(13 [M] Ge (G1 mns Inn (2)
(1 FC) (-)00(-0)0 10, [G)
(3) Dole mClinm (C (C48)) c2cue (2)
(2:Cele (C1) nae (Cu2cue c211001))2
1-0000000000000 (1)C)
(CL) (C10)) (2))
(2) (=0) 151C100001 1-10
[a84
(1))=0. (2]0=8100) (010_11
(010010011)e1 [2) (10-))0.
201 300e3ece 3130002
110*CE10- (0-1110-511110000)
(1)CH (CCL (CCeccc acc.ccccccccccc11ecccccceaaaaaae 0C cci.)
(3:00 [C) c-cccce (0c cC))
(3) [Na+]
(2)ciccnes_. 1331--CI011 clocenel. (31C1. (ICL.
12:clecnccl, (2]0-c (Cl) c4cc.nc1.
(3)=c0 (HO) [aB] cleccnc...cccl......cccccco (1)0=cl se (01 ((1))1
(5)0=C(0eled (40) [a3] celcnnel
Table 19: Sample of training chemical reactions of ChemAlgebra_$_f2.
13I (5 ine 101 clees 1C1 [[Ke] (e0t (o-1)ot.
JC.. (0e [N+] (-01 [0-).
[3)CHelccc (C1 01)) [3-1] (-0) [G-]_(430
[X-]
1410-C clccsiCLic (-0) [0-[)c1. (1)CI.
11 JCNcloee (=0) [N [N-] (=0) [o-1-
(2
[N-] (o: (4(j),, (1)))..111)). (130).
(5)00 (O) clccs (-0) (511
1510. (1001110-21000000 (C))cc (+0) [0-])c1
1510. 12161, (2108. (11...:44000 (4)0=c ((N) (-O) to Dee..
(310-c (3) elees (Clre ((8 (-0) (130.
(-01 CCNccoce
(3ICL. Ciccl
(4)
(310. [S)CNe
1310=C clocciille X+ (+0) (0-))
10+0 132 ICli (=3) (0-1)
12 [310
(0) (CI))([N)) (-0) 10- 181CH
[1]cNelooo (00) [M [0- (31.
(4]CH.(4)00 clece cloce (-0) (e-1)))
24


Reviews:
[{"review_id": "kX-va9_o8v", "reviewer": "Reviewer_xL5G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyses the performance of Transformers on reaction prediction, highlighting that these models break stochiometric rules and fail to follow \"common sense\". In light of this, the authors propose a benchmark – called ChemAlgebra – used to evaluate the ability of Transformer models to learn these generalizable chemical rules such as stochiometry. ", "summary_of_the_review": "Overall, I'm not convinced the comparisons made in this work are fully fair, or that they give useful signal, as in many cases the models seem to be evaluated too out-of-distribution for it to be meaningful. Moreover, I feel like some of the discussion in the paper may be misleading, and it's not clear which community is this work trying to benefit. Hence, for my initial evaluation I lean towards rejection.\n\n=== Update after author response ===\n\nThe author response clarified a few things, and the framing was changed to pivot more on the side of reasoning with Transformers, and less on usefulness for real-world reaction prediction. However, I'm still not very convinced about the OOD setups being sensible here (in particular, I share the concern of Reviewer 3JTU around some tokens not being seen during training), and about the signals from doing \"good reasoning\" and \"good reaction prediction\" being conflated. I raised my score to 5 to reflect the updates, but that already feels to be like a stretch; I would still vote for rejection.\n\nIf the paper gets rejected and resubmitted to the next venue, I think the suggestion from Reviewer 3JTU to include a task where the model only needs to predict the coefficients is very valuable.", "strength_and_weaknesses": "=== Strengths === \n\n(S1): The paper considers an important problem of chemical reaction prediction, and highlights limitations of current models (although these limitations were known before this work). \n\n \n\n=== Weaknesses === \n\n(W1): I am not convinced that the setup and evaluations done in this work are completely fair, and if it's realistic to expect the models to perform well in these settings. \n\n- As far as I understand, in Section 3.3 the authors train the models on the original USPTO dataset, and then evaluate on the newly introduced variants such as USPTO-BAL, reporting very low results. I'm not sure if it's realistic to expect the models to transfer here. As the author note, in USPTO some side products are omitted, hence the models are taught to omit side products; in a way, not making the reactions balanced is part of the task. The balanced reactions on the other hand teach the model to make different assumptions. Hence I'm not sure how the models could generalize from one to the other, if part of the task specification is contradictory. \n\n- Moreover, the authors refer to going from USPTO to USPTO-T{1,2} as a \"small distribution shift\", but I don't think this is accurate. Going from no repeated reactants to having repeats seems like a large shift, and I'm again not sure if it's realistic to expect the models to generalize. It would be a small shift to go from reactants repeated at most X times (for X > 1) to reactants repeated at most X' times (for X' > X). \n\n(W2): The language of the paper seems slightly misleading, as it seems to suggest not being able to capture stochiometry is a general problem that all reaction models would have. However, I think the discussion applies mostly to Transformer-based models (or other unconstrained ones). Models that predict the reactant-product transformation as a edit may capture stochiometry much better, and some may even satisfy it by design; for example, if the reactions are balanced and have a full atom mapping, and one trains a reaction model based on templates or MEGAN [1], only allowing bond changes, then atom counts would be preserved by design. Therefore, it's not completely clear that addressing the highlighted failures of Transformers is the right approach to getting generalizable reaction models, or at the very least it's not the only viable one. While the authors do discuss non-Transformer models in Appendix A, I'm not fully satisfied with the discussion there either; e.g. the authors mention several models, and summarize things by saying that these models are \"restrained to a particular class of reactions and cannot easily generalize to new types\", which I don't think is the case for MEGAN (which the authors do mention). \n\n(W3): The paper seems to pitch ChemAlgebra as a general benchmark that will help us improve the reasoning capabilities of DL models more broadly. However, all tasks in ChemAlgebra require not only learning generalizable chemical rules (such as stochiometry), but also predicting the actual reaction mechanism, and the latter is a harder task (since, as I mentioned in (W1), not breaking these simple rules can be baked in into the models by design, and one cannot bake in the ability to correctly predict reactions). Hence, I'm not sure improvements on ChemAlgebra would actually mean better reasoning, as opposed to just better reaction prediction. I think the paper should position itself here more explicitly: either as something that is supposed to be useful for the ML4Chem community (i.e. people working on reaction prediction) or something useful for the broad ML community (i.e. people that focus on Transformers but likely have no interest in chemistry). \n\n=== Other comments === \n\n(O1): \"To disentangle the reasoning task of predicting graph structures from that of counting molecule multiplicities, we devise a second, simpler, representation where molecules in the reactions are encoded as raw chemical formulas.\" - Are these tasks really disentangled here, given that even in the chemical formula representation one still has to predict the actual reaction mechanism (which the authors note is actually quite hard in this representation too)? I think fully disentangling these tasks would be e.g. giving the model both reactants and products, and asking only for the stochiometric coefficients. At that point however this becomes a toy task, and the fact that it came from a real task in chemistry is unimportant. \n\n(O2): Is each of the 8 tasks in ChemAlgebra comprised of its own train set and test set? If yes, I don't understand why the in-distribution results in Table 4 are so low (close to 0 for the SMILES representation). If this is in-distribution, shouldn't the models do similarly well to how they do on normal USPTO? \n\n(O3): I don't understand the last two sentences of Section 4.1, which talk about the formula-based tasks still being challenging. Why would we expect reaction prediction in the formula space to be easy? The authors themselves note that predicting the reaction mechanism from formulas themselves is actually harder. \n\n=== Nitpicks === \n\nHere I include some final nitpicks, which did not affect my score; they are here just to help improve the paper. \n\n- \"These impressive performance\" - maybe \"this\"? \n\n- \"This constraint over atoms of the molecules undergoes the true chemical mechanism behind reactions\" - maybe \"undergoes\" -> \"underpins\"? \n\n- \"In fact, predicted product molecules might sport\" - not sure what \"sport\" means here \n\n- \"apply it too all molecules\" - presumably \"too\" -> \"to\" \n\n \n\n=== References === \n\n[1] Sacha et al, \"Molecule Edit Graph Attention Network: Modeling Chemical Reactions as Sequences of Graph Edits\" ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GEgy97etBe", "reviewer": "Reviewer_7VBJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A new benchmark for chemical prediction is proposed, accounts for stoichiometry. this is supposedly harder than existing benchmarks.\ntransformer baselines are evaluated on the task, and don't seem to perform well.", "summary_of_the_review": "interesting benchmark, questions remain. ", "strength_and_weaknesses": "strengths\n- critical evaluation of the supposed state of the art in reaction prediction\n\nweaknesses\n- somewhat irrelevant to real-world reaction prediction\n- unclear whether baselines are run in a fair way\n- important baselines operating on graphs (e.g. MEGAN) missing", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8oxSd6DnstT", "reviewer": "Reviewer_3JTU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the ChemAlgebra dataset, a benchmark to evaluate the reasoning capabilities of deep learning models. The task of predicting the products and their multiplicities serves as a useful testbed for evaluating reasoning, as it involves not only manipulating complex discrete objects (reactants and products can be thoughts of bags of graphs), but also acting under precise algebraic constraints induced by the mass preservation principle. \n\nThe contributions of the authors include:\n* Casting chemical reaction prediction as a reasoning task,\n* Putting together the ChemAlgebra dataset,\n* Evaluating the performance of the vanilla transformer architecture on ChemAlgrebra, and a couple other more sophisticated models on a similar dataset, demonstrating that prediction of stoichiometrically-balanced chemical reactions is not yet a solved problem. \n", "summary_of_the_review": "The idea of using chemical reaction prediction to evaluate reasoning is a great one in my opinion. It is especially timely due to its difficulty yet amenability to rule-based solutions. I imagine current-day models (even large language models) would not be able to tackle this task very well. A well-executed dataset would serve well to community now and in the future, and this submission seems to be well on the track to achieve this. \n\nThis being said, I have a number concerns in the current version of the paper that motivate current score. I'll be revising my assessment based on author feedback.", "strength_and_weaknesses": "**STRENGTHS**: \n* **Central premise is sound:** Prediction of stoichiometrically-balanced chemical reactions do indeed appear to have a strong reasoning component (manipulation of complex and structured objects, and operating under algebraic constraints). It also seems like achieving perfect accuracy on this task does require actually capturing the correct algorithm as opposed to a handful of heuristics. This makes a well-executed dataset a valuable contribution for research on improving reasoning with deep learning methods. \n* **Sensible seed dataset:** The equations used in the dataset are originally collected from US Patents, which I think is a great way to ensure that the dataset is \"natural\" and possibly practically relevant. \n* **Potentially immune to chain-of-thought reasoning:** It appears that this task is challenging enough that even the largest and most capable models would suffer at tackling it via using chain-of-thought reasoning. \n\n****\n\n\n**WEAKNESSES** (some of these concerns might be proven wrong by the authors):\n* **Potential non-uniqueness of answers**: How common is it in the dataset that a given set of reactants can produce different products? I believe this could happen, especially when dealing with organic compounds (i.e. $C_5H_{12} + C_9H_{20}$ could produce $2C_7H_{16}$ or $C_3H_{8} + C_{11}H_{24}$, at least this algebraically checks out). How often does this happen in the dataset? If it does happen, what determines what the label is amongst all possible answers?\n* **SOTA appear not to have been evaluated on ChemAlgebra:** The authors argue that the reported results on ChemAlgebra use a transformer architecture \"akin to the state of the art models of Table 2\" (Table 2 contains results on experiments presented prior to the introduction to the ChemAlgebra dataset). The details in Appendix suggest that this model is a standard (i.e. the original Vaswani et. al. version) 6 layer encoder-decoder transformer. The models used to produce Table 2 appear to either be augmented with a GNN, or pretrained with a lot of data using the BART objective. Hence, it appears that the baseline used on ChemFormer seem weaker. (It's possible that I'm missing something here that the authors can rectify.)\n* **Absence of a version suitable for being used with large language models:** Current day DL-based reasoning research heavily relies on large models trained with huge pretraining datasets. Therefore, this submission would have been stronger if it had an \"inference only\" split that tests if large language models like GPT-3 (which are accessible for inference) can attain nontrivial accuracy. Providing explicit natural language instructions, expliting chain-of-thought reasoning and/or few-shot learning would all be interesting. \n* **The Type 1 and Type 2 corruptions possibly alter the labels:** It appear that both type 1 and type 2 corruptions could lead to a change in the labels. Here are some examples I can think of:\n  * Type 1: Take the following equation that shows a molecule of sugar being burnt: \"C6H12O6+6O2→6CO2+6H2O\". Doubling the coefficients on the left hand side, one could obtain the labels \"12CO2+12H2O\", or \"6CO2+6H2O + C6H12O6 + 6O2\" (the latter corresponds to only one of the sugar molecules being burnt, and the other remaining intact). Is one of the labels incorrect? Why? \n  * Type 2 (This is the corruption used in section 3.2): Consider the salt equation \"HCL + NaOH -> NaCl + H2O\". Add NH4OH on the left hand side to obtain \"HCL + NaOH + NH4OH\". Since NH4Cl is a viable molecule, the label to this input could both be \"NaCl + H2O + NH4OH\" or \"NH4Cl + HCL + H2O\". Is one of the labels incorrect? Why? \n* **Training details missing:** Details on how the models used to produce (including tuning details) Table 2 and 4 appear to be missing. (Most of the details for Table 4 are provided, though there still remain key design choices, like the loss function used)\n* **The OOD split of T1 predictions seem a bit problematic:** If I'm not mistaken, the ChemAlgebra dataset doesn't need any prior pretraining. Hence, the models need to learn \"what each digit means\" just by working with the ChemAlgebra dataset. It looks like when the OOD split of the ChemAlgebra dataset introduces significantly larger digits (in-dist 1-5 and ood: 6-10), the model is likely presented with tokens it has not even seen before. Isn't it expected that the performance will be terrible in this split, as the models doesn't have enough training signal to learn what the OOD digits mean? I don't see an easy way around it (i.e. as the authors argue, mixing in the OOD digits makes the split in-distribution.)\n\n****\n\n**QUESTIONS TO THE AUTHORS:**\n* **Why use a separate dataset (not ChemAlgebra) for Table 2?** \n  * I was wondering why ChemAlgebra appears to be distinct from the dataset used in Section 3. Why is the evaluation on that dataset much more comprehensive than the evalution on ChemAlgebra?\n* **How rule-based is the chemistry knowledge needed to perform perfectly?**  \n  * In other words, how would a novice human approach this task? While picking coefficients appear to be quite easy for a human, predicting what the output chemicals will be appear to be much more challenging. Is there a systematic way of approaching this? \n* **Is natural language prone to shortcuts:** In the 3rd paragraph of the Introduction, there's the sentence \"Natural language, despite its flexibility, is[...] prone to shortcuts.\" Do you mind giving an example for what a shortcut for natural language would be? \n* **Loss function:** What loss function did you use to train the transformer on ChemAlgebra?\n* **Soundness of the procedure to miss omitted molecules:** Is it guaranteed that the procedure used in Section 3.1 to balance unbalanced equations (by constructing one or more molecules using the excess/yet-unbalanced atoms) sound? Is it possible that this will lead to chemically unsound reactions, even though the molecules + coefficients check out? \n* **Tokenization:** How are the equations tokenized? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4T8QaqcJ6ZE", "reviewer": "Reviewer_Mbs4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduce a new dataset named CHEMALGEBRA, in order to robustly measure the reasoning abilities of deep learning models over complex objects, such as bags of graphs. CHEMALGEBRA offers a more controlled and versatile experimental setting – including different reasoning tasks and graph encodings – while remaining highly challenging in both the in- and out-of-distribution settings. In fact, it requires to combine statistical learning over graphs with algebraic reasoning over the (sub-)graph structures and their multiplicity. ", "summary_of_the_review": "The idea of this paper is interesting but seems useless.", "strength_and_weaknesses": "Strength:\n1. This dataset is somewhat interesting and novel. It is related to the problem of retro-synthesis.\n2. This paper evaluate the current state-of-the-art Transformers for chemical reaction predictions, showing that they fail to robustly generalise when reasoning on simple variants of the chemical reaction dataset.\n\nWeaknesses:\n1. This paper claims that chemical reaction prediction is a reasoning task. Why? Please define the reasoning task.\n2. I am not sure the role of this dataset. Did it could improve the reasoning capacity of neural networks? \n3. Suppose that we design a perfect for this dataset, what can this model be used for? The applications of this dataset are still not clear for me.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2cGYy-4xr5", "reviewer": "Reviewer_gwmV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new algebraic reasoning task and benchmark on chemical reactions. The new task is a modified version of chemical reaction prediction. Instead of only predicting the main product in the reaction, we need to predict all the products, including the by-products and the multiplicities of the product molecules. The proposed benchmark was constructed out of the USPTO-MIT dataset by doing a re-balancing on the product molecules. To establish the baselines, molecular transformers and graph neural networks were tested on the main benchmark and its variations. The task was shown to be a challenging one for the existing baselines from the experiment results.", "summary_of_the_review": "In general, the paper is well-written. The proposed task is novel and interesting for both math QA and ML for science communities. However, the paper needs to justify more on its impact on real-world applications and include a straight-forward but not end-to-end baseline.", "strength_and_weaknesses": "Strength:\n- This paper is well-written and easy to follow.\n- The formulation of the task is novel. It combines the flavor of algebraic reasoning from math QA and the flavor of learning chemical reactions. This would be an interesting task for researchers in both domains.\n- Several variations of the dataset were proposed (BAL, T1, T2, and other 8 variations) to help researchers understand more about the distribution of the datasets, the difficulties of each dataset, and what aspects of the datasets are more challenging.\n- Recent chemical reaction baselines are included in the proposed benchmark.\n\nWeaknesses:\n- Despite the novelty and interesting side of the proposed task, the usefulness of such a task in real-world applications is still less certain. The current reaction prediction model can be used to predict the main product, and with the re-balancing techniques, we can derive the side-product as well as the multiplicities of the product molecules. The paper needs to justify more why such a task would be meaningful in real-life.\n- Similar to the first argument, such a main product prediction + re-balancing method can be added as one of the baselines in the benchmarks. The method is straightforward and should be included as a counterpart of the end-to-end baselines.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject