Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A new benchmark for chemical prediction is proposed, accounts for stoichiometry. this is supposedly harder than existing benchmarks.
transformer baselines are evaluated on the task, and don't seem to perform well.
strength_and_weaknesses: strengths
- critical evaluation of the supposed state of the art in reaction prediction

weaknesses
- somewhat irrelevant to real-world reaction prediction
- unclear whether baselines are run in a fair way
- important baselines operating on graphs (e.g. MEGAN) missing
clarity,_quality,_novelty_and_reproducibility: clarity:
it is not clear whether the baselines are actually retrained on the new benchmark datasets, or whether checkpoint trained on the unbalanced USPTO is used. If the latter is the true, it's no surprise that the models perform poorly (because the main assumption for MLE is that training and test set should be sampled from the same distribution). the authors need to clarify this.

quality:
the task of predicting molecular formulas is very hard, since it is ambiguous. it's also not clear to me why this is relevant to chemistry.
summary_of_the_review: interesting benchmark, questions remain.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new algebraic reasoning task and benchmark on chemical reactions. The new task is a modified version of chemical reaction prediction. Instead of only predicting the main product in the reaction, we need to predict all the products, including the by-products and the multiplicities of the product molecules. The proposed benchmark was constructed out of the USPTO-MIT dataset by doing a re-balancing on the product molecules. To establish the baselines, molecular transformers and graph neural networks were tested on the main benchmark and its variations. The task was shown to be a challenging one for the existing baselines from the experiment results.
strength_and_weaknesses: Strength:
- This paper is well-written and easy to follow.
- The formulation of the task is novel. It combines the flavor of algebraic reasoning from math QA and the flavor of learning chemical reactions. This would be an interesting task for researchers in both domains.
- Several variations of the dataset were proposed (BAL, T1, T2, and other 8 variations) to help researchers understand more about the distribution of the datasets, the difficulties of each dataset, and what aspects of the datasets are more challenging.
- Recent chemical reaction baselines are included in the proposed benchmark.

Weaknesses:
- Despite the novelty and interesting side of the proposed task, the usefulness of such a task in real-world applications is still less certain. The current reaction prediction model can be used to predict the main product, and with the re-balancing techniques, we can derive the side-product as well as the multiplicities of the product molecules. The paper needs to justify more why such a task would be meaningful in real-life.
- Similar to the first argument, such a main product prediction + re-balancing method can be added as one of the baselines in the benchmarks. The method is straightforward and should be included as a counterpart of the end-to-end baselines.

clarity,_quality,_novelty_and_reproducibility: This paper is clearly written with high quality. The work is novel and reproducible.
summary_of_the_review: In general, the paper is well-written. The proposed task is novel and interesting for both math QA and ML for science communities. However, the paper needs to justify more on its impact on real-world applications and include a straight-forward but not end-to-end baseline.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper analyses the performance of Transformers on reaction prediction, highlighting that these models break stochiometric rules and fail to follow "common sense". In light of this, the authors propose a benchmark – called ChemAlgebra – used to evaluate the ability of Transformer models to learn these generalizable chemical rules such as stochiometry. 
strength_and_weaknesses: === Strengths === 

(S1): The paper considers an important problem of chemical reaction prediction, and highlights limitations of current models (although these limitations were known before this work). 

 

=== Weaknesses === 

(W1): I am not convinced that the setup and evaluations done in this work are completely fair, and if it's realistic to expect the models to perform well in these settings. 

- As far as I understand, in Section 3.3 the authors train the models on the original USPTO dataset, and then evaluate on the newly introduced variants such as USPTO-BAL, reporting very low results. I'm not sure if it's realistic to expect the models to transfer here. As the author note, in USPTO some side products are omitted, hence the models are taught to omit side products; in a way, not making the reactions balanced is part of the task. The balanced reactions on the other hand teach the model to make different assumptions. Hence I'm not sure how the models could generalize from one to the other, if part of the task specification is contradictory. 

- Moreover, the authors refer to going from USPTO to USPTO-T{1,2} as a "small distribution shift", but I don't think this is accurate. Going from no repeated reactants to having repeats seems like a large shift, and I'm again not sure if it's realistic to expect the models to generalize. It would be a small shift to go from reactants repeated at most X times (for X > 1) to reactants repeated at most X' times (for X' > X). 

(W2): The language of the paper seems slightly misleading, as it seems to suggest not being able to capture stochiometry is a general problem that all reaction models would have. However, I think the discussion applies mostly to Transformer-based models (or other unconstrained ones). Models that predict the reactant-product transformation as a edit may capture stochiometry much better, and some may even satisfy it by design; for example, if the reactions are balanced and have a full atom mapping, and one trains a reaction model based on templates or MEGAN [1], only allowing bond changes, then atom counts would be preserved by design. Therefore, it's not completely clear that addressing the highlighted failures of Transformers is the right approach to getting generalizable reaction models, or at the very least it's not the only viable one. While the authors do discuss non-Transformer models in Appendix A, I'm not fully satisfied with the discussion there either; e.g. the authors mention several models, and summarize things by saying that these models are "restrained to a particular class of reactions and cannot easily generalize to new types", which I don't think is the case for MEGAN (which the authors do mention). 

(W3): The paper seems to pitch ChemAlgebra as a general benchmark that will help us improve the reasoning capabilities of DL models more broadly. However, all tasks in ChemAlgebra require not only learning generalizable chemical rules (such as stochiometry), but also predicting the actual reaction mechanism, and the latter is a harder task (since, as I mentioned in (W1), not breaking these simple rules can be baked in into the models by design, and one cannot bake in the ability to correctly predict reactions). Hence, I'm not sure improvements on ChemAlgebra would actually mean better reasoning, as opposed to just better reaction prediction. I think the paper should position itself here more explicitly: either as something that is supposed to be useful for the ML4Chem community (i.e. people working on reaction prediction) or something useful for the broad ML community (i.e. people that focus on Transformers but likely have no interest in chemistry). 

=== Other comments === 

(O1): "To disentangle the reasoning task of predicting graph structures from that of counting molecule multiplicities, we devise a second, simpler, representation where molecules in the reactions are encoded as raw chemical formulas." - Are these tasks really disentangled here, given that even in the chemical formula representation one still has to predict the actual reaction mechanism (which the authors note is actually quite hard in this representation too)? I think fully disentangling these tasks would be e.g. giving the model both reactants and products, and asking only for the stochiometric coefficients. At that point however this becomes a toy task, and the fact that it came from a real task in chemistry is unimportant. 

(O2): Is each of the 8 tasks in ChemAlgebra comprised of its own train set and test set? If yes, I don't understand why the in-distribution results in Table 4 are so low (close to 0 for the SMILES representation). If this is in-distribution, shouldn't the models do similarly well to how they do on normal USPTO? 

(O3): I don't understand the last two sentences of Section 4.1, which talk about the formula-based tasks still being challenging. Why would we expect reaction prediction in the formula space to be easy? The authors themselves note that predicting the reaction mechanism from formulas themselves is actually harder. 

=== Nitpicks === 

Here I include some final nitpicks, which did not affect my score; they are here just to help improve the paper. 

- "These impressive performance" - maybe "this"? 

- "This constraint over atoms of the molecules undergoes the true chemical mechanism behind reactions" - maybe "undergoes" -> "underpins"? 

- "In fact, predicted product molecules might sport" - not sure what "sport" means here 

- "apply it too all molecules" - presumably "too" -> "to" 

 

=== References === 

[1] Sacha et al, "Molecule Edit Graph Attention Network: Modeling Chemical Reactions as Sequences of Graph Edits" 
clarity,_quality,_novelty_and_reproducibility: Clarity: Needs improvement (O1, O2, O3), and the expected use of this work (or the target community) should be made more explicit (W2, W3). 

 

Quality: On the high level the setup is reasonable, but I have doubts about many details (see the "Strengths and Weaknesses" section), and I'm not convinced the comparisons made in this work are very useful. 

 

Reproducibility: Seems OK, although some of the numbers look confusing (O2), making me think that some of the details were either not provided or I didn't fully understand them. 
summary_of_the_review: Overall, I'm not convinced the comparisons made in this work are fully fair, or that they give useful signal, as in many cases the models seem to be evaluated too out-of-distribution for it to be meaningful. Moreover, I feel like some of the discussion in the paper may be misleading, and it's not clear which community is this work trying to benefit. Hence, for my initial evaluation I lean towards rejection.

=== Update after author response ===

The author response clarified a few things, and the framing was changed to pivot more on the side of reasoning with Transformers, and less on usefulness for real-world reaction prediction. However, I'm still not very convinced about the OOD setups being sensible here (in particular, I share the concern of Reviewer 3JTU around some tokens not being seen during training), and about the signals from doing "good reasoning" and "good reaction prediction" being conflated. I raised my score to 5 to reflect the updates, but that already feels to be like a stretch; I would still vote for rejection.

If the paper gets rejected and resubmitted to the next venue, I think the suggestion from Reviewer 3JTU to include a task where the model only needs to predict the coefficients is very valuable.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose the ChemAlgebra dataset, a benchmark to evaluate the reasoning capabilities of deep learning models. The task of predicting the products and their multiplicities serves as a useful testbed for evaluating reasoning, as it involves not only manipulating complex discrete objects (reactants and products can be thoughts of bags of graphs), but also acting under precise algebraic constraints induced by the mass preservation principle. 

The contributions of the authors include:
* Casting chemical reaction prediction as a reasoning task,
* Putting together the ChemAlgebra dataset,
* Evaluating the performance of the vanilla transformer architecture on ChemAlgrebra, and a couple other more sophisticated models on a similar dataset, demonstrating that prediction of stoichiometrically-balanced chemical reactions is not yet a solved problem. 

strength_and_weaknesses: **STRENGTHS**: 
* **Central premise is sound:** Prediction of stoichiometrically-balanced chemical reactions do indeed appear to have a strong reasoning component (manipulation of complex and structured objects, and operating under algebraic constraints). It also seems like achieving perfect accuracy on this task does require actually capturing the correct algorithm as opposed to a handful of heuristics. This makes a well-executed dataset a valuable contribution for research on improving reasoning with deep learning methods. 
* **Sensible seed dataset:** The equations used in the dataset are originally collected from US Patents, which I think is a great way to ensure that the dataset is "natural" and possibly practically relevant. 
* **Potentially immune to chain-of-thought reasoning:** It appears that this task is challenging enough that even the largest and most capable models would suffer at tackling it via using chain-of-thought reasoning. 

****


**WEAKNESSES** (some of these concerns might be proven wrong by the authors):
* **Potential non-uniqueness of answers**: How common is it in the dataset that a given set of reactants can produce different products? I believe this could happen, especially when dealing with organic compounds (i.e. $C_5H_{12} + C_9H_{20}$ could produce $2C_7H_{16}$ or $C_3H_{8} + C_{11}H_{24}$, at least this algebraically checks out). How often does this happen in the dataset? If it does happen, what determines what the label is amongst all possible answers?
* **SOTA appear not to have been evaluated on ChemAlgebra:** The authors argue that the reported results on ChemAlgebra use a transformer architecture "akin to the state of the art models of Table 2" (Table 2 contains results on experiments presented prior to the introduction to the ChemAlgebra dataset). The details in Appendix suggest that this model is a standard (i.e. the original Vaswani et. al. version) 6 layer encoder-decoder transformer. The models used to produce Table 2 appear to either be augmented with a GNN, or pretrained with a lot of data using the BART objective. Hence, it appears that the baseline used on ChemFormer seem weaker. (It's possible that I'm missing something here that the authors can rectify.)
* **Absence of a version suitable for being used with large language models:** Current day DL-based reasoning research heavily relies on large models trained with huge pretraining datasets. Therefore, this submission would have been stronger if it had an "inference only" split that tests if large language models like GPT-3 (which are accessible for inference) can attain nontrivial accuracy. Providing explicit natural language instructions, expliting chain-of-thought reasoning and/or few-shot learning would all be interesting. 
* **The Type 1 and Type 2 corruptions possibly alter the labels:** It appear that both type 1 and type 2 corruptions could lead to a change in the labels. Here are some examples I can think of:
  * Type 1: Take the following equation that shows a molecule of sugar being burnt: "C6H12O6+6O2→6CO2+6H2O". Doubling the coefficients on the left hand side, one could obtain the labels "12CO2+12H2O", or "6CO2+6H2O + C6H12O6 + 6O2" (the latter corresponds to only one of the sugar molecules being burnt, and the other remaining intact). Is one of the labels incorrect? Why? 
  * Type 2 (This is the corruption used in section 3.2): Consider the salt equation "HCL + NaOH -> NaCl + H2O". Add NH4OH on the left hand side to obtain "HCL + NaOH + NH4OH". Since NH4Cl is a viable molecule, the label to this input could both be "NaCl + H2O + NH4OH" or "NH4Cl + HCL + H2O". Is one of the labels incorrect? Why? 
* **Training details missing:** Details on how the models used to produce (including tuning details) Table 2 and 4 appear to be missing. (Most of the details for Table 4 are provided, though there still remain key design choices, like the loss function used)
* **The OOD split of T1 predictions seem a bit problematic:** If I'm not mistaken, the ChemAlgebra dataset doesn't need any prior pretraining. Hence, the models need to learn "what each digit means" just by working with the ChemAlgebra dataset. It looks like when the OOD split of the ChemAlgebra dataset introduces significantly larger digits (in-dist 1-5 and ood: 6-10), the model is likely presented with tokens it has not even seen before. Isn't it expected that the performance will be terrible in this split, as the models doesn't have enough training signal to learn what the OOD digits mean? I don't see an easy way around it (i.e. as the authors argue, mixing in the OOD digits makes the split in-distribution.)

****

**QUESTIONS TO THE AUTHORS:**
* **Why use a separate dataset (not ChemAlgebra) for Table 2?** 
  * I was wondering why ChemAlgebra appears to be distinct from the dataset used in Section 3. Why is the evaluation on that dataset much more comprehensive than the evalution on ChemAlgebra?
* **How rule-based is the chemistry knowledge needed to perform perfectly?**  
  * In other words, how would a novice human approach this task? While picking coefficients appear to be quite easy for a human, predicting what the output chemicals will be appear to be much more challenging. Is there a systematic way of approaching this? 
* **Is natural language prone to shortcuts:** In the 3rd paragraph of the Introduction, there's the sentence "Natural language, despite its flexibility, is[...] prone to shortcuts." Do you mind giving an example for what a shortcut for natural language would be? 
* **Loss function:** What loss function did you use to train the transformer on ChemAlgebra?
* **Soundness of the procedure to miss omitted molecules:** Is it guaranteed that the procedure used in Section 3.1 to balance unbalanced equations (by constructing one or more molecules using the excess/yet-unbalanced atoms) sound? Is it possible that this will lead to chemically unsound reactions, even though the molecules + coefficients check out? 
* **Tokenization:** How are the equations tokenized? 
clarity,_quality,_novelty_and_reproducibility: **CLARITY:** The paper is well-written throughout and is structured well. That being said, it still takes a couple of pages to finally see how the inputs and outputs look like for the chemical reaction prediction task. This can easily be remedied by either adding a short paragraph in the introduction, or perhaps adding a figure early on that demonstrates how input-output pairs look like in ChemAlgebra.
  * **Potential typo:**  The first paragraph of Section 3.1 appear to refer to two types of USPTO-MIT dataset, one that cites Lowe (2012), the another that cites Jin et al. (2017). Just wanted to check whether this is a typo and the first one is meant to be just called USPTO? 

**QUALITY:** My assessment of quality hinges upon the answers to some of the clarification questions I've asked above. 

**REPRODUCIBILITY**: There doesn't seem to be enough details to retrain the models used to report the main performance results. Also, the ChemAlgebra dataset also don't appear to be linked in the submission (is there a way to look at the samples during review?)
summary_of_the_review: The idea of using chemical reaction prediction to evaluate reasoning is a great one in my opinion. It is especially timely due to its difficulty yet amenability to rule-based solutions. I imagine current-day models (even large language models) would not be able to tackle this task very well. A well-executed dataset would serve well to community now and in the future, and this submission seems to be well on the track to achieve this. 

This being said, I have a number concerns in the current version of the paper that motivate current score. I'll be revising my assessment based on author feedback.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduce a new dataset named CHEMALGEBRA, in order to robustly measure the reasoning abilities of deep learning models over complex objects, such as bags of graphs. CHEMALGEBRA offers a more controlled and versatile experimental setting – including different reasoning tasks and graph encodings – while remaining highly challenging in both the in- and out-of-distribution settings. In fact, it requires to combine statistical learning over graphs with algebraic reasoning over the (sub-)graph structures and their multiplicity. 
strength_and_weaknesses: Strength:
1. This dataset is somewhat interesting and novel. It is related to the problem of retro-synthesis.
2. This paper evaluate the current state-of-the-art Transformers for chemical reaction predictions, showing that they fail to robustly generalise when reasoning on simple variants of the chemical reaction dataset.

Weaknesses:
1. This paper claims that chemical reaction prediction is a reasoning task. Why? Please define the reasoning task.
2. I am not sure the role of this dataset. Did it could improve the reasoning capacity of neural networks? 
3. Suppose that we design a perfect for this dataset, what can this model be used for? The applications of this dataset are still not clear for me.


clarity,_quality,_novelty_and_reproducibility: This paper is clear, novel and can be reproduced.
summary_of_the_review: The idea of this paper is interesting but seems useless.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a benchmark for measuring reasoning capabilities of deep models through predicting chemical reactions. The paper is well-written and easy to follow. It also nicely shows some of the limitations of transformers. However, the main shortcoming of the paper is that as a reasoning benchmark, it has issues that limits its effectiveness in measure the reasoning capabilities of the models. In my mind, the most important limitations are regarding the separation of chemistry knowledge from reasoning capabilities and the issue with unseen digits raised by reviewer 3JTU. Therefore, I'm recommending rejection. However, I suggest authors to take reviewers suggestions into account and resubmit.
justification_for_why_not_higher_score: As explained by reviewer 3JTU, there are fundamental design choices made in the task that is problematic as reasoning benchmark.
Decision: Reject