Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-attention based architecture called PsiFormer for approximating many-electron Schrödinger equation. The architecture uses self-attention transformer to learn the embedding of electrons and the pairwise features are incorporated through Jastrow factor. The authors did a series of experiments on multiple molecular systems and compared PsiFormer with FermiNet, the previous state-of-the-art system for this task.
strength_and_weaknesses: The strength of this paper is that PsiFormer shows improvement over the FermiNet, especially on larger atoms with more electrons. The weakness, however, is mostly about the experiment section. There are many parts of the experiment that needs further clarification. 
1. It's quite unclear whether the results are significant. The difference between FermiNet and PsiFormer is very close (within 0.01 for smaller atoms and 0.1 for larger atoms). Also, what's the ground truth for each electron system? Is it the result coming from VMC or DMC? If so, shouldn't you report the mean absolute error (MAE) between FermiNet/PsiFormer's prediction and VMC/DMC? Should a system get a lower MAE instead?
2. What's the level of accuracy (or precision) needed for an algorithm to be useful? Is FermiNet good enough already? (again, improving results by 0.01 seems very small but I can be wrong).
3. Why do you only report $\Delta E_{mono}$ and $\Delta E_{10A}$ for Benzene Dimer but not third-row atoms?
clarity,_quality,_novelty_and_reproducibility: Overall, the clarity of the experimental section can be much improved by explaining why the results in tables 1-3 are significant and how to interpret these results. The novelty of the work is moderate -- it uses self-attention and combine the pairwise features through Jastrow factors. It's also unclear why such design is the best. For example, why not use relational attention to incorporate pairwise features in the self-attention layer? Why do you include the pairwise features at the very end? Is this modeling choice guided by first principles?
summary_of_the_review: Overall, it is unclear to me how significant the results are and hard for me to draw a conclusion. I am not an expert in the field of quantum chemistry calculations so I am not confident in my judgment. I vote for weak rejection for now but open to suggestions during author feedback and discussion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This is an interdisciplinary research paper on AI for science, it proposes a neural network framework based on self-attention mechanism, which can get an Anstazes solving the many-electron Schrodinger equation. The authors scaled the input features and concatenate the numerical spin into the input feature vector. Then they project input features to the same dimension of attention inputs by linear mapping to add self-attention mechanism into neural network framework. 
strength_and_weaknesses: The idea is novel for AI+Science. However, there are some incomplete or vague expressions in this paper. Please improve the language presentation carefully. The motivation for joining self-attention is not clear enough. How does the attention mechanism work on gating interactions between electrons and make the framework better? 
clarity,_quality,_novelty_and_reproducibility: The language used in this paper is very professional, but the explanation of terms is too brief, which is not very friendly to non-major readers. For the experimental part, there are too few baselines selected, which makes the result not strong enough. It is recommended to introduce more baselines or consider adding different kinds of attention mechanisms in these frameworks to compare with PsiFormer. I believe that the results will be more convincing. 
summary_of_the_review: AI for science is novel for our community. However, the contribution of this work is to introduce self-attention. It is an incremental contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new attention-based neural architecture for solving many-electron Schrodinger equation by optimizing the variational quantum Monte Carlo objective. While previous works proposed architectures like FermiNet, the proposed PsiFormer differs by (1) using self-attention between $O(N)$ electron-nuclear features and (2) incorporating $O(N^{2})$ electron-electron features through (parameterized) Jastrow factor. The proposed algorithm empirically outperforms the prior works in various experiments.
strength_and_weaknesses: Strength 1: This paper studies a very important problem in computational chemistry: solving the many-electron Schrodinger equation. Even a small progress in this area is likely to be impactful for future research.

Strength 2: The proposed self-attention mechanism is reasonable since the self-attention mechanism has shown success for many machine learning applications. 

Strength 3: The experiments are thorough and it looks like the authors did a good job at solving a complex problem that requires a lot of engineering. 

Weakness 1: The proposed idea itself is not very novel. To my knowledge, the novelty of this paper is limited to the self-attention mechanism and the Jastrow factor-based parameterization of electron-electron features. Both of them has already been considered in prior works, like the FermiNet.

Weakness 2: I hope the authors could further highlight the running time of their algorithm in the main text. I think this is importance since (1) running time is important for this type of problem and (2) the proposed self-attention mechanism is known to be quite slow. Currently, I find that the authors report their training time in Table 9 of the appendix. I guess this is the time required to solve the Schrodinger equation for a molecule. How does this time compare to the existing DNN-based or non-DNN-based solvers?

Weakness 3: I would like to have seen some justification behind using the self-attention mechanism. Besides the additional expressive power, is there any reason that self-attention is more useful for parameterization of the energy function? Is there any gating mechanism in “ground truth” for solving the Schrodinger equation?

Weakness 4: There is no explanation on which Markov chain Monte Carlo algorithm is used for the variational quantum Monte Carlo. 

Weakness 5: Since the authors do not provide a code repository for their paper, given complexity of the problem being solved, the possibility of reproducing this paper’s experiment is very low.

Question: There are some works on graph-based Transformer to incorporate pairwise features to self-attention like Graphormer. I wonder if the authors have already considered this type of incorporation for their work, but discarded due to poor performance.
clarity,_quality,_novelty_and_reproducibility: Clarity: This paper provides a nice overview of the problem being solved. There are some details missing (like the sampling algorithm for VMC), but I could understand the paper even though I come from a machine learning background.

Novelty: The novelty is not very high since self-attention is not a very new concept. However, I think this is fine given importance of the problem.

Reproducibility: Without any repository, I do not think it is possible to reproduce the paper. 
summary_of_the_review: This paper proposes a simple idea for solving an important problem. I believe significance and complexity of the problem being solved is enough to compensate for the (arguable) lack of novelty. I am concerned about reproducibility of this paper. I do not think the follow-up works will be able to reproduce this paper without allocating a significant amount of time.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed to use self-attention as ansatz in FermiNet. There are some minor changes, including using log distance and spin in the input feature. The proposed model is proved better than FermiNet in several experiments including third-row atoms, large molecules and benzene dimer.
strength_and_weaknesses: Strength: the main strength of this paper is that --- it really works. I believe there are many researchers actively working on incorporating complicated model structure in FermiNet, and many of them may have the idea to use self-attention. This paper is the first one to make this work, which is not a naive accomplishment. The runtime scaled cubically, which means for large systems, FermiNet and PsiFormer ran at nearly the same speed. Also the experiments are very detailed.

Weaknesses: I don't see any major weaknesses, but I have several questions or comments below.

1. Why do you only consider features of electrons? Can we include the features of atoms in the self-attention mechanism? (DeepWF https://arxiv.org/pdf/1807.07014.pdf did something like this.)
2. More ablation study would be helpful for understanding how the number of determinants, number of attention heads, attention dimensions and etc (basically the hyperparameters listed in table 6) make contributions to the accuracy of calculation.
3. The divergence of training energy as in Figure 6 without input rescaling seems quite outrageous. Does the authors see similar issue in FermiNet and FermiNet SchNet?
4. The authors claim that this work is the first time that KFAC is applied to self-attention layer (in Section 4). However, https://arxiv.org/abs/1907.04164 (from KFAC authors) already claimed that this is the first work where KFAC is applied to transformer model which include self-attention.
clarity,_quality,_novelty_and_reproducibility: The paper has good clarity, quality, novelty and reproducibility.
summary_of_the_review: This paper introduce self-attention in the ansatz in FermiNet. I guess the idea may have been studied by many researchers, and this paper is the first one to make it work, which is a good accomplishment.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a self-attention based architecture called PsiFormer for approximating many-electron Schrödinger equation. The architecture uses self-attention transformer to learn the embedding of electrons and the pairwise features are incorporated through Jastrow factor. The authors did a series of experiments on multiple molecular systems and compared PsiFormer with FermiNet, the previous state-of-the-art system for this task.

In general, this is a nice paper, with both algorithmic contribution and experimental advances. One reviewer raised his/her concern about the motivation and the explanability of the work. However, after several rounds of offline discussions among the reviewers, we believe these concerns are not that critical and it would be good to accept the paper to ICLR. We believe the audience will be interested in reading it.

justification_for_why_not_higher_score: Although we decided to make an ACCEPT recommendation, one of the reviewers actually remain a conservative view on the paper. 
justification_for_why_not_lower_score: The paper concerns an important topic and has its merits in many ways.
Decision: Accept: poster