Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work considers the problem of sampling transition paths between two metastable states of a molecular system. This problem is difficult as the energy barrier between the states may be large, making it computationally expensive for traditional MD simulation. This work first relates this problem to literature on optimal control and Schrodinger bridges then provides a parameterization in terms of second-order dynamics to learn likely transition paths. This method is demonstrated on three systems using two slightly different parameterizations, either directly modelling the control force, or indirectly modelling the force as $\nabla E$ and directly parameterizing energy. 
strength_and_weaknesses: Strengths:

- Interesting application of optimal control
- Neat parameterization in terms of second-order dynamics that allows easy integration with OpenMM.
- Well explained experiments which are understandable by non MD experts

Weaknesses:

- Limited theoretical novelty as the theory is established in prior work.
- Limited quantitative experiments on a single (simple) molecule. As a non-expert in MD I can’t evaluate how often good CVs are available and how much they help in this path sampling problem. Do the authors think an experiment with MD using knowledge of the CVs be useful? It would help me understand how this baseline method works. Presumably when the CVs are less accurate (such as in chignolin?) CV-based optimizations would not work as well as PIPS? Could this be confirmed experimentally?
- Another concern here is overfitting. How are we sure that the parameters are not simply overfit to each molecule making it difficult to apply to new molecule simulations? It would be helpful to state how hyperparameters were tuned and on what data.

Comments:

I don’t think I understand remark 3. Is this a claim that equations (2) and (9) are the first to establish the equivalence between sampling transition paths and the SB problem which hasn’t been previously established? I don’t understand how these equations are meant to establish this and would be very surprised if this is the first time this equivalence has been established. 

The notation in the “Physics inspired policy network” is a bit confusing to me with bold $u_\theta$ and regular $u_\theta$ having very different (and unspecified) dimensions. Instead of defining F(r_t) could we say instead that bold $u_\theta$ is either directly parameterized (force prediction) or parameterized as $\nabla_{r_t} E_\theta$ (energy prediction)?

I understand the current evaluation is on the quality of the paths. I would also be curious as to the quality in terms of the optimization, i.e. what is the value of $E_{\tau, \epsilon_t} C(x(\tau), u, \epsilon_t)$ for the force vs. energy models? 

$u_\theta$ with some invariances is mentioned, but the end network is an MLP. Was a more sophisticated network tried? Perhaps it is not helpful for these single-molecule studies.

Question: How does Temperature come in to play? Its slightly odd to me that MD without a fixed timescale is evaluated on a different temperature than PIPS. Could PIPS be evaluated on the same temperatures or could the authors explain why 300K is “better”?

Minor remarks:

Remark 3? is this a novelty claim?

misplaced comma last paragraph of page 5.

Courtsey —> Curtesy Remark 4

It would be useful to mention Table 1 is on Alanine Dipeptide in the caption. Also it slightly strange that it comes before figure 1, but is referenced afterwards.
clarity,_quality,_novelty_and_reproducibility: I found the manuscript quite easy to read and the experiments clearly described. In terms of novelty, this seems like a slight modification to an existing method for application in an interesting domain. As such I was slightly surprised by the relatively small amount of empirical (quantitative) evaluation. 

This paper could also use some more specificity for additional reproducibility. For example it is stated that “The width of the layers of the policy network is dependent on the number of atoms in the molecule under consideration.” What were these widths and how were they determined? Code (which is not available for review as far as I’m aware) could potentially help here.
summary_of_the_review: I found this paper interesting and timely with the increased interest in diffusion-based models. For me I would like to see more experimental validation and rigor in terms of quantitative comparisons and experimental setup.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a control algorithm to explore transition paths between reactant and product basins in molecular dynamics simulations. The work is strictly based on the theory of Knappen and Ruiz (2016), and implements it for molecular systems.

strength_and_weaknesses: 
The loss fucntion that is optimized is a combination of two:

Distance between desired and actual descriptors at the end of the trajectory: This part of the loss is unarguable. The trajectories minimizing this part are obviously transition paths.

A penalty term for the magnitude of the control: This is less justified. 
There is a very strong assumption underneath that the weight of the penalty is directly proportional to the noise. In a theoretical investigation of control this is fine. In a case when we one does not seek physically ideal bias and control strength e.g. when first part of the loss is much more important that the second one, this is also fine. But in case of transition paths and potentially free energy estimates - as proposed in the paper outline - such simplification does not stand and is likely too simple to produce realiable estimates of the potential of the means force.

The gradient update is the weighted average of the trajectories, reinforcing those that had low loss and forgetting the one with high loss. This will always produce a path connecting A and B. But in what sense is this a transition path? Are tehre guarantees that the underlying physics are respected? What would be the result if one optimized just MSE of the (actual - desired) descriptors in the final point?

Appendix A1 There is possibly a typo in \Delta \Theta_n -> .... line. Is the exponential weight with the control gradient or added to?

The authors claim they cannot compare the method to any other. I think the method potentially could be compared to differentiable simulation of protein folding, using codes like TorchMD or JAXMD.

The simulations construct what are essentially transition paths, but are not called so. If this method produces better TS paths then the second part of the loss helps in some way.

It might be easier then to explain the difference between figure 1 and 4 if they were plotted together. Illustrative molecules can be omitted for on of the results as there is not much insight gained from them being just printed.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, and the application of the existing theory to molecular systems is novel, but the underlying methods have been used before.
summary_of_the_review: The approach seems to generate a reasonable path in energy landscape. The proposed approach can be extraordinarily aggressive and may not work well with temperature or to produce potential of mean force. It may be useful for people trying to find transition paths but will likely require downstream refinement to be practically useful.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method for finding the most likely transition between two states in high-dimensional configuration spaces (for chemical compounds, typically). It does so on building on recent advances in the Schrödinger bridge problem (SBP) and stochastic optimal control (SOC).

It claims to show equivalence between sampling transition paths (the most likely one) and these two other equivalent problems (SBP and SOC).

The main novelty is the use of second order Hamiltonian dynamics, i.e. sec 3, in particular Eq (13) (which seems to me like a rather classic trick to assemble position and velocity to make a Dynamical System become Markov-like, under time discretization).

Another novelty, possibly (I am not so aware of the literature) is the very use of SOC / RL-style approaches to transition path sampling. But, this does not seem to be altogether new: people are discussing the use of RL for path sampling, which is very close to the proposed approach (apart from finite vs infinite time horizon).

The proposed approach is tested on 3 classic transition-paths small molecules, with some success (but no comparison with other similar approaches, as Reinforcement Learning).
strength_and_weaknesses: Strengths:
- the notations are overall clear, the content is understandable and concisely reported.
- the experiments are abundantly commented (in terms of commenting the results)

Weaknesses:
- Machine Learning, or Learning Representation, seems quite lacking from the paper. Neural networks are mentioned quickly, only on page 5. 
- mostly, the lack of novelty, in several points. I detail here:
- I recall that Eq (2) is the Schrödinger Bridge Problem (SBP). Then Eq (9), which is claimed to establish equivalence between sampling transition paths, Schrödinger bridge problem, and stochastic optimal control, however, has the term of terminal cost inside it, making it different from (2). Paper says that:
     > Therefore, when the terminal cost dominates the KL-divergence term above, it enforces the target boundary constraints of the problem.

     But in practice, in most problems of finding good transition paths, the initial and final configurations' energies are quite similar, and it's not obvious that the "terminal cost" dominates (I realized afterwards I probably inferred incorrectly what is the terminal cost.. it should be defined early and intuitively in the paper. If its expression depends on how we parameterize the configuration... I do not understand well how it can relate with weights of paths

- algorithmic novelty: here again, PIPS does not seem to significantly differ from PICE (Hilbert Johan Kappen and Hans Christian Ruiz. Adaptive importance sampling for control and inference. Journal of Statistical Physics, 162(5):1244–1266, 2016). Actually the code submitted contains the file PICE.py, which performs Algorithm A.1 (page 13). Indeed, the authors are very honest about it:
     > we note that (Kappen and Ruiz, 2016) is most in line with our work
- on the theory side, the paper seems to be strongly inspired (and it is not hidden at all, on the contrary, which is very much honorable) by works from Kappen:  Kappen, 2005; 2007; Kappen and Ruiz, 2016. All equations/theory/results stated up to end of sec. 2 are attributed to other papers, and mostly these 2 papers.
- On the interpretation side, similarly, Das 2021 and Rose 2021 (including Garrahan) are cited, but not exploited. By very quickly looking at these references, it seems like a good deal of the intuitive physics (that is I think only partially explained in the paper), are explained over there. For instance, papers says:
   > Remark 2. The last term in the cost function in eq. (5) relating the Brownian motion and the control is unusual and devoid of a clear intuition.
    I think intuition comes from considering, where could this external force we apply to the molecule, come from, in the original (no external force) setup ? I think that the answer is the noise: it can "conspire" to correlate to be equal or close to the control policy, .. if one is very lucky (hence the rare transitions are rare !)
- as I said in the summary, the main (only?) novelty is the use of second order Hamiltonian dynamics, i.e. sec 3, which is in 1 page long, in particular Eq (13). However, this seems to me like a rather classic trick to assemble position and velocity, in order to make a Dynamical System become Markov-like, under time discretization. However, I recognize that if it's the needed step to apply new techniques (SOC, SBP, PICE) to transition path sampling, well then, good job. (this could become a strength if I am convinced this is very much new and insightful compared to previous works).
- even sec. 3 is inspired from refs:
     > We, thus, take inspiration from recent computational advances for solving SBP (Vargas et al., 2021a; De Bortoli et al., 2021) to develop our solution in §3 to solve the problem of sampling transition paths that can efficiently cross the high free energy barriers

    but I did not have time to check these references.

clarity,_quality,_novelty_and_reproducibility: Clarity:
- paper is overall understandable, with many points under-defined or un-defined however:
- Eq (2) lacks context and it seem like pi* could just equal pi
- Eq (5) mentions the matrix R, which is equite mysterious to me. In the code however it is equal to the Identity matrix I ... which such complications to end up using I ?  This looks like obfuscation.
- what is lambda, how is interpreted ?
- terminal cost is not defined (can be guessed, but only at page 5..)
- The way in which the policy is learned is not explained at all. MLPs are mentioned in page 6. What is the task, input, output, etc, is not explained at all. 
- I do not understand how the learning accounts for what will happen in the future, in uncertain (noise) conditions. This is very mysterious to me, how a policy which only depends on t, can learn and work (it does seem to work !)

Quality
- overall, the scientific quality is ok, but then most things are recalls of prior works, not original work. 
- the important fact that terminal cost dominates eq (9) is not obvious at all.

Novelty:
- The paper very much lacks novelty. I detailed this at length above in Weaknesses.
- As remark 3 says:
    > This connection between the Schrödinger Bridge Problem and stochastic optimal control has been previously established (Chen et al., 2016; Pavon et al., 2021). However, through the formulations in Equations (2) and (9), we also establish the equivalence between sampling transition paths, Schrödinger bridge problem, and stochastic optimal control.


Reproducibility:
- the code is available anonymously, seems well packaged.
summary_of_the_review: Overall, this paper is rather sound and rather clear, but lacks novelty, by its own account, and additionally does not focus on ML at all.

Hence, I lean to reject, quite strongly, until further arguments are given about the originality of the approach and its merit for the community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The current paper propose a method for sampling transition paths across two meta-stable states when typically done using expensive, resource-intensive MD simulation method for a large macro-molecular system. The author demonstrated their approach using a set of examples. There is several merits of this work, including the well-motivated question that is being addressed here, the clear presentation of their methods and good demonstration and implementation. The weaknesses are mainly focused on the potential issues such as the overall novelty of this method, and if considering the application side, how broadly and useful the method would be across different settings.
justification_for_why_not_higher_score: As noted by several reviewers, while there is clear motivation for the overall work, and this work has very good presentation, the method is based on established theoretical work so thus is a relatively straightforward idea, and on the application side there are still limitations as the reviewers noted that would need to be addressed. The authors could consider alternative forum for presenting this work if they would like to focus more on the application side of this topic.
Decision: Reject