SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\eta(\tau) = \mathbb{E}_{\boldsymbol{x}_{0:\tau} \sim \pi^0}[\exp(-\frac{1}{\lambda}\varphi(\boldsymbol{x}_\tau)]$  is the normalization constant. This establishes the optimal distribution  $\pi^{\boldsymbol{u}^*}$  as a reweighing of any distribution induced by an arbitrary control  $\boldsymbol{u}$ .
PICE, subsequently, achieves this by minimizing the KL-divergence between the optimal controlled distribution  $\pi^{u^*}$  and a parameterized distribution  $\pi^{u_{\theta}}$  using gradient descent as follows:
$$
\frac {\partial \mathbb {D} _ {\mathrm {K L}} \left(\pi^ {\boldsymbol {u} ^ {*}} \mid \pi^ {\boldsymbol {u} _ {\theta}}\right)}{\partial \theta} = - \frac {1}{\eta} \mathbb {E} _ {\boldsymbol {x} _ {0: \tau} \sim \pi_ {\boldsymbol {u} _ {\theta}}} [ \exp (- C (\boldsymbol {x} _ {0: \tau}, \boldsymbol {u} _ {\theta})) \sum_ {t = 0} ^ {\tau} (\boldsymbol {R} \varepsilon_ {t} \cdot \frac {\partial \boldsymbol {u} _ {\theta}}{\partial \theta}) ] \tag {16}
$$
Similar to the optimal control in eq. (15), the gradient used to minimize the KL-divergence is found by reweighing for each sampled trajectory,  $\boldsymbol{x}_{0:\tau}$ , the gradient of the control policy  $\boldsymbol{u}_{\theta}$  by the cost of the trajectory. See Algorithm 1 in the appendix for an algorithmic description of PICE.
4.1 Adaptations to PICE
In this section we will specify the adaptations made to the PICE algorithm to apply it to solve the BPTP problem for the molecular transition path setting.
Smoothing the loss function As shown in the previous section, when using the target loss  $\varphi (\pmb {x}_{\tau}) = -\lambda \log (\mathbf{1}_P(\pmb {r}_\tau))$  , SOC solves the BPTP problem. However, while optimal, this loss function is hard to use in the PICE optimization task as it is infinite for all  $x_{0:\tau}$  where  $\pmb {r}_{\tau}\neq \pmb{r}_{\tau}^{*}$  . As such, we instead use a smoothed version  $\varphi (\pmb {r}_t) = \exp \sum_{i,j}^n\left(d_{ij}(\pmb {r}_t) - d_{ij}(\pmb {r}_\tau)\right)^2$  where  $d_{ij}(\pmb {r}_t) = \| (\pmb {r}_t)_i - (\pmb {r}_t)_j\| _2^2$  This exponentiated pairwise distance between the atoms is a commonly used distance metric [Shi et al., 2021] that is invariant to rotations and translations of the molecular system.
Architectural considerations The learnable component of PIPS is the bias potential  $b$ . However, as the BPTP dynamics show in eq. (5), instead of using the bias potential directly, MD depends on the bias force — the gradient of the bias potential  $\pmb{b}(\pmb{r},\pmb{v}) = \nabla_{\pmb{r}}b(\pmb{r},\pmb{v}) \in \mathbb{R}^{3n}$ . This consideration allows for two different modelling approaches for the bias force similar to the distinction between metadynamics and adaptive bias force discussed in section 2.3.1. One can either parameterise the bias force directly  $\pmb{b}(\pmb{r},\pmb{v}) = \pmb{b}_{\theta}(\pmb{r},\pmb{v})$  or, alternatively, model  $b_{\theta}(\pmb{r},\pmb{v})$  the bias potential and calculate the corresponding bias force by backpropagation,  $\pmb{b}(\pmb{r},\pmb{v}) = \nabla_{\pmb{r}}b_{\theta}(\pmb{r},\pmb{v})$ . The advantage of the latter is that the forces are conservative by construction.
In section 5.1 we will compare both these modelling approaches. In both cases we will use a MLP with ReLU activation for either the parameterized bias force or bias potential. Alternatively,  $\pmb{b}_{\theta}$  or  $b_{\theta}$  could be implemented using recent advances in physics inspired equivariant neural networks [Cohen and Welling, 2016, Satorras et al., 2021] that take into account the SE(3) symmetry of the system. We provide details for training the control network  $\pmb{u}_{\theta}$  in Appendix B.
Integration with MD simulation frameworks To efficiently calculate the Potential  $U(\pmb{x})$  and integrate the MD in eq. (1), various optimized simulation frameworks are available. In our work we use the OpenMM framework [Eastman et al., 2017]. The bias force  $\pmb{b}(\pmb{r},\pmb{v})$  is integrated in OpenMM as a custom external force. Implementing the control this way allows us to use the
optimized configuration capabilities of OpenMM, such as forcefield definitions (the potential function description) and integrators (for the time-discretization of our dynamics).
One downside of using OpenMM for integrating the MD is that it does not provide access to the noise  $\varepsilon_{t} \sim \sqrt{2m\gamma k_{B}T}$  dW used in the Langevin thermostat that is needed to calculate the update to the policy weights. To circumvent this, we instead sample an additional exploratory noise term  $\hat{\varepsilon}_{t} \sim \mathrm{d}\pmb{W}$  with variance  $\hat{\nu}$  that is used to optimize the policy and assume the Langevin noise to be part of the drift of the system  $\pmb{f}$ . While this loses the formal guarantees presented in section 3, we found this to be experimentally stable and provide close to optimal trajectory paths (as shown in section 5.1).
We evaluate PIPS using three molecular systems, namely (i) Alanine Dipeptide, to compare PIPS to CV free and CV dependent baselines, (ii) Polyproline, to evaluate PIPS as a method to select candidate CVs, and (iii) Chignolin, as a use-case of PIPS scalability to proteins without knowns CVs.
We report the molecule specific OpenMM configuration as well as the used neural network architecture to learn the bias potential/force in appendix C. Generally, we run our simulations at  $300\mathrm{K}$  and use 6 layer MLP with the width of the layers dependent on the number of atoms in the molecule under consideration. Our code, including a full stand-alone notebook re-implementation, is available here: https://github.com/LarsHoldijk/SOCTransitionPaths.
5.1Alanine Dipeptide
In this section we evaluate PIPS on the extensive studied Alanine Dipeptide (AD) molecule. AD is a relatively small protein for which the CVs (two dihedral angles  $\phi$  and  $\psi$ ) are readily available and is therefore well suited for the development of enhanced sampling methods that require CVs. While PIPS does not use the CVs during training, their availability does come in useful to evaluate the sampled transition. The transition evaluated here have a 500 fs time horizon.
5.1.1 Quantitative comparison to CV free baselines
As discussed, our work is the first to consider CV free sampling of transition paths at this scale and as such other baselines or metrics are not available. In table 1 we therefore evaluate PIPS using MD simulations with extended time-horizon and increased system temperature as baselines and introduce three metrics to evaluate the quality of the transition paths. (i) Expected Pairwise Distance (EPD) measures the euclidean distance between the final conformation in the trajectory and the target conformation, reflecting the goal of the transition to end in the target state, (ii) Target Hit Percentage (THP) assures that the final configuration is also close in terms of CVs by measuring the percentage of trajectories correctly transforming these CVs, and (iii) Energy Transition Point (ETP)
τ
fs	Temp. K	EPD (↓) nm × 10-3	THP (↑) %	ETP (↓) kJ mol-1
Bias Force Prediction	500	300	2.56 ± 0.34	45.0 %	0.55 ± 11.30
Bias Potential Prediction	500	300	1.21 ± 0.31	63.5 %	-8.35 ± 8.04
MD w. fixed timescale	500	300	8.50 ± 0.67	0%	-
500	1500	7.75 ± 1.72	0%	-
500	4500	6.77 ± 2.41	0.1%	317.79 ± 0.00
500	9000	6.99 ± 2.56	1.6 %	772.57 ± 108.55
MD w/ fixed timescale	6818.4 ± 5420.8	1500	3.08 ± 0.68	100%	393.76 ± 68.67
3471.7 ± 1646.5	4500	6.42 ± 2.67	100%	1186.84 ± 212.00
Table 1: Benchmark scores for the proposed method and extended MD baselines. From-left-to-right: Time-horizon  $\tau$  representing the trajectory length (note that we take one policy step every 1 fs), simulation temperature, Expected Pairwise distance (EPD), Target Hit Percentage (THP), and Energy Transition Point (ETP). ETP can only be calculate when a trajectory reaches the target. All metrics are averaged over 1000 trajectories except for MD w/ fixed timescale which is ran only for 10 trajectories.
Figure 2: Visualization of a trajectory sampled with PIPS. Left: The sampled trajectory projected on the free energy landscape of AD as a function of two CVs Right: Conformations along the sampled trajectory: A) starting conformation showing the CV dihedral angles, B-D) intermediate conformations with C being the highest energy point on the trajectory, and E) final conformation, which closely aligns with the target conformation. Bottom: Potential energy during transition.
which evaluates the capacity of each method to find transition paths that cross the high-energy barrier at a low point by taking the maximum potential energy of the molecule along the trajectory. A good trajectory will be one that passes through the minimal high-energy barrier and ETP aims to measure this. We provide more details in Appendix C.2.1.
Results: We find that the trajectories generated by both the policy networks outperform the MD baselines, but the more physics-aligned potential predicting policy performs best under our metrics. This policy network consistently reaches the target conformation both in terms of full geometry and the CVs orientation. Furthermore, our policy network generates these trajectories in a significantly shorter time than temperature enhanced MD simulations without a fixed timescale. When we do limit MD to run for the same timescale as the proposed method, we found that, in contrast to the proposed method, temperature enhanced MD simulations are unable to generate successful trajectories. We will use the bias potential predicting policy in the following.
5.1.2 Qualitative comparison to CV dependent metadynamics
In fig. 2 we visualise an AD transition sampled by PIPS using the bias potential predicting policy. In the top left, we overlay the transition projected onto CV space on the free-energy surface generated using metadynamics. The free-energy surface thus serves as a ground-truth generated using a method that requires extensive domain knowledge. We aim to show that the trajectory sampled using PIPS aligns with the saddle points of the metadynamics free-energy surface.
Results: The trajectory in Figure 2 demonstrates that the bias potential control policy transforms the molecule from the initial position (A) to the final position (E) by transitioning over the same saddle point in the free-energy barrier found by metadynamics (C). This shows that the trajectory follows the same transition in CV space as metadynamics despite, contrarily to metadynamics, not being biased to do so. The potential energy goes up during the transition until it reaches the lowest point of the energy barrier (C) and consecutively settles down in its new low-energy state.
5.2 Polyproline Helix
Second, we consider a Polyproline trimer with 3 proline residues. Polyproline is a more complex protein then AD and as such its CVs are less well understood. We therefore use this protein to determine if PIPS biases the system along the correct CVs when a collection of candidate CVs are available. Specifically, we consider the peptide bond orientation  $(\omega)$  and two backbone dihedral angles  $(\phi$  and  $\psi$ ). As initial and target state we provide a single example of Polyprolines PP-I form (with cis-isomer peptide bonds) and PP-II form (with trans-isomer peptide bonds) respectively. For this transition it is known that the CV of interest are the peptide bond orientation. Additionally, to
Figure 3: Visualization of the Polyproline transformation from PP-II to PP-I. From-top-to-bottom 5 stages of the transition,  $\psi$ ,  $\phi$ ,  $\omega$  candidate CVs, and Potential Energy. For the candidate CVs multiple instances of the same dihedral angles can be found in a single molecule. Stars indicate target candidate CV states. Colored bonds represent the bonds involved in the  $\omega$  CV.
study PIPS resilience to target misspecification, the supplied PP-II form also contains a transformation in one of the  $\psi$ -dihedral angles which is irrelevant to the transition. The transition time is 5000 fs.
Results: We visualize the transformation of the three collective variables  $(\omega, \phi, \psi)$  as well as the corresponding potential energy of the conformation in Figure 3 for a sampled transition path. We observe that the transition correctly occurs along the  $\omega$  CV going from  $180^{\circ}$  to  $0^{\circ}$ . This suggests that PIPS could be used for testing the validity of candidate CVs. However, we also observe that in addition to the peptide bonds PIPS also biases the system along one of the  $\psi$ -dihedral angles due to the introduced target misspecification. As the small fluctuations are to be expected when sampling a single target from the Boltzmann distribution, alternative methods for specifying the target state should be explored in future work.
5.3 Chignolin
Lastly, we consider the small  $\beta$ -hairpin protein Chignolin. Chignolin was artificially constructed to study protein folding mechanisms [Honda et al., 2004, Seibert et al., 2005]. Due to its small size, its folding process is easier to study than larger scale proteins while being similar enough to shed light on this complex process. In contrast to Alanine Dipeptide and Polyproline, there is no agreement on the transition mechanism describing the folding of Chignolin. Both the CVs involved [Satoh et al., 2006, Paissoni and Camilloni, 2021], as well as the sequence of steps [Harada and Kitao, 2011, Satoh et al., 2006, Suenaga et al., 2007, Enemark et al., 2012] describing the folding process have multiple different interpretations. Chignolin thus serves as a usecase study for scaling PIPS beyond traditional CV-based approaches to solve the BPTP-problem. We sample transition paths between the folded and unfolded state of the Chignolin protein using a total time horizon of 5000 fs. Note that the typical folding time of Chignolin is recorded to be  $0.6~\mu s$  [Lindorff-Larsen et al., 2011].
Results: In Figure 4, we visualize the transition of Chignolin at 5 different timesteps during the transition path. We observe that to transition the protein from its low energy unfolded state to the folded conformation, the proposed method guides the protein into a region of higher energy. This increase is initially more steep  $(0\rightarrow 1500)$  than in the later stages. Additionally, most of the finer-grained folding  $(2500\to 4000)$  occurs with a high potential energy before settling into the lower-energy folded state. We notice that for the restricted folding time we use in our experiments (5000 fs vs  $0.6\mu s$ ), the molecule does not end at the final configuration but reaches close to it as shown by the plot on pairwise distance. Furthermore, the learned policy network is able to transition through the high energy transition barrier in this restricted time. We do not encounter this for molecules with a shorter natural transition time (as illustrated by the potential energy of Alanine Dipeptide in fig. 2).
Figure 4: Visualization of the Chignolin folding process. Top: 5 stages of the folding process, Middle: Pairwise distance wrt to the target conformation of the molecule, Bottom: Potential Energy.
In this work, we have proposed PIPS—a path integral stochastic optimal control method for the problem of molecular sampling transition paths using a bias potential. In contrast to prior work, PIPS does not require prespecifying CVs along which the system should be biased. We show the benefits of PIPS using three different molecular systems of varying sizes. In passing, we gave an introductory description of the problem of sampling transition paths and related it to the stochastic optimal control and the Schrodinger bridge problem. With this, we hope to not only have motivated our own work but also provided a starting point for future work consideration of this important problem by the machine learning community. For future work, we specifically note that the use of PIPS for CV discovery and the exploration of other approaches for specifying the target state, possibly using an ensemble of samples, is a promising direction as exemplified by our Polyproline experiment.
We would like to thank Rianne van den Berg for their valuable feedback. Lars Holdijk is supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (EP/S024050/1).
Daan Frenkel and Berend Smit. Understanding molecular simulation: from algorithms to applications, volume 1. Elsevier, 2001.
Scott A. Hollingsworth and Ron O. Dror. Molecular Dynamics Simulation for All. Neuron, 99(6): 1129-1143, September 2018.
Oliver T. Unke, Stefan Chmiela, Huziel E. Sauceda, Michael Gastegger, Igor Poltavsky, Kristof T. Schütt, Alexandre Tkatchenko, and Klaus-Robert Müller. Machine Learning Force Fields. Chemical Reviews, 121(16):10142-10186, August 2021.
Xiang Fu, Zhenghao Wu, Wujie Wang, Tian Xie, Sinan Keten, Rafael Gomez-Bombarelli, and Tommi Jaakkola. Forces are not enough: Benchmark and critical evaluation for machine learning force fields with molecular simulations. arXiv preprint arXiv:2210.07237, 2022.
Peter G Bolhuis, David Chandler, Christoph Dellago, and Phillip L Geissler. Transition path sampling: Throwing ropes over rough mountain passes, in the dark. Annual review of physical chemistry, 53 (1):291-318, 2002.
Glenn M Torrie and John P Valleau. Nonphysical sampling distributions in Monte Carlo free-energy estimation: Umbrella sampling. Journal of Computational Physics, 23(2):187-199, 1977.
Alessandro Laio and Michele Parrinello. Escaping free-energy minima. Proceedings of the National Academy of Sciences, 99(20):12562-12566, October 2002.
Atsushi Suenaga, Tetsu Narumi, Noriyuki Futatsugi, Ryoko Yanai, Yousuke Ohno, Noriaki Okimoto, and Makoto Taiji. Folding dynamics of 10-residue  $\beta$ -hairpin peptide chignolin. *Chemistry-An Asian Journal*, 2(5):591-598, 2007.
Søren Enremark, Nicholas A Kurniawan, and Raj Rajagopalan.  $\beta$ -Hairpin forms by rolling up from C-terminal: Topological guidance of early folding dynamics. Scientific Reports, 2(1):1-6, 2012.
Kresten Lindorff-Larsen, Stefano Piana, Ron O Dror, and David E Shaw. How fast-folding proteins fold. Science, 334(6055):517-520, 2011.
Robert H Cameron and William T Martin. Transformations of weiner integrals under translations. Annals of Mathematics, pages 386-396, 1944.
David A Sivak, John D Chodera, and Gavin E Crooks. Using nonequilibrium fluctuation theorems to understand and correct errors in equilibrium and nonequilibrium simulations of discrete Langevin dynamics. Physical Review X, 3(1):011007, 2013.
Kresten Lindorff-Larsen, Stefano Piana, Kim Palmo, Paul Maragakis, John L Klepeis, Ron O Dror, and David E Shaw. Improved side-chain torsion potentials for the Amber ff99SB protein force field. Proteins: Structure, Function, and Bioinformatics, 78(8):1950-1958, 2010.
Ulrich Essmann, Lalith Perera, Max L Berkowitz, Tom Darden, Hsing Lee, and Lee G Pedersen. A smooth particle mesh Ewald method. The Journal of chemical physics, 103(19):8577-8593, 1995.
A Proof theorem: SOC solves the BPTP problem
Theorem A.1 (SOC solves the BPTP problem). Given  $\pmb{x}_t = (\pmb{r}_t, \pmb{v}_t)^T$ ,  $\pmb{f}(\pmb{x}_t) = (\pmb{v}_t, \frac{-\nabla_{\pmb{r}_t} U(\pmb{r}_t)}{\pmb{m}} - \gamma \pmb{v}_t)^T$ ,  $\pmb{G}(\pmb{x}_t) = (\mathbf{0}_{3n}, \mathbb{I}_{3n})^T$ ,  $\pmb{u}(\pmb{x}_t) = \frac{-\nabla_{\pmb{r}_t} b(\pmb{r}_t, \pmb{v}_t)}{\pmb{m}}$ ,  $\nu = 2m \gamma k_B T$ , and  $\pi_0 = \pi_G$ , such that the SOC dynamics (eq. (11)) describe the dynamics of the BPTP distribution  $\pi^b$  (eq. (5)).
If we define  $\varphi(\pmb{x}_{\tau}) = -\lambda \log(\mathbf{1}_P(\pmb{r}_{\tau}))$ ,  $\pmb{R} = \lambda \nu^{-1} = \lambda (2\pmb{m}\gamma k_B T)^{-1}$  and assume  $\pmb{r}_0 \in R$ , we have
$$
\underset {b} {\arg \min } \mathbb {E} _ {\boldsymbol {x} _ {0: \tau} \sim \pi^ {b}} \left[ C (\boldsymbol {x} _ {0: \tau}) \right] = \underset {b} {\arg \min } \mathbb {D} _ {\mathrm {K L}} \left(\pi^ {b} \mid \pi^ {*}\right), \tag {17}
$$
Proof. Let  $\pi^b$  be the BPTP distribution as defined in eq. (7). Crucially,  $\pi^b$  can be factored into a position and velocity component based on the conditional independence of  $r_{t + 1}$  and  $v_{t + 1}$  given  $r_t$  and  $v_{t}$ , respectively, as
$$
\pi^ {b} \left(\boldsymbol {x} _ {0: \tau}\right) = \pi_ {\boldsymbol {r}} ^ {b} \left(\boldsymbol {x} _ {0: \tau}\right) \cdot \pi_ {\boldsymbol {v}} ^ {b} \left(\boldsymbol {x} _ {0: \tau}\right) \tag {18}
$$
$$
\pi_ {\boldsymbol {r}} ^ {b} \left(\boldsymbol {x} _ {0: \tau}\right) = \prod_ {t = 0} ^ {\tau} \mathbb {1} _ {\left[ \boldsymbol {r} _ {t + 1} = \boldsymbol {r} _ {t} + \boldsymbol {v} _ {t} \right]} \left(\boldsymbol {r} _ {t + 1}\right) \tag {19}
$$
$$
\pi_ {\boldsymbol {v}} ^ {b} \left(\boldsymbol {x} _ {0: \tau}\right) = \prod_ {t = 0} ^ {\tau} \mathcal {N} \left(\boldsymbol {v} _ {t + 1} \mid \boldsymbol {\mu} _ {t}, \Sigma_ {t}\right). \tag {20}
$$
where  $\pmb{\mu}_t = (\pmb{v}_t \cdot \mathrm{d}t, \frac{-\nabla_r(U(\pmb{r}_t) + b(\pmb{r}_t,\pmb{v}_t))}{m} \cdot \mathrm{d}t - \gamma \pmb{v}_t \cdot \mathrm{d}t)^T$  and  $\Sigma = \operatorname{diag}(0, 2m\gamma k_B T)$ .
Now, if we define  $\pi^0$  to be the BPTP distribution where no additional bias potential is applied, i.e.  $b(\pmb{r}_t,\pmb{x}_t) = 0$  such that  $\pi^0 (\pmb{x}_{0:\tau}) = \mathbf{1}_R(\pmb{r}_0)\cdot \pi (\pmb{x}_{0:\tau})$ , we observe that the position component of the factorization are equal:  $\pi_r^b (\pmb{x}_{0:\tau}) = \pi_r^0 (\pmb{x}_{0:\tau})$ .
Following, we use Girsanov's [Cameron and Martin, 1944] theorem to relate  $\pi_{\pmb{v}}^{b}\big(\pmb{x}_{0:\tau}\big)$  and  $\pi_{\pmb{v}}^{0}\big(\pmb{x}_{0:\tau}\big)$  as
$$
\pi_ {\boldsymbol {v}} ^ {b} \left(\boldsymbol {x} _ {0: \tau}\right) = \pi_ {\boldsymbol {v}} ^ {0} \left(\boldsymbol {x} _ {0: \tau}\right) \cdot \exp \left(\frac {1}{\lambda} \sum_ {t = 0} ^ {\tau - 1} \frac {1}{2} \boldsymbol {u} \left(\boldsymbol {x} _ {t}\right) ^ {T} \boldsymbol {R} \boldsymbol {u} \left(\boldsymbol {x} _ {t}\right) + \boldsymbol {u} \left(\boldsymbol {x} _ {t}\right) ^ {T} \boldsymbol {R} \boldsymbol {\varepsilon} _ {t}\right) \tag {21}
$$
where  $\varepsilon = G^{-1}(\pmb{x}_t)(\mathrm{d}\pmb{x} - \pmb{f}(\pmb{x}_t)\mathrm{d}t) - \pmb{u}(\pmb{x}_t)$ . Which, given the previously established equality between the velocity components of the BPTP factorization, gives us
$$
\log \frac {\pi^ {b} \left(\boldsymbol {x} _ {0 : \tau}\right)}{\pi^ {0} \left(\boldsymbol {x} _ {0 : \tau}\right)} = \frac {1}{\lambda} \sum_ {t = 0} ^ {\tau - 1} \frac {1}{2} \boldsymbol {u} \left(\boldsymbol {x} _ {t}\right) ^ {T} \boldsymbol {R} \boldsymbol {u} \left(\boldsymbol {x} _ {t}\right) + \boldsymbol {u} \left(\boldsymbol {x} _ {t}\right) ^ {T} \boldsymbol {R} \boldsymbol {\varepsilon} _ {t} \tag {22}
$$
where  $\varepsilon = G^{-1}(\pmb{x}_t)(\mathrm{d}\pmb{x} - \pmb{f}(\pmb{x}_t)\mathrm{d}t) - \pmb{u}(\pmb{x}_t)$ .
This allows us to rewrite the control cost eq. (13) as
$$
C \left(x _ {0: \tau}\right) = \frac {1}{\lambda} \left(\varphi \left(\boldsymbol {x} _ {0: \tau}\right)\right) + \log \frac {\pi^ {b} \left(\boldsymbol {x} _ {0 : \tau}\right)}{\pi^ {0} \left(\boldsymbol {x} _ {0 : \tau}\right)} \tag {23}
$$
Finally, this gives